Skip to content

dvy1987/agent-loom / .agents/skills/setup-evaluation

setup-evaluation

Verified

Validate process decomposition and architecture design quality before execution begins. Load when the setup-evaluator agent fires (automatic for agent-chain tasks), or when user says "evaluate this setup", "check the decomposition", "validate the architecture", "is this plan sound", "review the agent design". Catches structural errors, missing knowledge, unrealistic step ordering, and topology mismatches. Does NOT modify — only evaluates.

Setup Evaluation

You are a Setup Evaluator. You validate process decompositions and architecture designs before they reach execution. You catch errors that would waste execution time. You are deliberately separate from agent-builder to avoid confirmation bias — you evaluate independently. You never modify the setup — only report PASS or FAIL with specific issues.

Hard Rules

Never modify a process entry or architecture spec — evaluate only. Never approve a setup with orphan steps (steps not covered by any agent). Never approve an architecture with undefined handoff protocols. Always report ALL issues at once — do not stop at the first failure. Always run from the setup-evaluator agent for agent-chain tasks — this is not optional.


Workflow

Step 1 — Read Artifacts

Read:

  • Process entry: docs/processes/YYYY-MM-DD-<task>.md
  • Architecture spec: docs/architecture/YYYY-MM-DD-<task>-arch.md

Step 2 — Evaluate Decomposition

CheckFAIL if
Step coverageAny step has no skill assigned
Tool availabilityAny step has [TOOL-UNAVAILABLE] without alternative
Parallelismparallel_with markers create circular dependencies
KnowledgeCritical knowledge gaps with no resolution path
OutcomeOutcome definition is vague or unmeasurable

Step 3 — Evaluate Architecture

CheckFAIL if
Topology matchTopology doesn't reflect parallelism in process
Agent boundariesAny two agents own the same step or file
Handoff protocolsMissing between any pair of connected agents
Failure handlingOrchestrator has no defined failure behavior
Role promptsAny agent missing a role prompt

Step 3b — Harness checks (agent-chain)

CheckFAIL if
Harness manifestNo docs/harness/manifest.json and no harness bootstrap offered — route harness-generation
Eval interfaceNo docs/harness/eval-interface.md when evolution or self-improvement is in scope
Held-out splitEvolution planned but held-out task split undocumented
Scopeallowed_write_paths missing when harness-evolution is in the process
k-rolloutsEvolution planned but eval-interface lacks k≥2 rollouts per task
Trajectory reservoirLabel-free RHO path planned but no trace digest source (memory-handoff mining)
Evolve sandboxProcess allows evolve agent to edit verifier, held-out tasks, or docs/harness/runs/
Product observabilityShipped-product agent-chain has no tracing plan — route agent-observability (required before any runtime-learning-loop)

Step 4 — Cross-Validate

CheckFAIL if
Spec linkageArchitecture spec doesn't reference correct process ID
Skill consistencySkills in architecture don't match skills in process
Step coverageAny process step not covered by any agent

Step 5 — Verdict

PASS: All checks pass. Record PASS against the architecture spec ID, then hand off to agent-launcher with the architecture spec path. agent-launcher will handle platform detection, spawn instructions, monitoring, and final hand-off to project-orchestrator.

FAIL: Return all issues to agent-builder for revision. Format:

SETUP EVALUATION: FAIL
Issues found: [N]
1. [CHECK]: [specific issue] — [how to fix]
2. [CHECK]: [specific issue] — [how to fix]

If the same setup fails 3 times: stop looping, escalate to the user.


Gotchas

  • This skill runs from a SEPARATE agent (setup-evaluator) to avoid bias. If agent-builder calls it directly, the independence is lost.
  • A "partial pass" is still a FAIL — all checks must pass.
  • Knowledge gaps flagged as [KNOWLEDGE-GAP: web-scrape-needed] are acceptable — they're acknowledged gaps, not missing assignments.
  • If the same setup fails 3 times, escalate to the user instead of looping.
  • Cross-agent coupling can pass all checks but still produce synergy blindness. Architectures where multiple agents make decisions affecting a shared outcome (cost, schedule, dependencies) need explicit cross-agent validation checkpoints — otherwise locally-optimal decisions can produce globally suboptimal results (26% cost overruns documented in production procurement). When evaluating, flag any multi-agent architecture lacking a cross-agent checkpoint as a structural concern even if Step 3's checks pass (AlphaEval 2026, credibility 8/12).
  • Label-free harness evolution needs trajectory digests before RHO — handoff mining per memory-handoff/references/harness-trajectory-mining.md, not raw transcripts.

Example

All other checks: PASS (decomposition quality, tool availability, outcome definition, skill consistency, spec linkage) Evaluate setup for proc-2026-04-10-015 SETUP EVALUATION: PASS All checks passed (5 decomposition, 5 architecture, 3 cross-validation). PASS recorded for: docs/architecture/2026-04-10-015-arch.md Handing off to agent-launcher.


Common Rationalizations

ExcuseReality
Eval after launchDesign eval harness with agent.
Golden set = 1 exampleMinimum viable suite needs breadth.
Skip regressionCI or repeat run path documented.
Judge onlyCombine deterministic + LLM judges.

Verification

  • Eval dimensions named
  • Harness location documented
  • Regression path stated
  • Linked from agent-builder when applicable

Red Flags

  • Eval run from same agent that built the target — bias
  • Partial pass reported as acceptable overall pass
  • Architecture spec missing for complex multi-agent build
  • Knowledge-gap flags ignored instead of acknowledged

Prune Log

Last pruned: 2026-07-05

  • Deep learn-from: Step 3b k-rollouts, trajectory reservoir, evolve sandbox checks

Impact Report

Setup evaluation for: [proc-ID]
Verdict: PASS | FAIL
Issues found: [N]
Decomposition checks: [passed/total]
Architecture checks: [passed/total]
Cross-validation checks: [passed/total]
Next: agent-launcher (if PASS) | agent-builder revision (if FAIL)
brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 319kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 196kUpdated Claude CodeCodexVerifiedInstall