undefined-ui/second-brain-os / plugins/agents-course/skills/evals-bootstrap
evals-bootstrap
UnverifiedScaffold a first eval suite for an agent: mine real failures into cases, write behavioural checks over traces, and generate the runner. Use when the user wants evals, regression tests for an agent, a golden set, or asks how to know a prompt/model change did not break things. Do NOT use for a single task's done-check (goal-test) or for auditing context (context-audit).
Bootstrap the eval suite
Theory: Two kinds of checks and the full walkthrough in Evals practice. An eval suite is the same test after every change. Behavioural checks read the steps of a trace; end-to-end checks read only the result. Start behavioural: they are deterministic, run in seconds, and diagnose instead of just scoring.
Workflow
- Find real failures. Ask where the agent's runs live (logs, transcripts, a traces directory). Read until you have up to twenty real failures — not imagined ones. If there are no logged runs yet, build the trace logging first (step 3) and seed the suite with the three failures the user can recall; a small honest suite beats a large invented one.
- One line per failure. For each: the input, and the one specific behaviour that should have happened and did not. Failures cluster into four to eight behaviours; name them.
- Ensure traces exist. Each run must be stored as
traces/<id>.json— a list of events including tool calls. If the user's harness is Claude Code, the transcript already is the trace; wire up whatever copies or converts it. No trace, no behavioural checks. - Write
cases.yaml. One entry per failure:
- id: refund_1042
input: "Refund order #1042, customer says it arrived broken"
expect: looks up the order before replying; asks approval before refund
check:
- trace has get_order before send_reply
- trace has approval_request before refund
The expect line is for humans; the check lines are the test. Keep the
rule language tiny: trace has X, trace has X before Y, trace lacks X.
5. Generate check_traces.py. A small runner: load cases.yaml, parse
each rule with a regex, walk the tool-call list, print one line per case,
exit non-zero on any failure. Keep it dependency-light (pyyaml only) and
fast — the whole suite should run in seconds, with no model calls.
6. Run it and hand over the flywheel. Show the pass/fail lines. Then
leave the loop in writing at the end of your report: read fresh traces
weekly, add every new failure as a case, fix the biggest cluster, re-run.
Rules
- Every case comes from a real failure; delete a case only when the behaviour it guards is retired, not when it is inconvenient.
- No LLM-as-judge in the bootstrap. Add a judge later, only for what assertions cannot reach, and calibrate it against human labels first.
- The suite must be one command (
python check_traces.py) so it can gate a CI job or a pre-release habit without ceremony.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 463k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 463k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 463k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |