ericrisco/rsc-harness / skills/agent-eval
agent-eval
VerifiedUse when measuring whether an LLM or agent system actually got better and gating merges on it: golden sets, fixing an inflated LLM-as-judge, scoring RAG (faithfulness, contextual recall) or agent trajectories (tool correctness, completion), or picking an eval framework. NOT building the agent loop, tools or RAG plumbing (that is `building-agents`).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 12.6 KB |
| evals/README.md | markdown | 847 B |
| evals/cases.yaml | config | 3.3 KB |
| references/judge-design.md | markdown | 5.1 KB |
| references/runner-and-gate.md | markdown | 5.8 KB |
| scripts/verify.shexecutable | script | 5.8 KB |
evals/cases.yaml
View on GitHubskill: agent-eval
should_trigger:
- prompt: "Is the new system prompt actually better than the old one, or am I just fooling myself?"
why: "Before/after on a fixed dataset is the core job. Non-obvious: uses no eval vocabulary at all."
- prompt: "Build me a golden set and a CI check that fails the PR if our support bot's answer quality drops."
why: "Dataset construction plus a block-on-regression gate, stated explicitly."
- prompt: "Our LLM judge gives every answer a 9 out of 10 — the scores are useless."
why: "Judge calibration / inflation; the fix is rationale-forcing rubric and pairwise over pointwise. Non-obvious symptom phrasing, no mention of 'eval'."
- prompt: "Score our RAG pipeline for faithfulness and whether it actually uses the retrieved context."
why: "RAG metrics — faithfulness and contextual recall — are squarely this skill."
- prompt: "I need to know if the agent took the right tool path, not just whether the final answer looks ok."
why: "Trajectory and tool-correctness scoring. Non-obvious: framed as behavior, not as 'eval'."
- prompt: "Necesito medir si el agente mejoró con el modelo nuevo antes de hacer merge."
why: "Spanish — before/after measurement plus a pre-merge gate."
- prompt: "Should we use DeepEval or Inspect AI for our eval suite, and how do we wire the gate?"
why: "Framework choice plus gate wiring; the cheat-sheet and runner are the answer."
should_not_trigger:
- prompt: "Rewrite this system prompt so the answers come out shorter and punchier."
route_to: prompt-engineering
why: "Changing the words, not measuring quality. Evals say it is worse; that skill changes it."
- prompt: "Build the agent loop and tool registry for our new assistant."
route_to: building-agents
why: "Building the system under test, not scoring it."
- prompt: "Add a dashboard showing token cost and latency of our live LLM traffic."
route_to: observability
why: "Online monitoring of production, not offline pre-merge evaluation."
- prompt: "Write pytest unit tests for our data-cleaning functions."
route_to: testing-py
why: "Deterministic code with assert-equals, not stochastic outputs scored by a judge."
- prompt: "Red-team our chatbot for jailbreaks and prompt injection."
route_to: agent-safety
why: "Adversarial safety coverage, not quality measurement on a golden set."
capability:
- scenario: "Set up an eval plus a regression gate for a RAG support bot that is regressing silently between releases."
must_include:
- "Builds a versioned JSONL golden set of 50-200 hand-labeled cases per failure mode, decontaminated, not synthetic-only"
- "Picks a scorer mix: deterministic first (schema/regex/latency), faithfulness and answer-relevancy judge where meaning matters"
- "Calibrates the LLM-as-judge against human labels and reports agreement (raw and/or Cohen's kappa) before trusting it"
- "Gate policy is block-on-regression vs a committed baseline with a bootstrap CI, not an absolute threshold"
- "Runner emits eval-report.json and exits non-zero to block the merge"
- "Names at least one concrete framework (DeepEval, Inspect AI, or promptfoo) and fits it to the case"
- "Records the eval policy in 02-DOCS/wiki/stack/evals.md if a harness exists — recorded, not gated"
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |