hashgraph-online/awesome-codex-plugins / plugins/ariaxhan/kernel-claude/skills/eval
eval
UnverifiedEval-Driven Development (EDD) for AI workflows. pass@k metrics, capability evals, regression evals. Triggers: eval, edd, pass@k, capability, regression, benchmark.
<core_principles>
- DEFINE BEFORE CODE: Evals written first force clear thinking about success criteria.
- CODE GRADERS > MODEL GRADERS: Deterministic checks beat probabilistic judgments.
- STRUCTURAL SEPARATION FOR HIGH-STAKES: When stakes are real (security, payments, eval-of-evals, agent quality scoring), use the blind-evaluator agent — never self-score. Self-scoring inflates results ~36% structurally; procedural separation ("I won't peek") does not fix it.
- TRACK PASS@K: pass@1 (first attempt), pass@3 (within 3 attempts). Target pass@3 > 90%.
- REGRESSION BEFORE SHIP: Every change must pass existing evals before merge.
- FAST EVALS GET RUN: Slow evals get skipped. Keep evaluation fast. </core_principles>
<blind_evaluation_protocol> Use when implementing agent would otherwise score its own output (high-stakes: security, payments, agent quality):
- Spawn
agents/blind-evaluator.mdas a fresh agent. - Pass ONLY: problem statement, rubric (3-7 criteria with PASS conditions + weights), artifact path.
- Do NOT pass: implementer's checkpoint, summary, commit message, prompt, or expected solution.
- (gate: blind evaluator runs contamination check — if forbidden inputs detected, returns INVALID; clean inputs and retry)
- (gate: confidence < 0.7 from blind evaluator → escalate to human grader)
Two-phase eval protocol:
- Run 1: implementing agent solves cold, no eval feedback. Blind evaluator scores. This is the externally-reportable number.
- Run 2: implementing agent gets Run 1 score + rubric breakdown, then optimizes. For iteration only. </blind_evaluation_protocol>
pass^k: "All k trials succeed"
- pass^3: 3 consecutive successes
- Use for critical paths (auth, payments)
See reference for calculation formula and worked examples.
<grader_selection>
- Code-based (preferred): grep, test suite, build, type-check — deterministic, fast.
- Model-based: for open-ended outputs that can't be checked deterministically. Run multiple times, take majority.
- Human: required for security-sensitive changes, UX evaluation, legal/compliance.
Before trusting a model grader, compare its pass and fail decisions with labels from a human subject-matter expert. Tune it on separate examples, then measure its ability to find failures and recognize passes on a held-out set. Report both rates; overall accuracy can hide a grader that labels every case as passing. Recheck after changing the grader.
See reference for full grader templates and examples. </grader_selection>
<eval_validity> Before trusting a score:
- Isolate trials: each starts from a clean environment (fresh checkout/fixture). Leftover state or git history from earlier trials correlates failures or inflates passes.
- Reference solution: every task has a known-good output that passes all graders; proves the task is solvable and the grader is wired right.
- Near-zero pass rate = suspect the task or grader first, not the agent.
- Read transcripts and grades from many trials; separates real agent errors from graders rejecting valid solutions.
- Grade the outcome produced, not the path taken; give partial credit on multi-part tasks.
- 100% pass tracks regressions only; add harder tasks once saturated. Source: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents </eval_validity>
<anti_patterns> Writing evals after implementation tests existing bugs, not requirements. Model-based grading is slow and probabilistic. Prefer code graders. Every change must pass regression evals. No exceptions. Evals that take > 30s get skipped. Keep them fast. Track pass@k over time. Declining reliability is a signal. For any user-facing or high-stakes eval, the implementing agent scoring its own work inflates results ~36%. Spawn blind-evaluator instead. Evaluating against a codebase that already contains the canonical solution = answer key in the eval set. Use pre-merge snapshots or a separate fixture. Greenfield tickets in the golden eval set collapse to self=10, blind=3. Greenfields are not evaluable as solved tasks — exclude them from the dataset. Optimizing how much context the evaluator gets before establishing a baseline score = can't distinguish signal from noise. Run minimal-context baseline first, then test additions one at a time. </anti_patterns>
<on_complete> agentdb write-end '{"skill":"eval","eval_type":"capability|regression","pass_at_1":"<X%>","pass_at_3":"<Y%>","failures":[""]}'
Record eval type, pass rates, and any failures for future reference. </on_complete>
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |