JPeetz/agent-skills / skills/ai-agent-engineering/agentic-eval
agentic-eval
VerifiedUse this skill when setting up or improving evaluation for AI agents and LLM outputs. Activates on "agent eval", "LLM evaluation", "eval harness", "agent benchmark", "rubric judge", "evaluator-optimizer", "self-critique", "LLM as judge", "pairwise rank", "eval pipeline", "calibration test", "agentic eval", or "eval-driven development". Covers rubric design, evaluator-optimizer architecture, self-critique loops, pairwise ranking, calibration, eval harness setup, and continuous eval pipelines.
| File | Kind | Size |
|---|---|---|
| CHANGELOG.md | markdown | 1.8 KB |
| LICENSE | other | 1.0 KB |
| SKILL.md | skill md | 31.1 KB |
| evals/evals.json | config | 8.3 KB |
| references/sources.md | markdown | 8.3 KB |
CHANGELOG.md
View on GitHub# Changelog
## [1.0.0] - 2026-11-14
### Added
- Initial release: agentic-eval skill under the new `ai-agent-engineering` category
- Eval pyramid (unit → component → integration → system → human) with guidance on testing at the lowest level that can catch the failure
- Eval-driven development loop: real cases, written criteria, saved baseline, one variable per run, per-case diffs, human reads, frozen regressions
- Rubric design: independent dimensions, anchored score levels, small scales, evidence-quoting prompts, chain-of-thought before the label
- Evaluator-optimizer architecture with bounded iterations, fresh-context re-scoring, best-artifact retention, and cost-ratio guidance
- Self-critique loops covering what they reliably catch, what they miss (confident factual errors, shared blind spots, sycophantic revision), and role separation
- Pairwise ranking with position-bias controls: both-order agreement, randomized order, blinded labels, legitimate ties; win rate, Bradley–Terry, Elo, and sample-size caveats
- Calibration: human-labeled sets, human–human ceiling, Cohen's κ / weighted κ / Spearman ρ, disagreement inspection, judge-swap invalidation
- Judge bias probes for verbosity, self-preference, position, format, and confidence/authority bias
- Overfitting defenses: holdout splits, production refresh, trajectory metrics, case rotation
- Eval harness setup: JSONL dataset schema, pinned configurations, caching, failure isolation, deterministic gate before the judge, per-case reporting, CI tiering with regression gates
- Anti-patterns table, red flags, and a pre-trust verification checklist
- Eval cases: four positive triggers across rubric, harness, pairwise bias, and evaluator-optimizer; two near-miss negatives
- `references/sources.md` with upstream skill, primary papers, and vendor documentation
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |