Skip to content

JPeetz/agent-skills / skills/ai-agent-engineering/agentic-eval

agentic-eval

Verified

Use this skill when setting up or improving evaluation for AI agents and LLM outputs. Activates on "agent eval", "LLM evaluation", "eval harness", "agent benchmark", "rubric judge", "evaluator-optimizer", "self-critique", "LLM as judge", "pairwise rank", "eval pipeline", "calibration test", "agentic eval", or "eval-driven development". Covers rubric design, evaluator-optimizer architecture, self-critique loops, pairwise ranking, calibration, eval harness setup, and continuous eval pipelines.

FileKindSize
CHANGELOG.mdmarkdown1.8 KB
LICENSEother1.0 KB
SKILL.mdskill md31.1 KB
evals/evals.jsonconfig8.3 KB
references/sources.mdmarkdown8.3 KB

CHANGELOG.md

View on GitHub
# Changelog

## [1.0.0] - 2026-11-14

### Added
- Initial release: agentic-eval skill under the new `ai-agent-engineering` category
- Eval pyramid (unit → component → integration → system → human) with guidance on testing at the lowest level that can catch the failure
- Eval-driven development loop: real cases, written criteria, saved baseline, one variable per run, per-case diffs, human reads, frozen regressions
- Rubric design: independent dimensions, anchored score levels, small scales, evidence-quoting prompts, chain-of-thought before the label
- Evaluator-optimizer architecture with bounded iterations, fresh-context re-scoring, best-artifact retention, and cost-ratio guidance
- Self-critique loops covering what they reliably catch, what they miss (confident factual errors, shared blind spots, sycophantic revision), and role separation
- Pairwise ranking with position-bias controls: both-order agreement, randomized order, blinded labels, legitimate ties; win rate, Bradley–Terry, Elo, and sample-size caveats
- Calibration: human-labeled sets, human–human ceiling, Cohen's κ / weighted κ / Spearman ρ, disagreement inspection, judge-swap invalidation
- Judge bias probes for verbosity, self-preference, position, format, and confidence/authority bias
- Overfitting defenses: holdout splits, production refresh, trajectory metrics, case rotation
- Eval harness setup: JSONL dataset schema, pinned configurations, caching, failure isolation, deterministic gate before the judge, per-case reporting, CI tiering with regression gates
- Anti-patterns table, red flags, and a pre-trust verification checklist
- Eval cases: four positive triggers across rubric, harness, pairwise bias, and evaluator-optimizer; two near-miss negatives
- `references/sources.md` with upstream skill, primary papers, and vendor documentation
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall