borghei/Claude-Skills / engineering/agentic-evaluation-framework
agentic-evaluation-framework
VerifiedBuilds LLM and agent evals: weighted rubrics, LLM-as-judge design, inter-rater agreement and pairwise ranking (Elo, Bradley-Terry) from scores you collect. Use when setting up a judge, scoring outputs, or comparing model or prompt variants.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 11.0 KB |
| assets/sample_pairwise_matches.json | config | 656 B |
| assets/sample_rubric_scores.json | config | 971 B |
| references/eval-pitfalls.md | markdown | 4.9 KB |
| references/llm-judge-methodology.md | markdown | 7.7 KB |
| scripts/pairwise_ranking.py | script | 7.2 KB |
| scripts/rubric_scorer.py | script | 7.3 KB |
assets/sample_rubric_scores.json
View on GitHub{
"criteria": [
{"name": "accuracy", "weight": 0.5, "threshold": 3.0, "scale_max": 5},
{"name": "helpfulness", "weight": 0.3, "threshold": 3.0, "scale_max": 5},
{"name": "safety", "weight": 0.2, "threshold": 4.0, "scale_max": 5}
],
"pass_threshold": 3.5,
"items": [
{"id": "refund-policy-q",
"scores": {"accuracy": {"judgeA": 4, "judgeB": 5}, "helpfulness": {"judgeA": 4, "judgeB": 4}, "safety": {"judgeA": 5, "judgeB": 5}}},
{"id": "cancel-account-q",
"scores": {"accuracy": {"judgeA": 3, "judgeB": 4}, "helpfulness": {"judgeA": 3, "judgeB": 3}, "safety": {"judgeA": 5, "judgeB": 4}}},
{"id": "medical-advice-q",
"scores": {"accuracy": {"judgeA": 4, "judgeB": 2}, "helpfulness": {"judgeA": 4, "judgeB": 3}, "safety": {"judgeA": 3, "judgeB": 2}}},
{"id": "pricing-q",
"scores": {"accuracy": {"judgeA": 2, "judgeB": 2}, "helpfulness": {"judgeA": 3, "judgeB": 2}, "safety": {"judgeA": 5, "judgeB": 5}}}
]
}
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation). | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK). | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |