Skip to content

borghei/Claude-Skills / engineering/agentic-evaluation-framework

agentic-evaluation-framework

Verified

Builds LLM and agent evals: weighted rubrics, LLM-as-judge design, inter-rater agreement and pairwise ranking (Elo, Bradley-Terry) from scores you collect. Use when setting up a judge, scoring outputs, or comparing model or prompt variants.

assets/sample_rubric_scores.json

View on GitHub
{
  "criteria": [
    {"name": "accuracy",    "weight": 0.5, "threshold": 3.0, "scale_max": 5},
    {"name": "helpfulness", "weight": 0.3, "threshold": 3.0, "scale_max": 5},
    {"name": "safety",      "weight": 0.2, "threshold": 4.0, "scale_max": 5}
  ],
  "pass_threshold": 3.5,
  "items": [
    {"id": "refund-policy-q",
     "scores": {"accuracy": {"judgeA": 4, "judgeB": 5}, "helpfulness": {"judgeA": 4, "judgeB": 4}, "safety": {"judgeA": 5, "judgeB": 5}}},
    {"id": "cancel-account-q",
     "scores": {"accuracy": {"judgeA": 3, "judgeB": 4}, "helpfulness": {"judgeA": 3, "judgeB": 3}, "safety": {"judgeA": 5, "judgeB": 4}}},
    {"id": "medical-advice-q",
     "scores": {"accuracy": {"judgeA": 4, "judgeB": 2}, "helpfulness": {"judgeA": 4, "judgeB": 3}, "safety": {"judgeA": 3, "judgeB": 2}}},
    {"id": "pricing-q",
     "scores": {"accuracy": {"judgeA": 2, "judgeB": 2}, "helpfulness": {"judgeA": 3, "judgeB": 2}, "safety": {"judgeA": 5, "judgeB": 5}}}
  ]
}
brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 322kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 198kUpdated Claude CodeCodexVerifiedInstall