sickn33/agentic-awesome-skills / plugins/aasb-aas-agent-mcp-builder/skills/llm-evaluation
llm-evaluation
VerifiedMaster comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 15.6 KB |
| references/benchmarking.md | markdown | 2.1 KB |
references/benchmarking.md
View on GitHub# Benchmark runner example
Supply the project's actual model and metric adapters and an already authorized, sanitized, materialized dataset. The pure control flow below does not supply inference, retrieval, external judges or a dataset. Bind tool versions, held-out coverage, hosted data/cost approval and a total run deadline before connecting a real model. Exceptions are incomplete runs, never a passed benchmark; retain the failed case and repair/re-run before publishing aggregate claims.
```python
import math
import numpy as np
class BenchmarkRunner:
def __init__(self, benchmark_dataset, max_examples=1000):
if not isinstance(max_examples, int) or max_examples < 1:
raise ValueError("Positive example limit required")
if not benchmark_dataset or len(benchmark_dataset) > max_examples:
raise ValueError("Require a nonempty bounded materialized dataset")
self.dataset = benchmark_dataset
def run_benchmark(self, model, metrics):
names = [metric.name for metric in metrics]
if not names or len(set(names)) != len(names):
raise ValueError("Require distinct declared metrics")
results = {name: [] for name in names}
for example in self.dataset:
prediction = model.predict(example["input"])
for metric in metrics:
score = metric.calculate(prediction=prediction,
reference=example["reference"], context=example.get("context"))
if not isinstance(score, (int, float)) or not math.isfinite(score):
raise ValueError("Missing or nonfinite metric result")
results[metric.name].append(score)
return {name: {"mean": np.mean(scores), "std": np.std(scores),
"min": min(scores), "max": max(scores)}
for name, scores in results.items()}
```
Report dataset/model/metric identity, actual coverage, failed cases, observation count and uncertainty alongside these descriptive statistics. A successful local fake-adapter example is not a real model benchmark, and these aggregates do not establish factual accuracy, safety or legal compliance.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |