agentsope/SkillAlchemy / skills/agentsop-domain-eval-set
agentsop-domain-eval-set
VerifiedBuild and govern a 50-200 example domain-specific held-out benchmark sampled from real traffic. Distinct from public benchmarks (MMLU/HumanEval/GSM8K via lm-evaluation-harness) which measure GENERAL capability. Only a held-out domain set predicts whether THIS system works on YOUR data. Collect real examples, label, hold out (never train/prompt on it), size 50-200, version it, refresh on drift.
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 2.7 KB |
| SKILL.md | skill md | 24.8 KB |
| intermediate/operation_candidates.json | config | 5.9 KB |
| references/R1-source-evidence.md | markdown | 4.9 KB |
README.md
View on GitHub# domain-eval-set skill
An **ENHANCE overlay** coder-agent skill for building and governing a **50–200
example domain-specific held-out benchmark** — sampled from real traffic,
human-labeled, sealed, versioned.
**Version**: 0.1.0
**Phase**: D · **Tier**: core · **Status**: opinionated · **Overlay**: ENHANCE
## The one idea
> Public benchmarks (MMLU, HumanEval, GSM8K) measure **general capability**.
> A 50–200 example held-out **domain** set measures **YOUR task**.
> Only the latter predicts production.
`[[lm-evaluation-harness]]` covers the public-benchmark axis. This skill covers the
domain held-out axis — the number you actually gate deployment on. They are
complementary, not substitutes.
## What this skill answers
- How to **source** eval examples from real traffic (stratified, edge-case-heavy)
instead of trusting a demo or an MMLU score.
- How to **label and curate** with inter-annotator agreement on the hard cases.
- **Hold-out discipline**: the test split is sealed — never to the optimizer,
never as a few-shot demo, never to pick chunk size / model, never in fine-tune
data.
- **Sizing**: why 50–200 (and why <30 is noise).
- **Versioning + leak audit**: hash/date/rubric the set; diff it against demos and
training data so the number isn't contaminated.
- **Refresh**: how an eval set goes stale as the domain drifts ("green but on
fire") and how to refresh + re-version.
- **When NOT to use it**: pure capability comparison → `[[lm-evaluation-harness]]`;
objective oracle exists → just run the oracle.
## Files
- `SKILL.md` — main skill (7 sections: 何时激活 / 核心心智模型 / SOP / 操作模型 /
困境决策案例 / 反模式与边界 / 跨框架对照)
- `references/R1-source-evidence.md` — verbatim source quotes per claim
- `intermediate/operation_candidates.json` — 8 operations (OP-DE01…OP-DE08) in
Trigger / Action / Output / Evidence form
## Cross-links
- `[[lm-evaluation-harness]]` — public-benchmark runner; the capability-floor axis.
- `[[agentsop-regression-gate]]` — consumes this set as the per-PR gate (produce vs enforce).
- `[[agentsop-metric-design]]` — defines the scoring function applied to each held-out
example (decomposed, calibrated, bool/float).
## Source basis
- DSPy SOP — train/dev/test discipline, held-out distinct from val, sizing floor
(`output/dspy-sop-skill/SKILL.md`) [dspy.ai/learn/optimization/overview/]
- LlamaIndex SOP — eval loop built from the corpus, gated on every change,
versioned artifacts (`output/llamaindex-sop-skill/SKILL.md`)
[developers.llamaindex.ai/python/framework-api-reference/evaluation/]
- lm-evaluation-harness — the PUBLIC-benchmark skill this overlay distinguishes
itself from (`~/.claude/skills/lm-evaluation-harness/SKILL.md`)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |