agentsope/SkillAlchemy / skills/agentsop-domain-eval-set
agentsop-domain-eval-set
VerifiedBuild and govern a 50-200 example domain-specific held-out benchmark sampled from real traffic. Distinct from public benchmarks (MMLU/HumanEval/GSM8K via lm-evaluation-harness) which measure GENERAL capability. Only a held-out domain set predicts whether THIS system works on YOUR data. Collect real examples, label, hold out (never train/prompt on it), size 50-200, version it, refresh on drift.
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 2.7 KB |
| SKILL.md | skill md | 24.8 KB |
| intermediate/operation_candidates.json | config | 5.9 KB |
| references/R1-source-evidence.md | markdown | 4.9 KB |
references/R1-source-evidence.md
View on GitHub# R1 — Source Evidence for `domain-eval-set`
Verbatim claims and their sources. This overlay distills the **held-out domain
benchmark** idea from two source skills and positions it explicitly against the
**public-benchmark** skill (`lm-evaluation-harness`).
---
## 1. DSPy — train/dev/test discipline, held-out distinct from val
Source: `/Users/5imp1ex/Desktop/Skill-Workplace/output/dspy-sop-skill/SKILL.md`
- **Held-out test set is the exit gate, not the val set:**
> "compiled program beats baseline on a *held-out* test set (not the val set used in optimization) by ≥ task-relevant delta."
— DSPy SOP Stage 3 exit criterion [dspy.ai/learn/optimization/overview/]
- **Dev-set sizing sweet spot:**
> "Documented sweet spot: 30 examples = minimum useful, 300 = recommended, 200+ required for MIPROv2 to avoid overfitting."
— DSPy SOP Stage 2 step 5 [dspy.ai/learn/optimization/overview/]
- **Held-out must be distinct from val when escalating compute:**
> "Only escalate to `heavy` if data ≥ 300 *and* you have a held-out test set distinct from val."
— DSPy Case A step 4 [dspy.ai/learn/optimization/optimizers/]
- **Metric must be human-validated before trust (the labeling/curation analogue):**
> "Never compile against a metric you haven't human-validated on ≥ 20 spot-checks."
— DSPy Case C [dspy.ai/learn/evaluation/metrics/]
- **Below ~30 you memorize, not measure:**
> "Below ~30 examples, you're not training — you're memorizing."
— DSPy anti-pattern 2 [dspy.ai/learn/optimization/overview/]
- **Artifacts are versioned and shipped, not free-floating:**
> "You ship `program.json`, not a `.txt` prompt."
— DSPy core mental model [dspy.ai/tutorials/saving/]
---
## 2. LlamaIndex — eval set built from the corpus, gated on every change
Source: `/Users/5imp1ex/Desktop/Skill-Workplace/output/llamaindex-sop-skill/SKILL.md`
- **Build the eval loop BEFORE optimizing — from the real corpus:**
> "Build the eval loop **before** optimizing anything … `DatasetGenerator.from_documents(docs).generate_dataset_from_nodes(num=50)`."
— LlamaIndex SOP Stage 2. Note `num=50` default — the low end of the 50–200 band.
- **Every change gates on the eval numbers:**
> "Every subsequent change must be gated on these numbers."
— LlamaIndex SOP Stage 2
- **Labeled gold passages are required for retrieval metrics:**
> "`RetrieverEvaluator(["mrr","hit_rate"])` … does retrieval pull the gold passage."
— LlamaIndex OP-10 EvalLoop [developers.llamaindex.ai/python/framework-api-reference/evaluation/]
- **Eval surfaces failures anecdote cannot:**
> "No eval loop; debug by anecdote" is anti-pattern A3; correct move is to stand up the evaluators first.
— LlamaIndex anti-patterns table
- **Versioned artifacts + reconcile on drift:**
> "Indices versioned as deployment artifacts" (Stage 5); live corpus must be reconciled via `IngestionPipeline … UPSERTS_AND_DELETE` (A10).
— LlamaIndex SOP Stage 5 / anti-pattern A10. This is the data-drift analogue for an eval set going stale.
### Caveat — generated sets are a starting point, not a held-out
LlamaIndex `DatasetGenerator` produces *synthetic* QA pairs from the corpus.
These are excellent for bootstrapping a dev set, but a model can look good on
auto-generated questions while failing real user phrasing. A held-out **domain**
set is sampled from *real traffic* and labeled by humans — generated sets do not
replace it. (See §7 cross-framework mapping in SKILL.md.)
---
## 3. lm-evaluation-harness — the PUBLIC-bench skill to distinguish from
Source: `~/.claude/skills/lm-evaluation-harness/SKILL.md` (frontmatter)
- **Scope is public academic benchmarks:**
> "Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress."
— `evaluating-llms-harness` description, v1.0.0
- **Invocation is over public task names, not your data:**
> `lm_eval --model hf --model_args pretrained=... --tasks mmlu,gsm8k,hellaswag`
— Quick start
**Distinction (the core of this overlay):** lm-evaluation-harness measures
*general capability* on shared, public, often-contaminated benchmarks. It says
nothing about whether the system answers *your* customers' questions on *your*
data. The two are complementary axes, not substitutes — see anti-pattern AP-1
and OP-DE08 in SKILL.md.
---
## 4. Cross-links
- `[[lm-evaluation-harness]]` — public benchmark runner; the capability-floor axis.
- `[[agentsop-regression-gate]]` — consumes this set as the per-PR gate; this skill produces the artifact, regression-gate enforces it on every change.
- `[[agentsop-metric-design]]` — defines *how* each held-out example is scored (decomposed sub-judges, bool/float, calibration). This skill defines *what* is in the set; metric-design defines the scoring function applied to it.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |