pedrohcgs/claude-code-my-workflow / .claude/skills/vaccinate
vaccinate
VerifiedQualify a check before it is allowed to clear anything — prove it can detect the failure it is meant to catch. Seeds known defects into a copy of a real artifact plus a clean control, runs the checker, and reports recall and false-positive rate into a qualification ledger. Use when the user says "does this check work", "qualify the gate", "test my reviewer", "seed defects", "vaccinate", "qualify the checks", "can I trust this review", "does it pass for the right reason", or before relying on any automated check or referee simulation for a decision that matters. NOT a code fixer and NOT a reviewer itself — it grades the grader.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 8.4 KB |
| evals/README.md | markdown | 5.9 KB |
| evals/cases/README.md | markdown | 1.6 KB |
| evals/cases/detects-stale-gate.md | markdown | 442 B |
| evals/cases/nottrigger-simple-question.md | markdown | 186 B |
| evals/cases/requires-clean-control.md | markdown | 277 B |
| references/defect-library.md | markdown | 4.2 KB |
evals/README.md
View on GitHub# Skill evals — the gap between *verified* and *measured*
The gates in `./scripts/backtest.sh` prove the repo is **internally consistent and
currently true**. They cannot prove that a skill's instructions actually produce good output.
That is a different claim and it needs a different instrument.
**Two things to measure, separately.** Seeing a skill trigger tells you Claude *found* it, not
that it *did what you intended*:
| Question | Instrument |
|---|---|
| Does Claude invoke this skill on the prompts it should — and not on the ones it shouldn't? | description-trigger evals |
| When it does run, is the output what you wanted? | output-quality evals |
## The method: a baseline comparison
Collect a few realistic prompts. Run each in a **fresh session** with the skill available, and
again with it disabled. Compare. A fresh session matters because leftover context from
authoring the skill masks gaps in the written instructions — you will believe the skill says
something it only implied.
If the with-skill run is not better, the skill is costing context for nothing.
## Running them
**The shipped harness is primary.** From a normal shell (NOT inside a Claude Code session —
nested headless calls hang), at the repo root:
```bash
./scripts/run-skill-eval.sh <skill> .claude/skills/<skill>/evals/cases [--replicates N]
```
It runs the behavioral manipulation check first (each arm must *prove* skill access by
retrieving a marker phrase — set per skill in `evals/marker.txt`), then grades each case in
both arms, N=3 replicates by default, with a variance gate on **both** arms and every headless
call sandboxed (no Write/Edit/Bash).
The `skill-creator` plugin is an optional alternative that automates a richer loop:
```
/plugin marketplace add anthropics/claude-plugins-official
/plugin install skill-creator@claude-plugins-official
```
Then ask it to evaluate a skill. It stores cases in `evals/evals.json`, spawns a **subagent per
case** so each starts clean, records tokens and duration, grades assertions into
`grading.json`, and aggregates with-skill vs without-skill into `benchmark.json` — so you can
weigh the pass-rate improvement against the token and time cost.
It also does the two things hardest to do by hand: a **blind A/B between two versions** of a
skill, so you can confirm an edit is an improvement before committing it, and
**description tuning** — generating should-trigger and should-not-trigger prompts, measuring
the hit rate, and proposing description edits when the skill fires on the wrong requests.
## Which skills to evaluate first
Rank by *cost of being wrong*, not by how often they run:
1. **`/review-paper --peer`** — informs submission decisions.
2. **`claim-verifier`** — HIGH-WARN fails the verification closed.
3. **`/audit-reproducibility`** — gates the replication package.
4. **`/challenge`** — its output becomes a robustness claim in a paper.
## The relationship to `/vaccinate`
They are complementary and neither substitutes for the other:
- **`/vaccinate`** asks *can this checker detect a defect I planted?* — a **recall** question
about a checker.
- **Evals** ask *does this skill produce better work than not having it?* — a **quality**
question about an instruction set.
A skill can pass evals and still be a useless reviewer: it produces well-formed, plausible
findings that miss real defects. Only `/vaccinate` catches that. And a checker can vaccinate
cleanly while its skill wrapper triggers on the wrong prompts. Run both.
## The harness is itself a check, and must be qualified
Everything in this repo that clears work has to prove it can detect a failure. **That includes
the eval harness.** Three defects were found in its first version by running it, not by reading
it:
| Defect | Symptom | Fix |
|---|---|---|
| **N=1 replicate** | the same case scored 3/3 in one run and 1/3 ten minutes later — opposite conclusions from sampling noise | default `--replicates 3`, and a **variance gate** that refuses to report when per-case sd > 0.5 |
| **`nottrigger` written backwards** | asserted a substring must be *present*, so it scored 1/1 whatever happened and measured nothing | `nottrigger-*` cases now **invert**: the assertion must be **absent** |
| **single-substring grading** | a correct answer phrased differently scored as a miss | assertions list **alternatives** (`a \| b \| c`); any one counts |
**Assertions are literal substrings on purpose.** A model grading a model reintroduces exactly
the failure being measured. Alternatives make that tractable without importing a judge.
> **Do not record a ledger row from a high-variance run.** The harness exits 3 and says so.
> A number that changes between runs is not a measurement, and writing it down as one is worse
> than having no number — it looks like evidence.
## Cost
Each case costs `2 x replicates` headless runs (with-skill and without-skill), at roughly
60–90 s each. Nine cases at N=3 is ~54 runs, well over an hour. Budget accordingly, and **do
not mechanically generate cases to raise coverage** — 180 cases that measure nothing are worse
than nine that measure something, because the ledger then carries green rows that mean nothing.
## Recording results
Eval results go in the same ledger as qualification runs
(`quality_reports/qualification/LEDGER.md`), with the skill name, the case count, pass rate
with and without, and the token delta. **An eval with no recorded baseline is an anecdote.**
> **Status (2026-08-22).** The harness is qualified (negative control delta 0; positive
> control delta +2; behavioral manipulation check), and the five priority skills have
> recorded results in `quality_reports/qualification/LEDGER.md` — including one composite the
> harness withheld for a noisy baseline arm, and two near-zero deltas recorded as-is. The
> measured benefit concentrates in distinctive doctrine; recall-style cases the baseline
> already answers measure ~0, so the next iteration is procedure-sensitive cases, not more
> recall cases.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |