wshobson/agents / plugins/llm-finetuning/skills/eval-harness-first
eval-harness-first
VerifiedBuild the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 7.8 KB |
| references/grader-templates.md | markdown | 9.9 KB |
| references/judge-calibration.md | markdown | 6.5 KB |
references/judge-calibration.md
View on GitHubLast verified: 2026-07-14
# Judge Calibration Protocol
The full procedure behind `SKILL.md`'s "Judge
Calibration Is a Prerequisite" section. Any grader
routed to an LLM-judge follows this before its
verdicts count toward a pass rate or a checkpoint
promotion decision.
## N/A Path: All-Deterministic Harness
If error analysis produced zero buckets that route to
an LLM-judge — every grader is regex, schema, or
execution-based — this entire protocol is N/A for the
run, not an unsatisfiable checklist item. Phase 0
Exit Checklist item 4 in `SKILL.md` is satisfied by
stating this explicitly (e.g. "Judge calibration:
N/A — all N graded criteria are deterministic") rather
than leaving it blank or blocking Phase 0 completion
on a judge that was never going to exist. This is
common on a greenfield strict-schema or exact-match
task with synthetic goldens — don't invent a
subjective criterion just to have something to
calibrate.
## 1. Label
Collect human labels for **≥100 items** covering the
failure bucket the judge will grade. Use the same
labelers (or a labeling rubric tight enough to be
interchangeable) that produced the axial-coding
buckets in `SKILL.md`'s Building Goldens section —
a judge calibrated against a different notion of
"pass" than the one used to build goldens will
silently diverge from what the harness is supposed
to measure.
## 2. Split
Divide the labeled set three ways and keep the
splits separate for the whole calibration cycle:
| Split | Purpose | Size |
|---|---|---|
| train | Few-shot examples embedded in the judge prompt | ~20-30% |
| dev | Iterate the prompt, catch obvious misses | ~30-40% |
| sealed test | Report TPR/TNR once; never re-touch after | ~30-40% |
The sealed-test split is sealed: if a dev-split
iteration cycle causes the reported test-split
number to move, that number is no longer a valid
generalization estimate — re-seal a fresh test split
instead of re-running against the same one.
## 3. Compute TPR/TNR
Run the judge (with train-split few-shot examples in
the prompt) against the sealed test split and compute:
```python
def tpr_tnr(judge_verdicts: list[bool], human_labels: list[bool]) -> tuple[float, float]:
"""True Positive Rate (sensitivity) and True
Negative Rate (specificity) against human labels.
A single blended accuracy number hides which
direction the judge is biased toward — always
report both, never accuracy alone.
Raises ValueError on empty or mismatched-length
input, or if the sealed test split lacks either
class — a `zip()` over unequal lists silently
drops the extra items, which can produce
apparently valid metrics from a partial or
miscollected split.
"""
if not judge_verdicts or not human_labels:
raise ValueError("tpr_tnr: judge_verdicts and human_labels must be non-empty")
if len(judge_verdicts) != len(human_labels):
raise ValueError(
f"tpr_tnr: length mismatch ({len(judge_verdicts)} verdicts vs "
f"{len(human_labels)} labels) — check the split for a collection bug"
)
tp = sum(j and h for j, h in zip(judge_verdicts, human_labels))
fn = sum((not j) and h for j, h in zip(judge_verdicts, human_labels))
tn = sum((not j) and (not h) for j, h in zip(judge_verdicts, human_labels))
fp = sum(j and (not h) for j, h in zip(judge_verdicts, human_labels))
if (tp + fn) == 0:
raise ValueError("tpr_tnr: no positive-labeled items in this split — TPR undefined")
if (tn + fp) == 0:
raise ValueError("tpr_tnr: no negative-labeled items in this split — TNR undefined")
return tp / (tp + fn), tn / (tn + fp)
```
Agree on a TPR/TNR bar with whoever owns the eval
harness before running this step — a common starting
bar is ≥0.85 on both, tightened per bucket based on
how costly a false pass or false fail is downstream.
## 4. Bias-Correct Reported Rates
A judge with unequal TPR/TNR does not report the true
pass rate of the model under test — it reports a
rate skewed by its own asymmetric error pattern.
Correct the observed pass rate before publishing it:
```python
def bias_corrected_pass_rate(observed_pass_rate: float, tpr: float, tnr: float) -> float:
"""Rogan-Gladen style correction: recovers the
true pass rate from the judge's observed rate and
its TPR/TNR, rather than reporting the judge's raw
output as if it were ground truth.
"""
denom = tpr + tnr - 1
if denom <= 0:
raise ValueError("Judge is at or below chance — do not correct, recalibrate")
return (observed_pass_rate + tnr - 1) / denom
```
If `tpr + tnr - 1` is small (judge close to chance),
the correction blows up and becomes unreliable — that
is itself a signal the judge needs a prompt rewrite,
not a correction formula.
## 5. Pin the Snapshot
Record the exact judge model snapshot/version used
for calibration alongside the TPR/TNR numbers. An
unpinned judge (a "latest" alias that moves under
you) invalidates the calibration the moment the
underlying model changes — the TPR/TNR numbers stop
describing the judge actually in use.
## 6. Recalibrate
Two triggers, either one is sufficient:
- **Judge-model change** — any change to the pinned
snapshot, intentional or forced (deprecation).
- **Quarterly, regardless** — schedule recalibration
even with no judge change, since the underlying
task distribution (what "hard" cases look like)
drifts as the model under test improves and error
analysis surfaces new failure modes.
Recalibration reruns steps 1-4 in full — it is not a
partial refresh of just the test split.
## 7. Miscalibration Fallback
If the judge cannot hit the agreed TPR/TNR bar after
prompt iteration on the dev split:
- The judge ships **advisory-only**: its verdicts
surface in review tooling for a human to consider,
but do not gate a checkpoint promotion and are not
counted into a reported pass rate.
- Do not lower the TPR/TNR bar to make an
uncalibrated judge "pass" — that defeats the point
of calibrating in the first place.
- Prefer routing the bucket to a deterministic grader
if the criterion can be reframed as one (see
`SKILL.md`'s Graders section) over shipping a
permanently advisory-only judge.
## Different Model Family, Always
The judge model must come from a different model
family than the model under test, for every
calibration cycle and every recalibration — a judge
evaluating outputs from its own family is a biased
grader, and this cannot be corrected away with the
TPR/TNR formula above since the bias is systematic
rather than random.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |