katopz/katgpt-rs / .agents/skills/calibration-gate
calibration-gate
VerifiedThe evidence-gated calibration loop for any tunable surface - drive weights, need sigmoids, gate rho, thresholds, dispatch narrowing, eval bars, UQ calibrators, rig-posture cells. Verdict-first authority, per-group evidence census, pre-registered bars and falsifiers, hardened queue cycles that die loud in pre-flight, separate calibration and production cells, box-state provenance on every number, per-group classification with honest NO VERDICT. Use BEFORE planning or running any calibration or tuning sweep, when a calibration run keeps dying pre-run, when pinning perf bars from a calibration run, when adjudicating Calibrated vs Sparse vs Fallback, or when asked how a calibration went. Sibling to build-hygiene (pre-registered A/B bars) and grammar-not-answers (contamination gates).
calibration-gate — the evidence-gated calibration loop
The loop the fleet already runs when it tunes anything measurable. It is written down here
because every lane re-derived it once — this file is the next lane's head start. The
canonical walk-through is ../seal-sprints/.plans/029_shopping_chain_useitem_and_shopper_chaos.md
(the shopping-drive calibration, 2026-11-13→15): read its Status line as the worked example —
delegated verdict round 1 (AGREE + binding amendments A1–A4), four machinery-review rounds
(REVISE ×3 → AGREE round 4), then calibration queue cycles v1/v2/v3 where each death bought a
permanent pre-flight defense. The precedents at the bottom name one lane per step.
The loop is a process representation, not a recipe: every area plugs in its own evidence classes, groups, and bar vocabulary. The skill removes the re-derivation, not the judgment.
When this fires
- BEFORE planning or running any calibration, tuning sweep, A/B, or bar-pinning run.
- When a calibration run keeps dying pre-run — that is the loop working; each death buys a permanent defense, never a weaker guard.
- When someone asks "how did calibration go" — answer in this loop's vocabulary (authority → census → pre-registration → cycles → classification → pinned bars).
- When adjudicating a result — Calibrated / Fallback / Sparse / NO VERDICT are the only verdicts.
The loop
-
Authority before runs. Plan-first for any calibration that touches content or spends rig time: file the plan, claim it (
**Claimed:** <session>, <epoch>, expires <ISO-UTC>— ONE claim line per file; a malformed line — prose after the expiry token, a millisecond epoch, a +07:00 offset — blinds the lease checker for EVERY lane), and get the delegated verdict (Claude ping-pong, proposal 001; max rounds peragent.verdict_max_rounds) BEFORE code or runs. Record binding amendments BY NAME (A1–A4 style) and carry them forward — verdict rounds revise the plan, never silently widen it. Machinery reviews are round-numbered in the plan so a successor thread picks up at the right round. -
Census the evidence before tuning anything. Count the labeled evidence PER GROUP before proposing values — pin RAW counts as data and derive the verdict, so a floor change never re-mines. Precedents — riir-refine i233 pinned per-(domain × shape) RAW census counts (180 labeled spans) before any narrowing; katgpt-rs bench 982 printed its label census (315 fail / 320 requests) and REFUSED to adjudicate when one class was degenerate. If a group has no evidence, it has no verdict — an empty window ABSTAINS (the adoption-gate posture), it never invents.
-
Pre-register the decision rule before pair 1. Bars, thresholds, floors, falsifiers, promotion bar, stop rule — written and committed BEFORE the run (the riir-ai coverage_seek A/B pre-registration — issue 1095,
d45e061bd— bound the promotion bar- stop rule before pair 1; riir-infer plan 633 landed every gate explicit — G1 FAIL +1.46 CER, G2 FAIL 0.57× vs the CPU lane, G4 PASS). A post-hoc bar is a tune-to-pass, and a tune-to-pass measures the tuner. Include up front: the small-group rule (e.g. n < 20 → Sparse; the 30/30 two-class label-census floor), the permutation-null or canary falsifier that proves the gate CAN fail, and the demote literal (the env-switch posture several repos pin — e.g. riir-infer / riir-refine / riir-rethink: a promoted arm's env is DEMOTE-ONLY, an opt-in arm's PROMOTE-ONLY — spell yours now).
-
Separate the calibration cell from the production cell. Bars are pinned FROM the calibration run; the production cell (sprint#029's example: N=100, real contention, don't go easy) then runs UNDER those bars — never the reverse (sprint#029 tasks 6/8; bars are never read off the calibration load). A green production run that never stressed the thing it claims to stress is a SKIP, not a pass. And a calibration law discovered in the calibration cell PROPAGATES to the production cell BY NAME (sprint#029's example: fresh press + money-only pre-mint + account key are load-bearing for purchases — a reused old vessel SKIPs on zero purchases by construction).
-
Queue cycles die loud, die cheap — and each death lands a defense. Expect the first queue cycles to die in pre-flight (v1 died-by-design on a missing pin; v2's vessel dump revealed three fatal gaps — empty shop/merchant books, penniless bots, a throwaway-key signature; v3 carried the full corrected sequence IN the job). The law: every pre-flight catch becomes a PERMANENT defense in code (boot strips, funded wallets, empty-book pre-flight refusals that name the fix), never a one-off patch on a job script. Never weaken a guard to get a green. Press fresh whatever the verdict depends on (content, signing key) — a stale artifact's green proves the stale artifact.
-
Measure with provenance and stability. Every quoted bar carries its
bench_preflightPROVENANCE line (box state, power source, load class). A/B pairs are interleaved and two-run stable (katgpt-rs/scripts/sequential_ab_timing_audit.py— two sequential arms measure the scheduler, +5–22%); paired non-inferiority (the riir-rethink bar δ = max(1.0pp, 2.5·SE) is one house example — carry your lane's own). Queue long cells viaagent-mail rig run --exclusive --timeout <bound>on a verified-quiet rig, the estimate named in--title/the plan (the owner's tiered-queue decree, 2026-11-12, wants per-job estimates — pass--estonly where the rig's agent-mail build supports it; a build without it refuses the invocation at launch, verified on m3 2026-11-15). Never bench a BUSY box, never bench beside a GPU consumer, and re-run any row a contender touched ALONE before quoting it. -
Classify per group, never pool; withhold on absent evidence. The verdict vocabulary is the calibrated_dispatch shape — Calibrated (Wilson 95% LCB ≥ τ over ≥ N labeled — riir-refine i233's values were τ 0.6 over n ≥ 20; pin yours), Fallback (measured under-floor → demote the lever), Sparse (n < N → withhold; narrowing stays OFF — the evidence DEMOTES the unproven lever, it never hopes). Conservative per-bit: one non-Calibrated bit falls the whole surface back to the full posture (the dispatch narrows, never widens). An undecidable cell prints NO VERDICT with every envelope term named (a distinct non-zero exit where the lane defines one — seal-remake i201's T2 verdict machinery uses exit 3) — never fold it into a pass; count unknowns loud; pool nothing across groups, ever (the per-predicate law).
-
Pin, inherit, demote, record. Pin the bars beside their PROVENANCE; carry the inherited laws into the next cell's plan (named, not folklore); demote the losing arm (never delete it); record verdict + re-open conditions in the plan/issue so the next lane starts at step 4. A negative calibration is a RESULT — riir-infer i1216's head-surface PoC died at its own T0 headroom gate (file removed; T0 record
ca9e7fcb, bench 1217); riir-train plan 498's QAD head collapsed; riir-refine i237's composition cell NO-GO (file removed505dffd5; recordaf5dd5f3, bench 151) — the record IS the yield.
The orchestration half — watched, not babysat
Long calibrations are park-and-notify: agent-mail rig run --exclusive --timeout <bound>, the estimate named in --title/the plan; END the turn; mail arrives
at start/poison/end. Continuity across context overflow is the plan's Status line — a
successor thread reads the plan and the out-dirs, never re-derives; fold superseded claims
into **Lease closed:** rows (the lint's exact spelling) so exactly one Claimed line
remains. Watchdog pickups poll on their cadence; a backstop wake is
agent-mail send --to <own-inbox>@<rig> --at <time> "check <job>" then END the turn.
Verdict pickups answer the verdict round in the SAME session, not a new thread. Rig log
purges completed jobs — check the WORK DIR before declaring a job vanished (the duel
d241af5d + amo 20979a65 near-mistakes, 2026-11-15). Housekeeping lanes verify their own
windows only; they never re-run or re-adjudicate a live lease.
Never rules
- Never tune to pass; a bar written after the run is not a bar.
- Never pool group verdicts; never widen on Sparse evidence.
- Never quote a number without its box state; never bench a BUSY rig.
- Never re-pin a floor from a dirty, poisoned, or stale-posture run — disclose, re-run clean.
- Never gate correctness behind the thing being calibrated (lossy surfaces gate on per-family conditional retention, never aggregates).
- Sigmoid, never softmax; decline and abstain are typed answers, not guesses.
- A calibration that cannot fail is not a calibration — arm the canary, prove it fires.
Precedents (one lane per step)
- sprint#029 shopping-drive calibration (
../seal-sprints/.plans/029_...) — the full walk-through: verdict rounds, v1/v2/v3 pre-flight deaths, the inherited-laws handoff. - riir-refine i233
calibrated_dispatch— census → Wilson LCB → Calibrated/Sparse/Fallback, conservative per-bit narrowing, the re-run census instrument. - katgpt-rs bench 982 (i1210 C2) — the census-floor NO VERDICT + permutation-null falsifier, re-open conditions recorded.
- riir-rethink ESC ρ gates (issues 052/054/064 — files removed; 064's landing
1a45ae6, close-out65e32b8; the live posture is the ESC block in riir-rethink AGENTS.md) — per-suite gate-rho tables, rate-targeted fit, collapse caveats on the verdict. - seal-remake sprint!053 — drive weights past the sigmoid ceiling → the zeroing form; verdicts as lane-posture pins with an armed witness row.
- riir-train plan 497 — the LoRA trial-shift gain re-attributed from an instrumentation
artifact (+700% claimed, landing commits
5a082a21/7568e6df) to +131.5% measured (per-dispatch shader compiles + blocking readbacks), paired harness as the record. - riir-infer plan 633 — every gate explicit and honest, negatives quoted beside positives.
- sprint#029 tasks 6/8 — calibration cell separate from the chaos cell, SKIP-not-pass, PROVENANCE beside every bar.
Instruments
agent-mail rig status/agent-mail rig run --exclusive --timeout— queue + quiet-gate; estimates ride--titleuntil the rig's build carries the decree's--est.riir-reflex/scripts/bench_preflight.sh— the PROVENANCE line (power, load, box state).katgpt-rs/scripts/sequential_ab_timing_audit.py— the interleaved-A/B law.- Wilson/Beta LCB precedents — katgpt-core
beta_lcb_order_into, riir-instinctsrc/stats.rs. katgpt-rs/scripts/claim_lease_check.py --lint <file>— the claim-line grammar, lint first.- seal-e2e lanes —
seal-e2e-multi§2/§4 owns the rig queue posture and the stress axes.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| planning-and-task-breakdownaddyosmani/agent-skillsBreaks work into ordered tasks. Use when you have a spec or clear requirements and need to break work into implementable tasks. Use when a task feels too large to start, when you need to estimate scope, or when parallel work is possible. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| screenpipethedotmack/claude-memSearch screen and audio history recorded by Screenpipe and optionally save selected context into claude-mem. Use when the user asks to recall desktop activity, a meeting, or something seen in another app through Screenpipe. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| contact-widgetnexu-io/open-designSelf-contained floating chat widget with welcome screen, social links, meeting button, and message input. Single HTML file, zero dependencies. | Stars: 108k | Updated | Claude CodeCodex | Verified | Install |
| meeting-notesnexu-io/open-designMeeting notes page — title bar with attendees, agenda checklist, decisions block, action items table with owners + dates, and a "next meeting" footer. Use when the brief mentions "meeting notes", "minutes", "1:1 notes", "all-hands recap", or "会议纪要". | Stars: 108k | Updated | Claude CodeCodex | Verified | Install |