Skip to content

katopz/katgpt-rs / .agents/skills/calibration-gate

calibration-gate

Verified

The evidence-gated calibration loop for any tunable surface - drive weights, need sigmoids, gate rho, thresholds, dispatch narrowing, eval bars, UQ calibrators, rig-posture cells. Verdict-first authority, per-group evidence census, pre-registered bars and falsifiers, hardened queue cycles that die loud in pre-flight, separate calibration and production cells, box-state provenance on every number, per-group classification with honest NO VERDICT. Use BEFORE planning or running any calibration or tuning sweep, when a calibration run keeps dying pre-run, when pinning perf bars from a calibration run, when adjudicating Calibrated vs Sparse vs Fallback, or when asked how a calibration went. Sibling to build-hygiene (pre-registered A/B bars) and grammar-not-answers (contamination gates).

calibration-gate — the evidence-gated calibration loop

The loop the fleet already runs when it tunes anything measurable. It is written down here because every lane re-derived it once — this file is the next lane's head start. The canonical walk-through is ../seal-sprints/.plans/029_shopping_chain_useitem_and_shopper_chaos.md (the shopping-drive calibration, 2026-11-13→15): read its Status line as the worked example — delegated verdict round 1 (AGREE + binding amendments A1–A4), four machinery-review rounds (REVISE ×3 → AGREE round 4), then calibration queue cycles v1/v2/v3 where each death bought a permanent pre-flight defense. The precedents at the bottom name one lane per step.

The loop is a process representation, not a recipe: every area plugs in its own evidence classes, groups, and bar vocabulary. The skill removes the re-derivation, not the judgment.

When this fires

  • BEFORE planning or running any calibration, tuning sweep, A/B, or bar-pinning run.
  • When a calibration run keeps dying pre-run — that is the loop working; each death buys a permanent defense, never a weaker guard.
  • When someone asks "how did calibration go" — answer in this loop's vocabulary (authority → census → pre-registration → cycles → classification → pinned bars).
  • When adjudicating a result — Calibrated / Fallback / Sparse / NO VERDICT are the only verdicts.

The loop

  1. Authority before runs. Plan-first for any calibration that touches content or spends rig time: file the plan, claim it (**Claimed:** <session>, <epoch>, expires <ISO-UTC> — ONE claim line per file; a malformed line — prose after the expiry token, a millisecond epoch, a +07:00 offset — blinds the lease checker for EVERY lane), and get the delegated verdict (Claude ping-pong, proposal 001; max rounds per agent.verdict_max_rounds) BEFORE code or runs. Record binding amendments BY NAME (A1–A4 style) and carry them forward — verdict rounds revise the plan, never silently widen it. Machinery reviews are round-numbered in the plan so a successor thread picks up at the right round.

  2. Census the evidence before tuning anything. Count the labeled evidence PER GROUP before proposing values — pin RAW counts as data and derive the verdict, so a floor change never re-mines. Precedents — riir-refine i233 pinned per-(domain × shape) RAW census counts (180 labeled spans) before any narrowing; katgpt-rs bench 982 printed its label census (315 fail / 320 requests) and REFUSED to adjudicate when one class was degenerate. If a group has no evidence, it has no verdict — an empty window ABSTAINS (the adoption-gate posture), it never invents.

  3. Pre-register the decision rule before pair 1. Bars, thresholds, floors, falsifiers, promotion bar, stop rule — written and committed BEFORE the run (the riir-ai coverage_seek A/B pre-registration — issue 1095, d45e061bd — bound the promotion bar

    • stop rule before pair 1; riir-infer plan 633 landed every gate explicit — G1 FAIL +1.46 CER, G2 FAIL 0.57× vs the CPU lane, G4 PASS). A post-hoc bar is a tune-to-pass, and a tune-to-pass measures the tuner. Include up front: the small-group rule (e.g. n < 20 → Sparse; the 30/30 two-class label-census floor), the permutation-null or canary falsifier that proves the gate CAN fail, and the demote literal (the env-switch posture several repos pin — e.g. riir-infer / riir-refine / riir-rethink: a promoted arm's env is DEMOTE-ONLY, an opt-in arm's PROMOTE-ONLY — spell yours now).
  4. Separate the calibration cell from the production cell. Bars are pinned FROM the calibration run; the production cell (sprint#029's example: N=100, real contention, don't go easy) then runs UNDER those bars — never the reverse (sprint#029 tasks 6/8; bars are never read off the calibration load). A green production run that never stressed the thing it claims to stress is a SKIP, not a pass. And a calibration law discovered in the calibration cell PROPAGATES to the production cell BY NAME (sprint#029's example: fresh press + money-only pre-mint + account key are load-bearing for purchases — a reused old vessel SKIPs on zero purchases by construction).

  5. Queue cycles die loud, die cheap — and each death lands a defense. Expect the first queue cycles to die in pre-flight (v1 died-by-design on a missing pin; v2's vessel dump revealed three fatal gaps — empty shop/merchant books, penniless bots, a throwaway-key signature; v3 carried the full corrected sequence IN the job). The law: every pre-flight catch becomes a PERMANENT defense in code (boot strips, funded wallets, empty-book pre-flight refusals that name the fix), never a one-off patch on a job script. Never weaken a guard to get a green. Press fresh whatever the verdict depends on (content, signing key) — a stale artifact's green proves the stale artifact.

  6. Measure with provenance and stability. Every quoted bar carries its bench_preflight PROVENANCE line (box state, power source, load class). A/B pairs are interleaved and two-run stable (katgpt-rs/scripts/sequential_ab_timing_audit.py — two sequential arms measure the scheduler, +5–22%); paired non-inferiority (the riir-rethink bar δ = max(1.0pp, 2.5·SE) is one house example — carry your lane's own). Queue long cells via agent-mail rig run --exclusive --timeout <bound> on a verified-quiet rig, the estimate named in --title/the plan (the owner's tiered-queue decree, 2026-11-12, wants per-job estimates — pass --est only where the rig's agent-mail build supports it; a build without it refuses the invocation at launch, verified on m3 2026-11-15). Never bench a BUSY box, never bench beside a GPU consumer, and re-run any row a contender touched ALONE before quoting it.

  7. Classify per group, never pool; withhold on absent evidence. The verdict vocabulary is the calibrated_dispatch shape — Calibrated (Wilson 95% LCB ≥ τ over ≥ N labeled — riir-refine i233's values were τ 0.6 over n ≥ 20; pin yours), Fallback (measured under-floor → demote the lever), Sparse (n < N → withhold; narrowing stays OFF — the evidence DEMOTES the unproven lever, it never hopes). Conservative per-bit: one non-Calibrated bit falls the whole surface back to the full posture (the dispatch narrows, never widens). An undecidable cell prints NO VERDICT with every envelope term named (a distinct non-zero exit where the lane defines one — seal-remake i201's T2 verdict machinery uses exit 3) — never fold it into a pass; count unknowns loud; pool nothing across groups, ever (the per-predicate law).

  8. Pin, inherit, demote, record. Pin the bars beside their PROVENANCE; carry the inherited laws into the next cell's plan (named, not folklore); demote the losing arm (never delete it); record verdict + re-open conditions in the plan/issue so the next lane starts at step 4. A negative calibration is a RESULT — riir-infer i1216's head-surface PoC died at its own T0 headroom gate (file removed; T0 record ca9e7fcb, bench 1217); riir-train plan 498's QAD head collapsed; riir-refine i237's composition cell NO-GO (file removed 505dffd5; record af5dd5f3, bench 151) — the record IS the yield.

The orchestration half — watched, not babysat

Long calibrations are park-and-notify: agent-mail rig run --exclusive --timeout <bound>, the estimate named in --title/the plan; END the turn; mail arrives at start/poison/end. Continuity across context overflow is the plan's Status line — a successor thread reads the plan and the out-dirs, never re-derives; fold superseded claims into **Lease closed:** rows (the lint's exact spelling) so exactly one Claimed line remains. Watchdog pickups poll on their cadence; a backstop wake is agent-mail send --to <own-inbox>@<rig> --at <time> "check <job>" then END the turn. Verdict pickups answer the verdict round in the SAME session, not a new thread. Rig log purges completed jobs — check the WORK DIR before declaring a job vanished (the duel d241af5d + amo 20979a65 near-mistakes, 2026-11-15). Housekeeping lanes verify their own windows only; they never re-run or re-adjudicate a live lease.

Never rules

  • Never tune to pass; a bar written after the run is not a bar.
  • Never pool group verdicts; never widen on Sparse evidence.
  • Never quote a number without its box state; never bench a BUSY rig.
  • Never re-pin a floor from a dirty, poisoned, or stale-posture run — disclose, re-run clean.
  • Never gate correctness behind the thing being calibrated (lossy surfaces gate on per-family conditional retention, never aggregates).
  • Sigmoid, never softmax; decline and abstain are typed answers, not guesses.
  • A calibration that cannot fail is not a calibration — arm the canary, prove it fires.

Precedents (one lane per step)

  • sprint#029 shopping-drive calibration (../seal-sprints/.plans/029_...) — the full walk-through: verdict rounds, v1/v2/v3 pre-flight deaths, the inherited-laws handoff.
  • riir-refine i233 calibrated_dispatch — census → Wilson LCB → Calibrated/Sparse/Fallback, conservative per-bit narrowing, the re-run census instrument.
  • katgpt-rs bench 982 (i1210 C2) — the census-floor NO VERDICT + permutation-null falsifier, re-open conditions recorded.
  • riir-rethink ESC ρ gates (issues 052/054/064 — files removed; 064's landing 1a45ae6, close-out 65e32b8; the live posture is the ESC block in riir-rethink AGENTS.md) — per-suite gate-rho tables, rate-targeted fit, collapse caveats on the verdict.
  • seal-remake sprint!053 — drive weights past the sigmoid ceiling → the zeroing form; verdicts as lane-posture pins with an armed witness row.
  • riir-train plan 497 — the LoRA trial-shift gain re-attributed from an instrumentation artifact (+700% claimed, landing commits 5a082a21/7568e6df) to +131.5% measured (per-dispatch shader compiles + blocking readbacks), paired harness as the record.
  • riir-infer plan 633 — every gate explicit and honest, negatives quoted beside positives.
  • sprint#029 tasks 6/8 — calibration cell separate from the chaos cell, SKIP-not-pass, PROVENANCE beside every bar.

Instruments

  • agent-mail rig status / agent-mail rig run --exclusive --timeout — queue + quiet-gate; estimates ride --title until the rig's build carries the decree's --est.
  • riir-reflex/scripts/bench_preflight.sh — the PROVENANCE line (power, load, box state).
  • katgpt-rs/scripts/sequential_ab_timing_audit.py — the interleaved-A/B law.
  • Wilson/Beta LCB precedents — katgpt-core beta_lcb_order_into, riir-instinct src/stats.rs.
  • katgpt-rs/scripts/claim_lease_check.py --lint <file> — the claim-line grammar, lint first.
  • seal-e2e lanes — seal-e2e-multi §2/§4 owns the rig queue posture and the stress axes.
planning-and-task-breakdownaddyosmani/agent-skillsBreaks work into ordered tasks. Use when you have a spec or clear requirements and need to break work into implementable tasks. Use when a task feels too large to start, when you need to estimate scope, or when parallel work is possible.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
screenpipethedotmack/claude-memSearch screen and audio history recorded by Screenpipe and optionally save selected context into claude-mem. Use when the user asks to recall desktop activity, a meeting, or something seen in another app through Screenpipe.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
contact-widgetnexu-io/open-designSelf-contained floating chat widget with welcome screen, social links, meeting button, and message input. Single HTML file, zero dependencies.Stars: 108kUpdated Claude CodeCodexVerifiedInstall
meeting-notesnexu-io/open-designMeeting notes page — title bar with attendees, agenda checklist, decisions block, action items table with owners + dates, and a "next meeting" footer. Use when the brief mentions "meeting notes", "minutes", "1:1 notes", "all-hands recap", or "会议纪要".Stars: 108kUpdated Claude CodeCodexVerifiedInstall