tvproductions/gzkit / .agents/skills/gz-adr-evaluate
gz-adr-evaluate
VerifiedUse after authoring an ADR or OBPI to score its quality and optionally run the red-team challenges.
gz-adr-evaluate
Purpose
Structured quality evaluation for ADRs and their OBPI decompositions. This skill provides rubrics, challenge protocols, and a red-team prompt that form a blocking QC step between ADR authoring and human proposal/defense review.
It can be invoked for newly authored ADRs or for retroactive evaluation of an existing ADR package.
Common Rationalizations
| Thought | Reality |
|---|---|
| "The ADR looks comprehensive, it'll score well" | Comprehensiveness does not equal quality. The rubric measures decision justification, not document length. |
| "The CLI gave a high score, no need for manual review" | The CLI uses heuristics that produce false negatives. Manual review supersedes CLI pre-screen. |
| "This is a retroactive evaluation, I'll be lenient" | The rubric is the rubric. Retroactive evaluation with relaxed standards produces misleading scorecards. |
| "The red-team challenges don't apply to this simple ADR" | Every challenge must be engaged. N/A is not acceptable. Simple ADRs often have unexamined assumptions. |
| "One dimension scored 1, but the overall weighted score is above 3.0" | Any dimension scoring 1 must be revised regardless of the weighted total. |
| "I'll skip the OBPI scoring since the ADR scored well" | ADR quality and OBPI quality are independent assessments. Both are required. |
Red Flags
- EVALUATION_SUBSTANCE.md written without reading the actual ADR document
- A judged scorecard hand-written into
EVALUATION_SCORECARD.md, whichgz adr evaluateregenerates and will destroy (GHI #769) - CLI pre-screen score accepted without manual verification (Steps 3–4 skipped)
- Manual score differs from CLI score but no explanation of the heuristic mismatch
- Red-team challenges marked N/A instead of engaged
- ADR proceeds to proposal/defense with a NO GO or CONDITIONAL GO verdict
Persona
Active driver: pipeline-orchestrator — read .gzkit/personas/pipeline-orchestrator.md and adopt its behavioral identity before executing this skill. Evaluation is a sequenced ceremony (score dimensions → engage red-team → render verdict); step-discipline keeps the rubric honest. Evaluation is independent judgment, not confirmation. Score what you read, not what you hope.
Persona Dispatch
Evaluation is the highest-risk read-only judgment ceremony — the rubric's 8 dimensions split cleanly along the persona functional boundaries, and a single driver scoring its own scoring is the precise optimistic-bias defect spec-reviewer's anti-traits name. Dispatch the following personas to produce independent dimension scores the driver synthesizes:
| Persona | Function in this ceremony | Invoked at |
|---|---|---|
spec-reviewer | Scores spec/requirement dimensions of the rubric: requirement clarity, target-scope falsifiability, OBPI decomposition coverage, REQ-trace integrity — anything that asks "is the spec honest, falsifiable, traceable?" | Steps 3–4 (manual rubric scoring) |
quality-reviewer | Scores architectural dimensions: decision justification, alternatives considered, SOLID-analogues for ADR design, single-responsibility per ADR, decision boundaries, maintainability of the proposed surface | Steps 3–4 (manual rubric scoring) |
narrator | Composes the operator-facing scorecard and frames red-team challenge findings — each challenge's finding rendered as evidence-to-decision, not as raw analysis | Step 5 (red-team challenges, if invoked) and final scorecard render |
Personas not dispatched: implementer (evaluation is pre-implementation review — no code exists to write or evaluate).
The receipt machinery that would attest these dispatches is not yet authored (GHI #846; ADR-pool.obpi-pipeline-dispatch-attestation is Superseded into ADR-0.0.73, which delivered no receipts). This ceremony does have a dispatch channel: gzkit.adr_eval_dispatch reports NOT DISPATCHED absent a receipt and never infers dispatch from the presence of scores (GHI #770). See § Degraded mode.
Degraded mode — when dispatch cannot run (binding, GHI #770)
Dispatch is sometimes genuinely unavailable — a session that forbids subagents, a headless or cron run, a harness without the Agent tool. Until GHI #770 there was no compliant path for those runs and no way to mark the output, so the mandate was simply skipped: on 2026-08-07 an evaluation of ADR-0.35.0 ran single-driver, produced 8 dimension scores and a GO verdict, and nothing recorded that the mandated dispatch had not happened. An un-compliable mandate gets worked around.
The degraded mode is therefore declared and legitimate, never silent:
- Running single-driver is permitted. Do not fabricate a dispatch, and do not abandon the evaluation.
- The scorecard states it mechanically.
render_scorecard_markdownalways emits a--- Persona Dispatch ---channel: one row per mandated persona,NOT DISPATCHED / no dispatch receipt recordedabsent a receipt, plus aDISPATCH MODE: SINGLE-DRIVERverdict. This is not something you remember to write — the renderer emits it unconditionally, andtests/test_adr_eval_dispatch.pyfails closed if a dispatched and an undispatched scorecard could ever render identically. - A single-driver scorecard is not an independent review. Say so when relaying the verdict; do not present it as one.
A dispatch is credited only from a recorded receipt (gzkit.adr_eval_dispatch), never inferred from the presence of scores — the same discipline the substance channel applies to judge verdicts (GHI #624). Nothing emits that receipt yet, so every scorecard truthfully reads SINGLE-DRIVER until that machinery is authored (GHI #846); the channel populates then with no change here.
Persona doctrine reference: ADR-0.0.11-persona-driven-agent-identity-frames (Validated).
Trust Model
Layer 1 - Evidence Gathering. This tool reads ADR documents and produces evaluation scorecards. It does not modify ADR or brief content.
- Reads: ADR document, OBPI briefs, evaluation framework template
- Writes:
EVALUATION_SUBSTANCE.mdin the ADR directory (the judge's file;EVALUATION_SCORECARD.mdis machine-owned and regenerated by the CLI) - Books: one
adr-evaluationledger event per successfulgz adr evaluaterun (ADR-0.0.26, OBPI-0.0.26-01) - Does NOT modify: ADR content, brief content, or registries
Invocation
/gz-adr-evaluate ADR-X.Y.Z # evaluate a specific ADR
/gz-adr-evaluate ADR-X.Y.Z --red-team # include the 10-challenge red-team protocol
Arguments:
| Argument | Required | Description |
|---|---|---|
adr_id | Yes | ADR identifier (for example ADR-0.13.0) |
--red-team | No | Include the 10-challenge adversarial review protocol |
When to Use
- After drafting a new ADR and its OBPIs
- Before a Draft ADR goes to human proposal/defense review
- When benchmarking the quality of an existing ADR package
- When you want a structured red-team pass against scope, evidence, and decomposition
Procedure
Step 1: Run CLI deterministic scoring (pre-screen)
uv run gz adr evaluate ADR-X.Y.Z
This produces structural quality scores (8 ADR dimensions, OBPI scores) and
writes an initial EVALUATION_SCORECARD.md.
The CLI score is a pre-screen, never a gate. The CLI uses pattern-matching heuristics (keyword detection, section structure) that can under-score or over-score dimensions when an ADR distributes content across non-standard sections. Common false negatives:
- Problem Clarity: CLI looks for before/after language in Intent — misses ADRs that carry depth in Rationale or Agent Context Frame
- Decision Justification: CLI looks for numbered items in Decision — misses bullet-point decisions or justification distributed across Decision + Alternatives Considered
Mandatory next step: Always proceed to Step 2 regardless of CLI verdict. The agent reads the ADR, analyzes every CLI dimension score for false-positive and false-negative artifacts, and produces the authoritative manual scorecard that supersedes the CLI pre-screen.
Step 2: Locate the ADR and its OBPIs
- Resolve the ADR document under
docs/design/adr/**/ADR-X.Y.Z-*/ADR-X.Y.Z-*.md - List all OBPI briefs in
obpis/(preferred) orbriefs/(legacy) - Read the evaluation framework from
assets/ADR_EVALUATION_FRAMEWORK.md - Read the CLI-generated
EVALUATION_SCORECARD.md— note each dimension score and the CLI's stated findings for analysis in Step 3
Step 3: Score ADR Quality (Part 1 - 8 Dimensions)
Read Part 1 of the framework and score the ADR on each dimension (1-4 scale):
| # | Dimension | Weight |
|---|---|---|
| 1 | Problem Clarity | 15% |
| 2 | Decision Justification | 15% |
| 3 | Feature Checklist Completeness | 15% |
| 4 | OBPI Decomposition Quality | 15% |
| 5 | Lane Assignment Correctness | 10% |
| 6 | Scope Discipline | 10% |
| 7 | Evidence Requirements | 10% |
| 8 | Architectural Alignment | 10% |
For each dimension, work through the checklist items in the framework and score based on how many checklist items pass with path-level evidence.
CLI reconciliation (mandatory per dimension): Compare the agent's manual score against the CLI pre-screen score. When they differ, the scorecard MUST document why — naming the specific CLI heuristic that misfired (false positive or false negative) and the evidence that justifies the override. The manual score is authoritative; the CLI score is recorded for traceability.
Step 4: Score OBPI Quality (Part 2 - 5 Dimensions)
For each OBPI, score on 5 dimensions (1-4 scale):
| Dimension | Question |
|---|---|
| Independence | Can this OBPI be completed without waiting for others? |
| Testability | Can completion be verified with commands? |
| Value | What concrete capability would be lost if this OBPI were removed? |
| Size | Is this a 1-3 day work unit? |
| Clarity | Could a different agent implement this without ambiguity? |
Step 5: Run Red-Team Challenges (Optional - Part 3)
If --red-team is specified, or if the evaluator wants stronger adversarial
review, work through all 10 structured challenges from the framework.
Every challenge must be engaged. N/A is not acceptable.
Step 6: Determine Verdict
Apply the framework thresholds:
| ADR Weighted Total | Verdict |
|---|---|
| >= 3.0 | GO - Ready for proposal/defense review |
| 2.5 to < 3.0 | CONDITIONAL GO - Address weaknesses, then re-evaluate |
| < 2.5 | NO GO - Structural revision required |
OBPI threshold: Average >= 3.0 per OBPI. Any OBPI scoring 1 on any dimension must be revised.
Red-team threshold: <= 2 failures = GO, 3-4 = CONDITIONAL GO, >= 5 = NO GO.
Step 7: Record Substance Scorecard
Write EVALUATION_SUBSTANCE.md in the ADR directory using the summary
template from the framework.
Never write
EVALUATION_SCORECARD.mdby hand (GHI #769). That path is machine-owned:gz adr evaluaterewrites it wholesale on every run, so a judged scorecard authored there is destroyed by the next invocation. The two files measure different things and GHI #624 forbids compositing them — the split makes that separation structural instead of a warning banner the writer ignores. The closeout NO-GO gate reads this file first, so a recorded NO GO now survives regeneration.
Cite the CLI pre-screen for traceability rather than restating it as your own finding. Include:
- CLI pre-screen verdict and weighted total (for traceability)
- All ADR dimension scores with weighted totals and rationale — when the manual score differs from the CLI score, state the CLI score, the manual score, and the specific heuristic mismatch that caused the divergence
- All OBPI dimension scores with averages
- Red-team challenge results when run
- Overall verdict (GO / CONDITIONAL GO / NO GO)
- Action items for any deficiencies
Low-Score Footer Guidance
Append the footer under the condition gz validate --evaluation-justify-binding
fails closed on (ADR-0.0.26 Decision 2): any dimension scores below
low_score_threshold (it ships as < 3.0; the file below, never this line, is
authoritative), or at least red_team_count_threshold red-team challenges
fired — both read from data/eval_feedback_thresholds.json. The weighted total
is not the trigger: a single dimension below threshold fails the gate while the
total stays healthy. The footer requires an OBPI brief under the evaluated ADR
to exist, and names it:
> Consider: uv run -m gzkit justify OBPI-<X.Y.Z>-<NN> --save
When the ADR has no OBPI brief yet there is no footer to append; the evaluation is answered instead on the ADR itself as a draft slug — its id with dots as dashes:
> Consider: uv run -m gzkit justify --draft "<what the evaluation found>" --draft-slug adr-<x>-<y>-<z> --save
Only these two forms discharge the gate: the walkthrough's own frontmatter must name the evaluated ADR, an OBPI under it, or its draft slug (operator ruling 2026-09-14, GHI #996). A walkthrough on a tracking GHI does NOT — the GHI's link to the ADR lives on GitHub, which the gate does not consult — so never route the footer to one.
Amendment record (REQ-0.0.19-04-05, 2026-09-14, GHI #996). That attested REQ
asserts this block cites the < 3.0 threshold, requires a tracking GHI or OBPI
to exist, and suggests uv run -m gzkit justify. All three still hold. What the
operator's ruling retired is the weighted total as the trigger and the
tracking-GHI route: the footer now fires on the gate's own condition and names
only an OBPI, because a GHI walkthrough never discharged the gate it answers.
A trigger means the ADR's structural weakness will leave implementing agents
guessing on at least one OBPI. The
pre-execution walkthrough (gz-justify skill) surfaces the hidden ambiguity
before promotion, and the lifecycle refuses to advance the ADR past Pending
until a qualifying walkthrough exists. Skipping this footer on a triggered
evaluation is the adjacent rationalization pattern the walkthrough exists to
close.
Step 8: Gate Decision
- GO: proceed to human proposal/defense review
- CONDITIONAL GO: revise the ADR or OBPIs, then re-run evaluation
- NO GO: return to authoring; do not proceed to proposal/defense
External Red-Team Review
For adversarial review by a separate model:
- Read the red-team model prompt from Part 4 of the framework
- Copy the prompt and append the target ADR document
- Send it to a separate model
- Review the challenge report and address findings before proposal/defense
Assets
- Evaluation Framework:
assets/ADR_EVALUATION_FRAMEWORK.md- Part 1: ADR quality rubric (8 weighted dimensions)
- Part 2: OBPI quality rubric (5 dimensions)
- Part 3: Red-team challenge protocol (10 challenges)
- Part 4: Red-team model prompt
- Part 5: Scoring summary template
Failure Modes
| Failure | Cause | Resolution |
|---|---|---|
| Framework not found | assets/ADR_EVALUATION_FRAMEWORK.md missing | Repair skill directory structure |
| ADR not found | No ADR package matches the requested ID | Verify ADR exists and uses canonical naming |
| No OBPIs found | obpis/ and briefs/ are empty | ADR must have co-created OBPIs before evaluation |
| Scorecard not written | Path or permission error | Verify ADR directory is writable |
| NO GO verdict | ADR has structural deficiencies | Revise ADR and re-run evaluation before proposal/defense |
Acceptance Rules
- All 8 ADR dimensions are scored with evidence-based rationale
- All OBPIs are scored on all 5 dimensions
EVALUATION_SUBSTANCE.mdis written to the ADR directory (neverEVALUATION_SCORECARD.md, which the CLI regenerates — GHI #769)- Verdict follows the threshold rules with no manual override
NO GOblocks progression to human proposal/defense
Related Skills
| Skill | Relationship |
|---|---|
gz-adr-create | Authoring workflow that should invoke evaluation before proposal |
gz-adr-audit | Post-completion audit; downstream phase |
gz-adr-closeout-ceremony | Closeout occurs after implementation, not authoring |
References
- Evaluation framework:
assets/ADR_EVALUATION_FRAMEWORK.md - ADR lifecycle:
docs/governance/GovZero/adr-lifecycle.md - GovZero charter:
docs/governance/GovZero/charter.md - Parity origin: the AirlineOps
gz-adr-evaluateskill (sibling repository; not resolvable from this one)
Related ADRs
- ADR-0.0.19 — Pre-execution reasoning walkthrough. The Low-Score Footer
Guidance section routes operators from a triggered evaluation into the
gz-justifywalkthrough so hidden ambiguity is surfaced before the ADR is promoted into active work.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |