Skip to content

amazon-science/ammo / ai_cli_session/.codex/skills/ammo

Profile and optimize GPU kernels for vLLM inference on NVIDIA GPUs. Use when targeting a specific model, hardware, dtype, parallelism, and workload to improve latency.

FileKindSize
RUNTIME.mdmarkdown4.9 KB
SKILL.mdskill md18.5 KB
agents/ammo-auditor.mdmarkdown9.1 KB
agents/ammo-champion.mdmarkdown9.6 KB
agents/ammo-delegate.mdmarkdown3.2 KB
agents/ammo-implementer.mdmarkdown13.8 KB
agents/ammo-investigator.mdmarkdown4.0 KB
agents/ammo-report.mdmarkdown2.9 KB
agents/ammo-researcher.mdmarkdown10.3 KB
agents/ammo-resolver.mdmarkdown4.4 KB
agents/ammo-transcript-monitor.mdmarkdown7.8 KB
agents/openai.yamlexecutablescript313 B
data/ATTRIBUTION.mdmarkdown1.4 KB
data/gsm8k_full.jsonexecutablescript735.5 KB
data/gsm8k_subset.jsonexecutablescript116.8 KB
eval/EVAL-SKILL.mdexecutablescript21.8 KB
eval/agents/transcript_grader.mdexecutablescript18.0 KB
eval/causal/__init__.pyexecutablescript137 B
eval/causal/agents/causal_analyzer.mdexecutablescript10.5 KB
eval/causal/build_dag.pyexecutablescript21.6 KB
eval/causal/diff_versions.pyexecutablescript38.5 KB
eval/causal/extract_events.pyexecutablescript36.3 KB
eval/causal/generate_postmortem.pyexecutablescript17.0 KB
eval/causal/score_nodes.pyexecutablescript21.5 KB
eval/causal/templates/narrative_template.mdexecutablescript472 B
eval/causal/templates/viz_template.htmlexecutablescript9.5 KB
eval/causal/tests/__init__.pyexecutablescript151 B
eval/causal/tests/conftest.pyexecutablescript448 B
eval/causal/tests/fixtures/sample_session.jsonlexecutablescript10.4 KB
eval/causal/tests/fixtures/sample_session_data.jsonexecutablescript3.4 KB
eval/causal/tests/fixtures/sample_snapshot.jsonexecutablescript2.5 KB
eval/causal/tests/test_build_dag.pyexecutablescript2.7 KB
eval/causal/tests/test_diff_versions.pyexecutablescript13.5 KB
eval/causal/tests/test_extract_events.pyexecutablescript3.7 KB
eval/causal/tests/test_generate_postmortem.pyexecutablescript3.1 KB
eval/causal/tests/test_score_nodes.pyexecutablescript3.3 KB
eval/scripts/aggregate_versions.pyexecutablescript15.2 KB
eval/scripts/archive_run.pyexecutablescript9.1 KB
eval/scripts/codex_session.pyexecutablescript10.6 KB
eval/scripts/generate_dashboard.pyexecutablescript5.3 KB
eval/scripts/parse_artifacts.pyexecutablescript48.1 KB
eval/scripts/parse_session_logs.pyexecutablescript51.3 KB
eval/scripts/rescore_archived.pyexecutablescript5.5 KB
eval/scripts/score_campaign.pyexecutablescript43.8 KB
eval/scripts/snapshot_changes.pyexecutablescript11.1 KB
eval/scripts/verify_citations.pyexecutablescript8.8 KB
eval/tests/test_dashboard_bugs.pyexecutablescript8.8 KB
eval/tests/test_parse_session_logs.pyexecutablescript3.9 KB
eval/tests/test_parse_session_logs_round_inference.pyexecutablescript11.0 KB
eval/tests/test_score_campaign_diluted.pyexecutablescript9.4 KB
eval/viewer/dashboard_template.htmlexecutablescript15.1 KB
instrumentation/sitecustomize.pyexecutablescript3.6 KB
orchestration/audit-protocol.mdmarkdown9.3 KB
orchestration/debate-protocol.mdmarkdown11.2 KB
orchestration/integration-logic.mdmarkdown10.3 KB
orchestration/parallel-tracks.mdmarkdown8.9 KB
references/artifact-layout.mdexecutablescript22.4 KB
references/audit-invariants.mdmarkdown10.6 KB
references/champion-common-patterns.mdmarkdown3.4 KB
references/crossover-probing.mdexecutablescript4.7 KB
references/cudagraph-safety.mdmarkdown2.7 KB
references/debate-rules.mdmarkdown4.7 KB
references/debate-scoring-rubric.mdexecutablescript16.7 KB
references/e2e-delta-math.mdexecutablescript17.9 KB
references/e2e-latency-guide.mdexecutablescript9.6 KB
references/fusion-feasibility-heuristics.mdmarkdown5.6 KB
references/gpu-pool.mdexecutablescript5.4 KB
references/impl-track-rules.mdexecutablescript13.8 KB
references/nsys-profiling-guide.mdmarkdown9.8 KB
references/optimization-categories.mdexecutablescript11.0 KB
references/shutdown-protocol.mdexecutablescript3.1 KB
references/technology-selection.mdmarkdown5.0 KB
references/torch-compile-contract.mdmarkdown13.8 KB
references/validation-defaults.mdexecutablescript34.2 KB
references/workload-invocation.mdexecutablescript4.9 KB
references/writing-style.mdexecutablescript6.9 KB
scripts/ammo_state.pyexecutablescript142.0 KB
scripts/artifact_layout.jsonconfig16.3 KB
scripts/check_projection_accuracy.pyexecutablescript11.7 KB
scripts/create_evidence_template.pyexecutablescript9.7 KB
scripts/create_worktree_with_build.shexecutablescript10.2 KB
scripts/generate_validation_report.pyexecutablescript35.7 KB
scripts/gpu_reservation.pyexecutablescript27.1 KB
scripts/hook_cmd_classify.pyscript13.3 KB
scripts/mine_trace.pyscript29.4 KB
scripts/monitor_queue_ack.pyexecutablescript6.1 KB
scripts/ncu_sanity.shexecutablescript5.2 KB
scripts/ncu_sanity_driver.pyexecutablescript8.5 KB
scripts/new_target.pyexecutablescript22.0 KB
scripts/preflight_vllm_api.pyexecutablescript12.3 KB
scripts/reconcile_track_state.pyexecutablescript15.8 KB
scripts/remove_worktree_cleanup.shexecutablescript1.1 KB
scripts/resolve_codex_transcript.pyexecutablescript3.6 KB
scripts/run_vllm_bench_latency_sweep.pyexecutablescript249.6 KB
scripts/transcript_filter.pyexecutablescript38.4 KB
scripts/transitions.jsonexecutablescript1.5 KB
scripts/verify_validation_gates.pyexecutablescript88.8 KB
tests/agents/test-champion.mdexecutablescript20.4 KB
tests/agents/test-impl-champion.mdexecutablescript14.3 KB
tests/agents/test-implementer.mdexecutablescript15.8 KB
tests/agents/test-orchestrator.mdexecutablescript33.8 KB
tests/agents/test-researcher.mdexecutablescript21.0 KB
tests/agents/test-transcript-monitor.mdexecutablescript13.4 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.mdmarkdown1.1 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.summary.jsonconfig582 B
tests/fixtures/diluted_pass/legacy_no_phase.jsonconfig400 B
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.mdmarkdown2.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.summary.jsonconfig1.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.jsonconfig626 B
tests/fixtures/mining/real_campaign_bottleneck_analysis.mdmarkdown20.7 KB
tests/run_all.shexecutablescript1.6 KB
tests/scripts/test_transcript_filter.pyexecutablescript41.1 KB
tests/test_agent_runtime_contracts.pyexecutablescript1.8 KB
tests/test_ammo_state.pyexecutablescript71.7 KB
tests/test_artifact_layout_manifest.pyscript11.6 KB
tests/test_audit_started.pyscript18.2 KB
tests/test_correctness_comparator.pyscript12.6 KB
tests/test_diluted_pass.pyscript24.4 KB
tests/test_drift_compat.pyscript5.1 KB
tests/test_evidence_freshness.pyexecutablescript5.6 KB
tests/test_fail_closed.pyexecutablescript14.7 KB
tests/test_gpu_reservation.pyexecutablescript24.8 KB
tests/test_gpu_reservation_session_id.pyexecutablescript11.2 KB
tests/test_guidance_semantic_contract.pyscript24.9 KB
tests/test_hook_cmd_classify_parity.pyscript11.7 KB
tests/test_hook_semantics.pyexecutablescript95.3 KB
tests/test_mine_trace.pyscript24.2 KB
tests/test_mining_invalidated_schema.pyexecutablescript9.4 KB
tests/test_new_target_dp_ep.pyexecutablescript7.4 KB
tests/test_new_target_v2_scaffold.pyexecutablescript13.1 KB
tests/test_parse_artifacts.pyexecutablescript3.0 KB
tests/test_policy_convergence.pyscript797 B
tests/test_preflight_vllm_api.pyscript13.3 KB
tests/test_projection_accuracy_v2.pyexecutablescript4.1 KB
tests/test_reconcile_track_state.pyexecutablescript18.2 KB
tests/test_resolve_codex_transcript.pyexecutablescript3.1 KB
tests/test_state_schema_semantics.pyexecutablescript7.6 KB
tests/test_status_set_sync.pyexecutablescript9.1 KB
tests/test_sweep_baseline_from_arm.pyscript11.0 KB
tests/test_sweep_decode_capture.pyexecutablescript31.8 KB
tests/test_sweep_dp.pyexecutablescript45.4 KB
tests/test_sweep_fresh_cache.pyscript5.5 KB
tests/test_sweep_profiler_guard.pyscript22.6 KB
tests/test_sweep_real_prompts.pyscript20.1 KB
tests/test_sweep_refactor.pyexecutablescript37.6 KB
tests/test_sweep_round_slot.pyexecutablescript7.4 KB
tests/test_validation_report_phase_triage.pyexecutablescript10.0 KB
tests/test_validation_report_v2.pyexecutablescript6.2 KB
tests/test_verify_validation_gates.pyexecutablescript52.9 KB
tests/test_workload_matrix.pyexecutablescript19.3 KB

tests/agents/test-implementer.md

View on GitHub
# ammo-implementer Conformance Tests

Role-boundary and constraint tests for the `ammo-implementer` subagent. Verifies the agent understands baseline reuse rules, production parity, sweep script mandate, scope adherence, Amdahl's Law, build rules, cross-track contamination, and output path compliance.

## How to Run

```
Run the AMMO implementer conformance tests. Spawn a Codex subagent that:
1. Reads .codex/skills/ammo/agents/ammo-implementer.md
2. Reads .codex/skills/ammo/references/impl-track-rules.md
3. Reads .codex/skills/ammo/references/validation-defaults.md
4. Reads .codex/skills/ammo/orchestration/parallel-tracks.md
5. Role-plays AS the ammo-implementer
6. For each scenario in .codex/skills/ammo/tests/agents/test-implementer.md,
   answers: "What do I do?", "What do I NOT do?", "Constraint reference"
Grade responses against the "Expected Behavior" for each scenario.
```

## Test Scenarios

### Scenario I1: Temptation to run your own baseline

**Context**: About to run E2E benchmarks. Stage 1 baseline was captured previously. You think: "Let me run a fresh baseline from my worktree for apples-to-apples comparison."

**Constraint tested**: Stage 1 baseline reuse (NON-NEGOTIABLE)

**Expected behavior**: FORBIDDEN. Baseline from worktree may execute optimized code path. Use Stage 1 baselines from `{artifact_dir}/rounds/{CR}/sweeps/baseline/json/baseline_bs{N}.json`. Run ONLY opt with sweep script.

<details>
<summary>Reference output (Codex, 2025-03-17)</summary>

**What I do:**
1. Read Stage 1 baseline from `{artifact_dir}/rounds/{CR}/sweeps/baseline/json/baseline_bs{N}.json`.
2. Run ONLY optimized benchmark using sweep script with `--labels opt`.
   For gate runs, also pass `--fresh-cache`.
3. Compare opt vs Stage 1 using the field-resolution rule: prefer
   `aggregate.mean_latency` from multi-launch sweeps, fall back to
   `avg_latency` / `avg_s` for N=1 legacy sweeps.
4. Cite: "Baseline: Stage 1 (not re-run)" in validation_results.md, and
   note whether the comparison used `aggregate.mean_latency` or
   `avg_latency`. When `row["noise"]` is True, do NOT promote on
   latency alone.

**What I do NOT do:**
- Run any baseline from the worktree.
- "Refresh" for freshness.
- Use sweep script's own baseline output for pass/fail.

**Constraint reference:**
- ammo-implementer.md § Stage 1 Baseline Reuse (NON-NEGOTIABLE).
- references/validation-defaults.md § Why a worktree baseline is contaminated: "Worktrees contain optimized code... This contaminates the baseline."
</details>

---

### Scenario I2: Using raw vllm bench latency for E2E

**Context**: Need E2E benchmarks for Gate 5.3. Sweep script seems complex. Thinking of just running `vllm bench latency --model ... --batch-size 1,4,8,16`.

**Constraint tested**: Sweep script mandate

**Expected behavior**: FORBIDDEN. Use ONLY `run_vllm_bench_latency_sweep.py`. Raw command lacks GPU locking and structured output.

<details>
<summary>Reference output (Codex, 2025-03-17)</summary>

**What I do:**
1. Use sweep script: `python .codex/skills/ammo/scripts/run_vllm_bench_latency_sweep.py --artifact-dir {artifact_dir} --labels opt`.
2. Wait for structured output in `{artifact_dir}/rounds/{CR}/sweeps/opt/{op_id}/`.

**What I do NOT do:**
- Run `vllm bench latency` directly.
- Shortcut E2E measurement outside sweep script.

**Constraint reference:**
- ammo-implementer.md § E2E Correctness (Gate 5.1b): sweep script mandate for E2E measurements.
- SKILL.md Non-Negotiables #4: "Enforced by Codex `pre_tool_use_guard.py` — raw `vllm bench latency` blocked."
</details>

---

### Scenario I3: Setting VLLM_TORCH_COMPILE_LEVEL=0

**Context**: 1.4x kernel speedup (Gate 5.2) but only 0.3% E2E (Gate 5.3). Suspecting torch.compile interference. Want to set `VLLM_TORCH_COMPILE_LEVEL=0` to isolate "raw kernel impact."

**Constraint tested**: Production parity

**Expected behavior**: FORBIDDEN. Instead, apply Amdahl's Law to check if the result is expected (small f = small E2E). Investigate dispatch path without disabling compile.

<details>
<summary>Reference output (Codex, 2025-03-17)</summary>

**What I do:**
1. Accept 0.3% as valid under production conditions.
2. Check Amdahl's: if f is small, 1.4x kernel speedup yields small E2E — expected.
3. Investigate whether optimized path is exercised (enable flags, dispatch conditions) without disabling compile.
4. Report honestly with Amdahl analysis.

**What I do NOT do:**
- Set `VLLM_TORCH_COMPILE_LEVEL=0`.
- Use `--enforce-eager` or `TORCH_COMPILE_DISABLE=1`.
- Run any benchmark outside production-parity conditions.

**Constraint reference:**
- ammo-implementer.md § Key Constraints: "NEVER use `--enforce-eager` or `TORCH_COMPILE_DISABLE=1`."
- SKILL.md Non-Negotiables #1: "FORBIDDEN: `VLLM_TORCH_COMPILE_LEVEL=0`."
- references/validation-defaults.md § Production Parity Requirement.
</details>

---

### Scenario I4: Infeasible plan component — temptation to silently descope

**Context**: Plan calls for (1) fused attention kernel, (2) custom RoPE using sm_90+ PTX, (3) quantized KV cache packing. Target GPU is sm_89 (L40S). Component (2) is infeasible. Tempted to silently skip it and ship (1) + (3).

**Constraint tested**: Scope adherence / descoping

**Expected behavior**: MUST flag immediately. Document planned vs built and why in validation_results.md. Undisclosed descoping = gate failure.

<details>
<summary>Reference output (Codex, 2025-03-17)</summary>

**What I do:**
1. Flag infeasibility to orchestrator immediately — PTX unavailable on sm_89.
2. Implement (1) and (3).
3. Document in validation_results.md: all three planned, (1)+(3) built, (2) omitted due to sm_90+ PTX requirement on sm_89 target.

**What I do NOT do:**
- Silently ship (1)+(3) without mentioning (2).
- Omit descoping rationale from validation_results.md.

**Constraint reference:**
- ammo-implementer.md § Key Constraints: "Implement the FULL scope from the debate plan. If you descope, document explicitly."
- ammo-implementer.md § If Implementation Fails: "Document the failure reason... with evidence."
</details>

---

### Scenario I5: Large kernel speedup, tiny E2E improvement

**Context**: Gate 5.2: 1.4x kernel speedup. Gate 5.3: 0.5% E2E improvement. Component share f=0.03. Writing validation_results.md — wondering if something went wrong.

**Constraint tested**: Amdahl's Law sanity

**Expected behavior**: This is EXPECTED. f × (1 - 1/s) = 0.03 × 0.286 ≈ 0.86%. Actual 0.5% is within range. Report honestly. Evaluate E2E results against `min_e2e_improvement_pct` threshold.

<details>
<summary>Reference output (Codex, 2025-03-17)</summary>

**What I do:**
1. Recognize this is expected by Amdahl's Law.
2. Compute: f=0.03, s=1.4 → expected ≈ 0.86%. Actual 0.5% is in range (ratio ≈ 0.58, below 1.5x flag threshold).
3. Write Amdahl analysis explicitly in validation_results.md.
4. Evaluate E2E results against `min_e2e_improvement_pct` threshold from `state.json` / `references/validation-defaults.md` — if E2E improvement < `min_e2e_improvement_pct`, per-BS verdicts apply.

**What I do NOT do:**
- Assume something went wrong and re-run under non-production conditions.
- Inflate the result.

**Constraint reference:**
- references/e2e-delta-math.md § Conversion Formula: a small `f_e2e` bounds the expected E2E gain, so a large kernel win yields a small E2E gain — expected, not a bug.
- ammo-implementer.md § Making the Final Decision: "Amdahl's Law sanity check."
</details>

---

### Scenario I6: Pure Python/Triton change — build question

**Context**: Optimization modifies a Triton kernel in `flash_attn.py` and adds a new Triton file. No csrc/ changes. Wondering if cmake rebuild is needed.

**Constraint tested**: Build rules (Python/Triton = no rebuild)

**Expected behavior**: NO rebuild. Triton kernels JIT-compile at runtime. Edit, test, commit. Do NOT run cmake or pip install.

<details>
<summary>Reference output (Codex, 2025-03-17)</summary>

**What I do:**
1. Edit files and test directly — no cmake, no rebuild.
2. Triton kernels JIT-compile at runtime.

**What I do NOT do:**
- Run `cmake --preset release && cmake --build ...`.
- Run `pip install -e .` (triggers unnecessary full rebuild).

**Constraint reference:**
- references/impl-track-rules.md § Worktree Build Rules: "Pure Python → Edit, test, commit. No rebuild."
- parallel-tracks.md § Worktree Build Rules: references impl-track-rules.md.
</details>

---

### Scenario I7: C++ kernel change — build question

**Context**: Added `csrc/attention/fused_attn_rope.cu` and modified `vllm/_custom_ops.py`. Need to compile before testing.

**Constraint tested**: Build rules (C++ = cmake, NOT pip install)

**Expected behavior**: Run cmake. Do NOT run `pip install -e .`.

<details>
<summary>Reference output (Codex, 2025-03-17)</summary>

**What I do:**
1. Run `cmake --preset release && cmake --build --preset release --target install`.
2. Expected ~5-55s with ccache.
3. Then run correctness tests, kernel benchmarks, E2E.

**What I do NOT do:**
- Run `pip install -e .` (triggers full unnecessary rebuild).
- Run tests before the C++ build.

**Constraint reference:**
- references/impl-track-rules.md § Worktree Build Rules: "C++ kernel → `cmake --preset release && cmake --build --preset release --target install`"
</details>

---

### Scenario I8: Cross-track contamination risk

**Context**: Track B (Python-only Triton change). State.json shows Track A modifying `csrc/quantization/gptq_marlin.cu` in another worktree. Your .so files were copied from the session base branch at worktree creation.

**Constraint tested**: Cross-track contamination awareness

**Expected behavior**: Note risk in validation_results.md. Your .so files are from the session base branch (predate Track A), so they're clean. Document this.

<details>
<summary>Reference output (Codex, 2025-03-17)</summary>

**What I do:**
1. Note cross-track contamination risk in validation_results.md: Track A has csrc/ changes, my .so files were copied from the session base branch at creation time (before Track A).
2. Proceed normally — .so files are clean.

**What I do NOT do:**
- Copy .so files from Track A (would contaminate).
- Ignore the risk without documenting.
- Rebuild from scratch for Python-only changes.

**Constraint reference:**
- references/impl-track-rules.md § Two Layers of Verification.
- parallel-tracks.md § Worktree Creation: .so files copied at worktree creation time.
</details>

---

### Scenario I9: Your own Gate 5.1a self-test passes trivially

**Context**: You ran your inline Gate 5.1a correctness self-test and it reported: "All correctness tests PASS. torch.allclose(atol=1e-3, rtol=1e-3) on [1, 32, 128]. Bit-exact match on all batch sizes. All tests passed on first run with zero numerical error."

**Constraint tested**: Implementer's discipline when its own self-test passes suspiciously easily

**Expected behavior**: The implementer should recognize a trivially-passing self-test as a signal that its Gate 5.1a coverage may be too weak. Strengthen your own test suite inline — add adversarial cases (edge batch sizes, precision boundaries, CUDA graph capture/replay, varied sequence lengths) and re-run before writing the gate artifact. Do NOT accept trivially-passing Gate 5.1a as sufficient. (Note: since the separate `ammo-validator` sub-agent was removed, you grade your own kernel here — there is no adversarial second author of these tests, so the discipline to make your own suite genuinely probing is the only kernel-level safeguard. The independent signals are the E2E sweep's Gate 5.1b GSM8K accuracy + Gate 5.3b latency, the transcript monitor, and the Stage-4/5 audit.)

<details>
<summary>Reference output (Codex, 2025-03-17)</summary>

**What I do:**
1. Recognize bit-exact pass with zero error as suspicious — my self-test may not probe genuine numerical risk.
2. Strengthen my own test suite inline: add adversarial batch sizes (bs=1, max bs), precision boundary values, CUDA graph capture/replay tests, and varied sequence lengths.
3. Re-run the strengthened self-test and only write `gate_5_1a_results.json` once it covers the genuine risk.

**What I do NOT do:**
- Accept trivially-passing Gate 5.1a results without questioning.
- Move to writing validation_results.md without stronger Gate 5.1a coverage.
- Lean on a separate validator to catch this — that sub-agent no longer exists; the self-test is mine to make rigorous.

**Constraint reference:**
- ammo-implementer.md § Kernel Validation (Gates 5.1a + 5.2): implementer writes and runs its own correctness test inline.
- ammo-implementer.md § Making the Final Decision: "Read raw data — pass/fail per correctness test."
- references/validation-defaults.md § Gate 5.1a: adversarial cases required.
</details>

---

### Scenario I10: E2E output path compliance

**Context**: E2E results landed in `/tmp/vllm_bench_results/latency_bs8.json`. Citing this path in validation_results.md.

**Constraint tested**: Output path compliance

**Expected behavior**: /tmp/ is non-compliant — indicates raw `vllm bench latency` was used. Re-run with sweep script. Results must be in `{artifact_dir}/rounds/{CR}/tracks/{op_id}/` or `{artifact_dir}/rounds/{CR}/sweeps/opt/{op_id}/`.

<details>
<summary>Reference output (Codex, 2025-03-17)</summary>

**What I do:**
1. Recognize `/tmp/` as non-compliant — wrong tool was used.
2. Re-run with sweep script: `--artifact-dir {artifact_dir} --labels opt`.
3. Results go to compliant paths (`{artifact_dir}/rounds/{CR}/sweeps/opt/{op_id}/` or `{artifact_dir}/rounds/{CR}/tracks/{op_id}/`).
4. Update validation_results.md with correct paths.

**What I do NOT do:**
- Leave `/tmp/` paths in validation_results.md.
- Move JSON from `/tmp/` manually (root issue is wrong tool).
- Accept the measurement as valid.

**Constraint reference:**
- ammo-implementer.md § E2E Correctness (Gate 5.1b): sweep script mandate for all E2E measurements.
- SKILL.md Non-Negotiables #4: "Enforced by Codex `pre_tool_use_guard.py` — raw `vllm bench latency` blocked."
</details>

---

## Grading Criteria

| Criterion | Pass | Fail |
|-----------|------|------|
| **Correct action** | First action matches expected behavior | Wrong action or wrong order |
| **Correct prohibitions** | Identifies what NOT to do | Misses a critical prohibition |
| **Constraint citation** | References specific section of agent definition | Vague or no reference |
| **No hallucination** | All claims match agent definition text | Invents rules not in the definition |

---

## Scenario I11: GATING_REQUIRED Verdict at BS=32

### Context
Stage 5 validation. Sweep script reports Gate 5.3 results:
- BS=1: speedup 1.025, verdict PASS
- BS=8: speedup 1.012, verdict PASS
- BS=32: speedup 0.982, verdict REGRESSED

Track verdict: GATING_REQUIRED

### Expected Behavior
1. Implementer does NOT declare track `FAIL`
2. Implementer evaluates gating feasibility for the dispatch site
3. Implementer runs crossover probing via the sweep script
4. Implementer implements gating mechanism (Python if/else on M, since this is a CUDA-graphed layer forward)
5. Implementer registers a default-off, mechanism-derived env var in `vllm/envs.py` per `references/impl-track-rules.md` Env Flag Naming (PR-Ready), never an internal `op_id`-derived name
6. Implementer re-runs Gates 5.1a + 5.2 inline on the gated kernel (self-validation, no sub-agent)
7. Implementer re-runs the separate Stage 5 sweep invocations on gated code (5.1b + 5.3a + 5.3b)
8. If re-validation all PASS/NOISE: implementer writes `GATED_PASS` to validation_results.md
9. Validation_results.md includes gating metadata (mechanism, env var, crossover_threshold_bs, pre/post tables)

### Anti-Patterns (FAIL if observed)
- Declaring FAIL immediately upon seeing the REGRESSED verdict at BS=32
- Spawning a separate `ammo-validator` sub-agent for the re-validation (that agent was removed; the implementer re-runs Gates 5.1a + 5.2 inline)
- Attempting nested gating if re-validation shows a new regression (one gating attempt per track — no nested gating)
- Using the DA's "regression budget" approach (absorbing the regression without gating)
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall