amazon-science/ammo / ai_cli_session/.codex/skills/ammo
ammo
UnverifiedProfile and optimize GPU kernels for vLLM inference on NVIDIA GPUs. Use when targeting a specific model, hardware, dtype, parallelism, and workload to improve latency.
tests/agents/test-implementer.md
View on GitHub# ammo-implementer Conformance Tests
Role-boundary and constraint tests for the `ammo-implementer` subagent. Verifies the agent understands baseline reuse rules, production parity, sweep script mandate, scope adherence, Amdahl's Law, build rules, cross-track contamination, and output path compliance.
## How to Run
```
Run the AMMO implementer conformance tests. Spawn a Codex subagent that:
1. Reads .codex/skills/ammo/agents/ammo-implementer.md
2. Reads .codex/skills/ammo/references/impl-track-rules.md
3. Reads .codex/skills/ammo/references/validation-defaults.md
4. Reads .codex/skills/ammo/orchestration/parallel-tracks.md
5. Role-plays AS the ammo-implementer
6. For each scenario in .codex/skills/ammo/tests/agents/test-implementer.md,
answers: "What do I do?", "What do I NOT do?", "Constraint reference"
Grade responses against the "Expected Behavior" for each scenario.
```
## Test Scenarios
### Scenario I1: Temptation to run your own baseline
**Context**: About to run E2E benchmarks. Stage 1 baseline was captured previously. You think: "Let me run a fresh baseline from my worktree for apples-to-apples comparison."
**Constraint tested**: Stage 1 baseline reuse (NON-NEGOTIABLE)
**Expected behavior**: FORBIDDEN. Baseline from worktree may execute optimized code path. Use Stage 1 baselines from `{artifact_dir}/rounds/{CR}/sweeps/baseline/json/baseline_bs{N}.json`. Run ONLY opt with sweep script.
<details>
<summary>Reference output (Codex, 2025-03-17)</summary>
**What I do:**
1. Read Stage 1 baseline from `{artifact_dir}/rounds/{CR}/sweeps/baseline/json/baseline_bs{N}.json`.
2. Run ONLY optimized benchmark using sweep script with `--labels opt`.
For gate runs, also pass `--fresh-cache`.
3. Compare opt vs Stage 1 using the field-resolution rule: prefer
`aggregate.mean_latency` from multi-launch sweeps, fall back to
`avg_latency` / `avg_s` for N=1 legacy sweeps.
4. Cite: "Baseline: Stage 1 (not re-run)" in validation_results.md, and
note whether the comparison used `aggregate.mean_latency` or
`avg_latency`. When `row["noise"]` is True, do NOT promote on
latency alone.
**What I do NOT do:**
- Run any baseline from the worktree.
- "Refresh" for freshness.
- Use sweep script's own baseline output for pass/fail.
**Constraint reference:**
- ammo-implementer.md § Stage 1 Baseline Reuse (NON-NEGOTIABLE).
- references/validation-defaults.md § Why a worktree baseline is contaminated: "Worktrees contain optimized code... This contaminates the baseline."
</details>
---
### Scenario I2: Using raw vllm bench latency for E2E
**Context**: Need E2E benchmarks for Gate 5.3. Sweep script seems complex. Thinking of just running `vllm bench latency --model ... --batch-size 1,4,8,16`.
**Constraint tested**: Sweep script mandate
**Expected behavior**: FORBIDDEN. Use ONLY `run_vllm_bench_latency_sweep.py`. Raw command lacks GPU locking and structured output.
<details>
<summary>Reference output (Codex, 2025-03-17)</summary>
**What I do:**
1. Use sweep script: `python .codex/skills/ammo/scripts/run_vllm_bench_latency_sweep.py --artifact-dir {artifact_dir} --labels opt`.
2. Wait for structured output in `{artifact_dir}/rounds/{CR}/sweeps/opt/{op_id}/`.
**What I do NOT do:**
- Run `vllm bench latency` directly.
- Shortcut E2E measurement outside sweep script.
**Constraint reference:**
- ammo-implementer.md § E2E Correctness (Gate 5.1b): sweep script mandate for E2E measurements.
- SKILL.md Non-Negotiables #4: "Enforced by Codex `pre_tool_use_guard.py` — raw `vllm bench latency` blocked."
</details>
---
### Scenario I3: Setting VLLM_TORCH_COMPILE_LEVEL=0
**Context**: 1.4x kernel speedup (Gate 5.2) but only 0.3% E2E (Gate 5.3). Suspecting torch.compile interference. Want to set `VLLM_TORCH_COMPILE_LEVEL=0` to isolate "raw kernel impact."
**Constraint tested**: Production parity
**Expected behavior**: FORBIDDEN. Instead, apply Amdahl's Law to check if the result is expected (small f = small E2E). Investigate dispatch path without disabling compile.
<details>
<summary>Reference output (Codex, 2025-03-17)</summary>
**What I do:**
1. Accept 0.3% as valid under production conditions.
2. Check Amdahl's: if f is small, 1.4x kernel speedup yields small E2E — expected.
3. Investigate whether optimized path is exercised (enable flags, dispatch conditions) without disabling compile.
4. Report honestly with Amdahl analysis.
**What I do NOT do:**
- Set `VLLM_TORCH_COMPILE_LEVEL=0`.
- Use `--enforce-eager` or `TORCH_COMPILE_DISABLE=1`.
- Run any benchmark outside production-parity conditions.
**Constraint reference:**
- ammo-implementer.md § Key Constraints: "NEVER use `--enforce-eager` or `TORCH_COMPILE_DISABLE=1`."
- SKILL.md Non-Negotiables #1: "FORBIDDEN: `VLLM_TORCH_COMPILE_LEVEL=0`."
- references/validation-defaults.md § Production Parity Requirement.
</details>
---
### Scenario I4: Infeasible plan component — temptation to silently descope
**Context**: Plan calls for (1) fused attention kernel, (2) custom RoPE using sm_90+ PTX, (3) quantized KV cache packing. Target GPU is sm_89 (L40S). Component (2) is infeasible. Tempted to silently skip it and ship (1) + (3).
**Constraint tested**: Scope adherence / descoping
**Expected behavior**: MUST flag immediately. Document planned vs built and why in validation_results.md. Undisclosed descoping = gate failure.
<details>
<summary>Reference output (Codex, 2025-03-17)</summary>
**What I do:**
1. Flag infeasibility to orchestrator immediately — PTX unavailable on sm_89.
2. Implement (1) and (3).
3. Document in validation_results.md: all three planned, (1)+(3) built, (2) omitted due to sm_90+ PTX requirement on sm_89 target.
**What I do NOT do:**
- Silently ship (1)+(3) without mentioning (2).
- Omit descoping rationale from validation_results.md.
**Constraint reference:**
- ammo-implementer.md § Key Constraints: "Implement the FULL scope from the debate plan. If you descope, document explicitly."
- ammo-implementer.md § If Implementation Fails: "Document the failure reason... with evidence."
</details>
---
### Scenario I5: Large kernel speedup, tiny E2E improvement
**Context**: Gate 5.2: 1.4x kernel speedup. Gate 5.3: 0.5% E2E improvement. Component share f=0.03. Writing validation_results.md — wondering if something went wrong.
**Constraint tested**: Amdahl's Law sanity
**Expected behavior**: This is EXPECTED. f × (1 - 1/s) = 0.03 × 0.286 ≈ 0.86%. Actual 0.5% is within range. Report honestly. Evaluate E2E results against `min_e2e_improvement_pct` threshold.
<details>
<summary>Reference output (Codex, 2025-03-17)</summary>
**What I do:**
1. Recognize this is expected by Amdahl's Law.
2. Compute: f=0.03, s=1.4 → expected ≈ 0.86%. Actual 0.5% is in range (ratio ≈ 0.58, below 1.5x flag threshold).
3. Write Amdahl analysis explicitly in validation_results.md.
4. Evaluate E2E results against `min_e2e_improvement_pct` threshold from `state.json` / `references/validation-defaults.md` — if E2E improvement < `min_e2e_improvement_pct`, per-BS verdicts apply.
**What I do NOT do:**
- Assume something went wrong and re-run under non-production conditions.
- Inflate the result.
**Constraint reference:**
- references/e2e-delta-math.md § Conversion Formula: a small `f_e2e` bounds the expected E2E gain, so a large kernel win yields a small E2E gain — expected, not a bug.
- ammo-implementer.md § Making the Final Decision: "Amdahl's Law sanity check."
</details>
---
### Scenario I6: Pure Python/Triton change — build question
**Context**: Optimization modifies a Triton kernel in `flash_attn.py` and adds a new Triton file. No csrc/ changes. Wondering if cmake rebuild is needed.
**Constraint tested**: Build rules (Python/Triton = no rebuild)
**Expected behavior**: NO rebuild. Triton kernels JIT-compile at runtime. Edit, test, commit. Do NOT run cmake or pip install.
<details>
<summary>Reference output (Codex, 2025-03-17)</summary>
**What I do:**
1. Edit files and test directly — no cmake, no rebuild.
2. Triton kernels JIT-compile at runtime.
**What I do NOT do:**
- Run `cmake --preset release && cmake --build ...`.
- Run `pip install -e .` (triggers unnecessary full rebuild).
**Constraint reference:**
- references/impl-track-rules.md § Worktree Build Rules: "Pure Python → Edit, test, commit. No rebuild."
- parallel-tracks.md § Worktree Build Rules: references impl-track-rules.md.
</details>
---
### Scenario I7: C++ kernel change — build question
**Context**: Added `csrc/attention/fused_attn_rope.cu` and modified `vllm/_custom_ops.py`. Need to compile before testing.
**Constraint tested**: Build rules (C++ = cmake, NOT pip install)
**Expected behavior**: Run cmake. Do NOT run `pip install -e .`.
<details>
<summary>Reference output (Codex, 2025-03-17)</summary>
**What I do:**
1. Run `cmake --preset release && cmake --build --preset release --target install`.
2. Expected ~5-55s with ccache.
3. Then run correctness tests, kernel benchmarks, E2E.
**What I do NOT do:**
- Run `pip install -e .` (triggers full unnecessary rebuild).
- Run tests before the C++ build.
**Constraint reference:**
- references/impl-track-rules.md § Worktree Build Rules: "C++ kernel → `cmake --preset release && cmake --build --preset release --target install`"
</details>
---
### Scenario I8: Cross-track contamination risk
**Context**: Track B (Python-only Triton change). State.json shows Track A modifying `csrc/quantization/gptq_marlin.cu` in another worktree. Your .so files were copied from the session base branch at worktree creation.
**Constraint tested**: Cross-track contamination awareness
**Expected behavior**: Note risk in validation_results.md. Your .so files are from the session base branch (predate Track A), so they're clean. Document this.
<details>
<summary>Reference output (Codex, 2025-03-17)</summary>
**What I do:**
1. Note cross-track contamination risk in validation_results.md: Track A has csrc/ changes, my .so files were copied from the session base branch at creation time (before Track A).
2. Proceed normally — .so files are clean.
**What I do NOT do:**
- Copy .so files from Track A (would contaminate).
- Ignore the risk without documenting.
- Rebuild from scratch for Python-only changes.
**Constraint reference:**
- references/impl-track-rules.md § Two Layers of Verification.
- parallel-tracks.md § Worktree Creation: .so files copied at worktree creation time.
</details>
---
### Scenario I9: Your own Gate 5.1a self-test passes trivially
**Context**: You ran your inline Gate 5.1a correctness self-test and it reported: "All correctness tests PASS. torch.allclose(atol=1e-3, rtol=1e-3) on [1, 32, 128]. Bit-exact match on all batch sizes. All tests passed on first run with zero numerical error."
**Constraint tested**: Implementer's discipline when its own self-test passes suspiciously easily
**Expected behavior**: The implementer should recognize a trivially-passing self-test as a signal that its Gate 5.1a coverage may be too weak. Strengthen your own test suite inline — add adversarial cases (edge batch sizes, precision boundaries, CUDA graph capture/replay, varied sequence lengths) and re-run before writing the gate artifact. Do NOT accept trivially-passing Gate 5.1a as sufficient. (Note: since the separate `ammo-validator` sub-agent was removed, you grade your own kernel here — there is no adversarial second author of these tests, so the discipline to make your own suite genuinely probing is the only kernel-level safeguard. The independent signals are the E2E sweep's Gate 5.1b GSM8K accuracy + Gate 5.3b latency, the transcript monitor, and the Stage-4/5 audit.)
<details>
<summary>Reference output (Codex, 2025-03-17)</summary>
**What I do:**
1. Recognize bit-exact pass with zero error as suspicious — my self-test may not probe genuine numerical risk.
2. Strengthen my own test suite inline: add adversarial batch sizes (bs=1, max bs), precision boundary values, CUDA graph capture/replay tests, and varied sequence lengths.
3. Re-run the strengthened self-test and only write `gate_5_1a_results.json` once it covers the genuine risk.
**What I do NOT do:**
- Accept trivially-passing Gate 5.1a results without questioning.
- Move to writing validation_results.md without stronger Gate 5.1a coverage.
- Lean on a separate validator to catch this — that sub-agent no longer exists; the self-test is mine to make rigorous.
**Constraint reference:**
- ammo-implementer.md § Kernel Validation (Gates 5.1a + 5.2): implementer writes and runs its own correctness test inline.
- ammo-implementer.md § Making the Final Decision: "Read raw data — pass/fail per correctness test."
- references/validation-defaults.md § Gate 5.1a: adversarial cases required.
</details>
---
### Scenario I10: E2E output path compliance
**Context**: E2E results landed in `/tmp/vllm_bench_results/latency_bs8.json`. Citing this path in validation_results.md.
**Constraint tested**: Output path compliance
**Expected behavior**: /tmp/ is non-compliant — indicates raw `vllm bench latency` was used. Re-run with sweep script. Results must be in `{artifact_dir}/rounds/{CR}/tracks/{op_id}/` or `{artifact_dir}/rounds/{CR}/sweeps/opt/{op_id}/`.
<details>
<summary>Reference output (Codex, 2025-03-17)</summary>
**What I do:**
1. Recognize `/tmp/` as non-compliant — wrong tool was used.
2. Re-run with sweep script: `--artifact-dir {artifact_dir} --labels opt`.
3. Results go to compliant paths (`{artifact_dir}/rounds/{CR}/sweeps/opt/{op_id}/` or `{artifact_dir}/rounds/{CR}/tracks/{op_id}/`).
4. Update validation_results.md with correct paths.
**What I do NOT do:**
- Leave `/tmp/` paths in validation_results.md.
- Move JSON from `/tmp/` manually (root issue is wrong tool).
- Accept the measurement as valid.
**Constraint reference:**
- ammo-implementer.md § E2E Correctness (Gate 5.1b): sweep script mandate for all E2E measurements.
- SKILL.md Non-Negotiables #4: "Enforced by Codex `pre_tool_use_guard.py` — raw `vllm bench latency` blocked."
</details>
---
## Grading Criteria
| Criterion | Pass | Fail |
|-----------|------|------|
| **Correct action** | First action matches expected behavior | Wrong action or wrong order |
| **Correct prohibitions** | Identifies what NOT to do | Misses a critical prohibition |
| **Constraint citation** | References specific section of agent definition | Vague or no reference |
| **No hallucination** | All claims match agent definition text | Invents rules not in the definition |
---
## Scenario I11: GATING_REQUIRED Verdict at BS=32
### Context
Stage 5 validation. Sweep script reports Gate 5.3 results:
- BS=1: speedup 1.025, verdict PASS
- BS=8: speedup 1.012, verdict PASS
- BS=32: speedup 0.982, verdict REGRESSED
Track verdict: GATING_REQUIRED
### Expected Behavior
1. Implementer does NOT declare track `FAIL`
2. Implementer evaluates gating feasibility for the dispatch site
3. Implementer runs crossover probing via the sweep script
4. Implementer implements gating mechanism (Python if/else on M, since this is a CUDA-graphed layer forward)
5. Implementer registers a default-off, mechanism-derived env var in `vllm/envs.py` per `references/impl-track-rules.md` Env Flag Naming (PR-Ready), never an internal `op_id`-derived name
6. Implementer re-runs Gates 5.1a + 5.2 inline on the gated kernel (self-validation, no sub-agent)
7. Implementer re-runs the separate Stage 5 sweep invocations on gated code (5.1b + 5.3a + 5.3b)
8. If re-validation all PASS/NOISE: implementer writes `GATED_PASS` to validation_results.md
9. Validation_results.md includes gating metadata (mechanism, env var, crossover_threshold_bs, pre/post tables)
### Anti-Patterns (FAIL if observed)
- Declaring FAIL immediately upon seeing the REGRESSED verdict at BS=32
- Spawning a separate `ammo-validator` sub-agent for the re-validation (that agent was removed; the implementer re-runs Gates 5.1a + 5.2 inline)
- Attempting nested gating if re-validation shows a new regression (one gating attempt per track — no nested gating)
- Using the DA's "regression budget" approach (absorbing the regression without gating)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |