Skip to content

amazon-science/ammo / ai_cli_session/.codex/skills/ammo

Profile and optimize GPU kernels for vLLM inference on NVIDIA GPUs. Use when targeting a specific model, hardware, dtype, parallelism, and workload to improve latency.

FileKindSize
RUNTIME.mdmarkdown4.9 KB
SKILL.mdskill md18.5 KB
agents/ammo-auditor.mdmarkdown9.1 KB
agents/ammo-champion.mdmarkdown9.6 KB
agents/ammo-delegate.mdmarkdown3.2 KB
agents/ammo-implementer.mdmarkdown13.8 KB
agents/ammo-investigator.mdmarkdown4.0 KB
agents/ammo-report.mdmarkdown2.9 KB
agents/ammo-researcher.mdmarkdown10.3 KB
agents/ammo-resolver.mdmarkdown4.4 KB
agents/ammo-transcript-monitor.mdmarkdown7.8 KB
agents/openai.yamlexecutablescript313 B
data/ATTRIBUTION.mdmarkdown1.4 KB
data/gsm8k_full.jsonexecutablescript735.5 KB
data/gsm8k_subset.jsonexecutablescript116.8 KB
eval/EVAL-SKILL.mdexecutablescript21.8 KB
eval/agents/transcript_grader.mdexecutablescript18.0 KB
eval/causal/__init__.pyexecutablescript137 B
eval/causal/agents/causal_analyzer.mdexecutablescript10.5 KB
eval/causal/build_dag.pyexecutablescript21.6 KB
eval/causal/diff_versions.pyexecutablescript38.5 KB
eval/causal/extract_events.pyexecutablescript36.3 KB
eval/causal/generate_postmortem.pyexecutablescript17.0 KB
eval/causal/score_nodes.pyexecutablescript21.5 KB
eval/causal/templates/narrative_template.mdexecutablescript472 B
eval/causal/templates/viz_template.htmlexecutablescript9.5 KB
eval/causal/tests/__init__.pyexecutablescript151 B
eval/causal/tests/conftest.pyexecutablescript448 B
eval/causal/tests/fixtures/sample_session.jsonlexecutablescript10.4 KB
eval/causal/tests/fixtures/sample_session_data.jsonexecutablescript3.4 KB
eval/causal/tests/fixtures/sample_snapshot.jsonexecutablescript2.5 KB
eval/causal/tests/test_build_dag.pyexecutablescript2.7 KB
eval/causal/tests/test_diff_versions.pyexecutablescript13.5 KB
eval/causal/tests/test_extract_events.pyexecutablescript3.7 KB
eval/causal/tests/test_generate_postmortem.pyexecutablescript3.1 KB
eval/causal/tests/test_score_nodes.pyexecutablescript3.3 KB
eval/scripts/aggregate_versions.pyexecutablescript15.2 KB
eval/scripts/archive_run.pyexecutablescript9.1 KB
eval/scripts/codex_session.pyexecutablescript10.6 KB
eval/scripts/generate_dashboard.pyexecutablescript5.3 KB
eval/scripts/parse_artifacts.pyexecutablescript48.1 KB
eval/scripts/parse_session_logs.pyexecutablescript51.3 KB
eval/scripts/rescore_archived.pyexecutablescript5.5 KB
eval/scripts/score_campaign.pyexecutablescript43.8 KB
eval/scripts/snapshot_changes.pyexecutablescript11.1 KB
eval/scripts/verify_citations.pyexecutablescript8.8 KB
eval/tests/test_dashboard_bugs.pyexecutablescript8.8 KB
eval/tests/test_parse_session_logs.pyexecutablescript3.9 KB
eval/tests/test_parse_session_logs_round_inference.pyexecutablescript11.0 KB
eval/tests/test_score_campaign_diluted.pyexecutablescript9.4 KB
eval/viewer/dashboard_template.htmlexecutablescript15.1 KB
instrumentation/sitecustomize.pyexecutablescript3.6 KB
orchestration/audit-protocol.mdmarkdown9.3 KB
orchestration/debate-protocol.mdmarkdown11.2 KB
orchestration/integration-logic.mdmarkdown10.3 KB
orchestration/parallel-tracks.mdmarkdown8.9 KB
references/artifact-layout.mdexecutablescript22.4 KB
references/audit-invariants.mdmarkdown10.6 KB
references/champion-common-patterns.mdmarkdown3.4 KB
references/crossover-probing.mdexecutablescript4.7 KB
references/cudagraph-safety.mdmarkdown2.7 KB
references/debate-rules.mdmarkdown4.7 KB
references/debate-scoring-rubric.mdexecutablescript16.7 KB
references/e2e-delta-math.mdexecutablescript17.9 KB
references/e2e-latency-guide.mdexecutablescript9.6 KB
references/fusion-feasibility-heuristics.mdmarkdown5.6 KB
references/gpu-pool.mdexecutablescript5.4 KB
references/impl-track-rules.mdexecutablescript13.8 KB
references/nsys-profiling-guide.mdmarkdown9.8 KB
references/optimization-categories.mdexecutablescript11.0 KB
references/shutdown-protocol.mdexecutablescript3.1 KB
references/technology-selection.mdmarkdown5.0 KB
references/torch-compile-contract.mdmarkdown13.8 KB
references/validation-defaults.mdexecutablescript34.2 KB
references/workload-invocation.mdexecutablescript4.9 KB
references/writing-style.mdexecutablescript6.9 KB
scripts/ammo_state.pyexecutablescript142.0 KB
scripts/artifact_layout.jsonconfig16.3 KB
scripts/check_projection_accuracy.pyexecutablescript11.7 KB
scripts/create_evidence_template.pyexecutablescript9.7 KB
scripts/create_worktree_with_build.shexecutablescript10.2 KB
scripts/generate_validation_report.pyexecutablescript35.7 KB
scripts/gpu_reservation.pyexecutablescript27.1 KB
scripts/hook_cmd_classify.pyscript13.3 KB
scripts/mine_trace.pyscript29.4 KB
scripts/monitor_queue_ack.pyexecutablescript6.1 KB
scripts/ncu_sanity.shexecutablescript5.2 KB
scripts/ncu_sanity_driver.pyexecutablescript8.5 KB
scripts/new_target.pyexecutablescript22.0 KB
scripts/preflight_vllm_api.pyexecutablescript12.3 KB
scripts/reconcile_track_state.pyexecutablescript15.8 KB
scripts/remove_worktree_cleanup.shexecutablescript1.1 KB
scripts/resolve_codex_transcript.pyexecutablescript3.6 KB
scripts/run_vllm_bench_latency_sweep.pyexecutablescript249.6 KB
scripts/transcript_filter.pyexecutablescript38.4 KB
scripts/transitions.jsonexecutablescript1.5 KB
scripts/verify_validation_gates.pyexecutablescript88.8 KB
tests/agents/test-champion.mdexecutablescript20.4 KB
tests/agents/test-impl-champion.mdexecutablescript14.3 KB
tests/agents/test-implementer.mdexecutablescript15.8 KB
tests/agents/test-orchestrator.mdexecutablescript33.8 KB
tests/agents/test-researcher.mdexecutablescript21.0 KB
tests/agents/test-transcript-monitor.mdexecutablescript13.4 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.mdmarkdown1.1 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.summary.jsonconfig582 B
tests/fixtures/diluted_pass/legacy_no_phase.jsonconfig400 B
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.mdmarkdown2.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.summary.jsonconfig1.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.jsonconfig626 B
tests/fixtures/mining/real_campaign_bottleneck_analysis.mdmarkdown20.7 KB
tests/run_all.shexecutablescript1.6 KB
tests/scripts/test_transcript_filter.pyexecutablescript41.1 KB
tests/test_agent_runtime_contracts.pyexecutablescript1.8 KB
tests/test_ammo_state.pyexecutablescript71.7 KB
tests/test_artifact_layout_manifest.pyscript11.6 KB
tests/test_audit_started.pyscript18.2 KB
tests/test_correctness_comparator.pyscript12.6 KB
tests/test_diluted_pass.pyscript24.4 KB
tests/test_drift_compat.pyscript5.1 KB
tests/test_evidence_freshness.pyexecutablescript5.6 KB
tests/test_fail_closed.pyexecutablescript14.7 KB
tests/test_gpu_reservation.pyexecutablescript24.8 KB
tests/test_gpu_reservation_session_id.pyexecutablescript11.2 KB
tests/test_guidance_semantic_contract.pyscript24.9 KB
tests/test_hook_cmd_classify_parity.pyscript11.7 KB
tests/test_hook_semantics.pyexecutablescript95.3 KB
tests/test_mine_trace.pyscript24.2 KB
tests/test_mining_invalidated_schema.pyexecutablescript9.4 KB
tests/test_new_target_dp_ep.pyexecutablescript7.4 KB
tests/test_new_target_v2_scaffold.pyexecutablescript13.1 KB
tests/test_parse_artifacts.pyexecutablescript3.0 KB
tests/test_policy_convergence.pyscript797 B
tests/test_preflight_vllm_api.pyscript13.3 KB
tests/test_projection_accuracy_v2.pyexecutablescript4.1 KB
tests/test_reconcile_track_state.pyexecutablescript18.2 KB
tests/test_resolve_codex_transcript.pyexecutablescript3.1 KB
tests/test_state_schema_semantics.pyexecutablescript7.6 KB
tests/test_status_set_sync.pyexecutablescript9.1 KB
tests/test_sweep_baseline_from_arm.pyscript11.0 KB
tests/test_sweep_decode_capture.pyexecutablescript31.8 KB
tests/test_sweep_dp.pyexecutablescript45.4 KB
tests/test_sweep_fresh_cache.pyscript5.5 KB
tests/test_sweep_profiler_guard.pyscript22.6 KB
tests/test_sweep_real_prompts.pyscript20.1 KB
tests/test_sweep_refactor.pyexecutablescript37.6 KB
tests/test_sweep_round_slot.pyexecutablescript7.4 KB
tests/test_validation_report_phase_triage.pyexecutablescript10.0 KB
tests/test_validation_report_v2.pyexecutablescript6.2 KB
tests/test_verify_validation_gates.pyexecutablescript52.9 KB
tests/test_workload_matrix.pyexecutablescript19.3 KB

references/e2e-latency-guide.md

View on GitHub
# E2E Latency Benchmark Guide (vLLM) for Kernel Optimizations

Read this before you measure any end-to-end latency number. It tells you how to run the vLLM E2E benchmarks and how to read what they print. Use them to show that a kernel optimization improves *real* inference latency under **production parity**:
- CUDA graphs enabled (or the exact mode used in production)
- torch.compile enabled (or the exact mode used in production)
- same TP/EP topology and serving knobs

Default gates and required reporting are not in this file. They live in `references/validation-defaults.md`.

## Contents
- Tool Selection (which invocation each stage uses)
- Quickstart (baseline vs optimized)
- Workload selection (decode-heavy vs prefill-heavy)
- Batch-size sweep
- Parity checklist (must match baseline)
- Interpreting output + speedup math
- Troubleshooting
- Recording results

## Tool Selection

Pick the invocation that matches your stage.

| Stage | Tool | Why |
|-------|------|-----|
| Stage 1 (baseline) | `run_vllm_bench_latency_sweep.py --round {N} --slot baseline --labels baseline --capture-golden-refs` | Clean E2E timing — **no profiling flags allowed** (guard enforced). Authoritative numbers for speedup calculations. |
| Stage 1 (profiling) | `run_vllm_bench_latency_sweep.py --round {N} --slot profiling --nsys-profile --nsys-mode node --nsys-capture-output-steps 2,50%,100% --nsys-num-iters 1 --nsys-timeout-s 1800` | Bounded selected-step nsys attribution capture. The sweep resolves `--nsys-trace` from `target.hardware` (`cuda-sw` on Blackwell). E2E numbers here are profiler-contaminated and not used for comparisons. See `nsys-profiling-guide.md`. |
| Stage 5 (validation) | `run_vllm_bench_latency_sweep.py --round {N} --slot opt/{op_id} --labels opt --baseline-from $STAGE1_DIR --fresh-cache` | GPU-locked optimized-only sweep compared against the Stage 1 baseline (never re-run a baseline from the worktree — § E2E Baseline Reuse in `validation-defaults.md`) |
| Stage 6 (integration) | `run_vllm_bench_latency_sweep.py --round {N} --slot integration --labels opt --baseline-from $STAGE1_DIR --fresh-cache` (round >= 2 adds `--baseline-from-arm opt`) | Gate-quality combined measurement vs the Stage 1 baseline when two or more tracks pass; a single unchanged passer uses the short-circuit. EXHAUSTED rounds skip it. Round >= 2 MUST import the promoted arm, or the delta is cumulative, not incremental — see § Combined Validation Workflow in `orchestration/integration-logic.md`. |

For all measurements reported in `validation_results.md` or used for profiling, use the sweep script.

## Quickstart

Run the sweep script from the worktree root, and always pass the artifact directory explicitly:

```bash
# Stage 1a: Clean E2E baseline (no profiling — authoritative timing)
.venv/bin/python .codex/skills/ammo/scripts/run_vllm_bench_latency_sweep.py \
  --artifact-dir {artifact_dir} --round 1 --slot baseline --labels baseline \
  --capture-golden-refs

# Stage 1b: Profiling traces (separate invocation — overhead doesn't pollute baseline)
.venv/bin/python .codex/skills/ammo/scripts/run_vllm_bench_latency_sweep.py \
  --artifact-dir {artifact_dir} --round 1 --slot profiling --labels baseline \
  --nsys-profile --nsys-mode node \
  --nsys-capture-output-steps 2,50%,100% \
  --nsys-num-iters 1 --nsys-timeout-s 1800

# Stage 5: Per-track validation sweep (opt-only vs Stage 1 baseline, isolated cache)
.venv/bin/python .codex/skills/ammo/scripts/run_vllm_bench_latency_sweep.py \
  --artifact-dir {artifact_dir} --round {N} --slot opt/{op_id} \
  --labels opt --baseline-from {artifact_dir}/rounds/{N}/sweeps/baseline --fresh-cache

# Stage 6: Integration sweep (--fresh-cache for gate-quality measurement)
# This replaces the former T16 re-profile step.
# Round >= 2 MUST append `--baseline-from-arm opt` to the command below: the
# comparator is then the promoted arm, not that directory's stale pre-SHIP
# baseline arm. Without it the measured delta is cumulative, not incremental.
.venv/bin/python .codex/skills/ammo/scripts/run_vllm_bench_latency_sweep.py \
  --artifact-dir {artifact_dir} --round {N} --slot integration \
  --labels opt --baseline-from {artifact_dir}/rounds/{N}/sweeps/baseline --fresh-cache

# Post-SHIP: Golden-refs capture after environment promotion
.venv/bin/python .codex/skills/ammo/scripts/run_vllm_bench_latency_sweep.py \
  --artifact-dir {artifact_dir} --round {N} --slot golden_capture \
  --labels baseline --capture-golden-refs
```

Do not put the workload on the command line. The sweep script reads model, workload, and env config from `target.json`, so you never need `--model`, `--dtype`, `--batch-size`, or the rest.

### Fresh-cache isolation (`--fresh-cache`, v3.1)

This flag keeps a sweep from inheriting warm compile caches
from a previous sweep. It allocates `{out_root}/cache/{sweep_id}/`
and injects `VLLM_CACHE_ROOT` and `TRITON_CACHE_DIR` into the child
env. The cache is removed on success. First launch of N pays full
compile (~5 min for large models); launches 2..N hit the warm
in-sweep cache.

## Workload selection

### Decode-heavy (recommended for decode-bucket optimizations)

Benchmark decode-heavy when the optimization targets **decode buckets** (small `M` per step). Many optimization fast-paths are tuned for those buckets, and a decode-heavy benchmark makes that visible:

- `--input-len 64` (short prefill)
- `--output-len 512` (long decode)
- Sweep your decode bucket `--batch-size` set

### Prefill-heavy (optional)

Run a second, prefill-heavy benchmark if you claim a prefill win — large input length, and usually a smaller output length. Keep those results separate from the decode-heavy results.

## Batch-size sweep

Sweep the **same bucket set** you profiled in Stage 1 and plan to enable in Stage 6.

Let the sweep script do the multi-bucket runs; it loads the model once per label. Pass `--round {N} --slot {SLOT}` so results land in the canonical round-scoped layout:

```bash
.venv/bin/python .codex/skills/ammo/scripts/run_vllm_bench_latency_sweep.py \
  --artifact-dir {artifact_dir} --round {CR} --slot baseline
```

`{SLOT}` is one of: `baseline` (Stage 1), `opt/{op_id}` (Stage 5 per-track), `integration` (Stage 6 combined sweep), `golden_capture` (post-SHIP golden-ref refresh).

The script takes the workload config from `target.json`. Both formats work: the flat
format (`input_len`, `output_len`, `batch_sizes`), and `workload_matrix` for
multi-dimensional `(input_len x output_len x batch_size)` sweeps.

Per-bucket nsys profiles need a **separate short profiling invocation** (`--slot profiling`), run after the clean baseline. `--slot baseline` plus any profiler flag is a hard error, so profiling overhead can never contaminate authoritative timing. The E2E numbers from a profiling invocation are themselves contaminated by nsys overhead and are NOT authoritative. For the exact command, the architecture trace backend (Blackwell `cuda-sw`), selected-step capture semantics, and the bounded `--nsys-output-len 2` fallback, see `nsys-profiling-guide.md § Required Stage 1 Commands`.

## Parity checklist (must match baseline)

Your numbers are not trustworthy until every item below holds.

- Same model weights + same revision/commit.
- Same dtype/quantization (FP8 formats and scale shapes matter).
- Same TP/EP topology and identical routing/dispatch mode.
- Same CUDA graphs mode and torch.compile mode.
- Same scheduler knobs that affect bucketing (e.g., max batched tokens / chunked prefill).
- Confirm **optimized path actually executed**, with one of:
  - enablement log line, or
  - instrumentation counter, or
  - an unmistakable kernel name in Nsight Systems.

### Debug-only: run eager

Use eager mode only to debug correctness or functional issues. Never use it for a production-parity perf claim.

```bash
vllm bench latency --enforce-eager ...
```

## Interpreting output

`vllm bench latency` prints iteration latencies. The value you want is usually **Avg latency**.

Compute speedup and improvement:

```python
baseline_s = 10.95
opt_s = 10.20

speedup = baseline_s / opt_s
improvement_pct = (baseline_s - opt_s) / baseline_s * 100
```

If your measured E2E improvement is small, sanity-check the component share with `references/e2e-delta-math.md`.

## Troubleshooting

### Optimized path not activating
- Verify the enable flag is set (env var / config).
- Verify a compiled specialization exists for your `(dtype, TP/EP, bucket set)`.
- Verify your bucket guard matches the validated envelope.
- Use Nsight Systems to confirm which kernels run under the captured graph.

### "No E2E win" even though microbench is faster
Common causes:
- The target component is a small fraction of end-to-end (`f` small), so the expected E2E gain is bounded (see `references/e2e-delta-math.md`).
- Graph breaks or unexpected fallbacks (different kernels between baseline and optimized runs).
- Another bottleneck dominates (attention/KV/cache/scheduler).

### High variance / poor reproducibility
- Increase iterations.
- Isolate the GPU: no other jobs, and no power or thermal limit.
- Use a consistent warmup protocol.

## Recording results

Write the Stage 5 per-track results to `{artifact_dir}/rounds/{CR}/tracks/{op_id}/validation_results.md` with:
- full repro commands (baseline + optimized) and env vars
- the bucket set and capture/compile settings
- baseline vs optimized tables (speedup + improvement)
- evidence that the optimized path executed

The Stage 6 integration record is `{artifact_dir}/rounds/{CR}/sweeps/integration/e2e_latency_results.json`, written by the sweep script. `references/artifact-layout.md` owns every artifact path.

Use `references/validation-defaults.md` for the minimum reporting template and default gates.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall