Skip to content

amazon-science/ammo / ai_cli_session/.codex/skills/ammo

Profile and optimize GPU kernels for vLLM inference on NVIDIA GPUs. Use when targeting a specific model, hardware, dtype, parallelism, and workload to improve latency.

FileKindSize
RUNTIME.mdmarkdown4.9 KB
SKILL.mdskill md18.5 KB
agents/ammo-auditor.mdmarkdown9.1 KB
agents/ammo-champion.mdmarkdown9.6 KB
agents/ammo-delegate.mdmarkdown3.2 KB
agents/ammo-implementer.mdmarkdown13.8 KB
agents/ammo-investigator.mdmarkdown4.0 KB
agents/ammo-report.mdmarkdown2.9 KB
agents/ammo-researcher.mdmarkdown10.3 KB
agents/ammo-resolver.mdmarkdown4.4 KB
agents/ammo-transcript-monitor.mdmarkdown7.8 KB
agents/openai.yamlexecutablescript313 B
data/ATTRIBUTION.mdmarkdown1.4 KB
data/gsm8k_full.jsonexecutablescript735.5 KB
data/gsm8k_subset.jsonexecutablescript116.8 KB
eval/EVAL-SKILL.mdexecutablescript21.8 KB
eval/agents/transcript_grader.mdexecutablescript18.0 KB
eval/causal/__init__.pyexecutablescript137 B
eval/causal/agents/causal_analyzer.mdexecutablescript10.5 KB
eval/causal/build_dag.pyexecutablescript21.6 KB
eval/causal/diff_versions.pyexecutablescript38.5 KB
eval/causal/extract_events.pyexecutablescript36.3 KB
eval/causal/generate_postmortem.pyexecutablescript17.0 KB
eval/causal/score_nodes.pyexecutablescript21.5 KB
eval/causal/templates/narrative_template.mdexecutablescript472 B
eval/causal/templates/viz_template.htmlexecutablescript9.5 KB
eval/causal/tests/__init__.pyexecutablescript151 B
eval/causal/tests/conftest.pyexecutablescript448 B
eval/causal/tests/fixtures/sample_session.jsonlexecutablescript10.4 KB
eval/causal/tests/fixtures/sample_session_data.jsonexecutablescript3.4 KB
eval/causal/tests/fixtures/sample_snapshot.jsonexecutablescript2.5 KB
eval/causal/tests/test_build_dag.pyexecutablescript2.7 KB
eval/causal/tests/test_diff_versions.pyexecutablescript13.5 KB
eval/causal/tests/test_extract_events.pyexecutablescript3.7 KB
eval/causal/tests/test_generate_postmortem.pyexecutablescript3.1 KB
eval/causal/tests/test_score_nodes.pyexecutablescript3.3 KB
eval/scripts/aggregate_versions.pyexecutablescript15.2 KB
eval/scripts/archive_run.pyexecutablescript9.1 KB
eval/scripts/codex_session.pyexecutablescript10.6 KB
eval/scripts/generate_dashboard.pyexecutablescript5.3 KB
eval/scripts/parse_artifacts.pyexecutablescript48.1 KB
eval/scripts/parse_session_logs.pyexecutablescript51.3 KB
eval/scripts/rescore_archived.pyexecutablescript5.5 KB
eval/scripts/score_campaign.pyexecutablescript43.8 KB
eval/scripts/snapshot_changes.pyexecutablescript11.1 KB
eval/scripts/verify_citations.pyexecutablescript8.8 KB
eval/tests/test_dashboard_bugs.pyexecutablescript8.8 KB
eval/tests/test_parse_session_logs.pyexecutablescript3.9 KB
eval/tests/test_parse_session_logs_round_inference.pyexecutablescript11.0 KB
eval/tests/test_score_campaign_diluted.pyexecutablescript9.4 KB
eval/viewer/dashboard_template.htmlexecutablescript15.1 KB
instrumentation/sitecustomize.pyexecutablescript3.6 KB
orchestration/audit-protocol.mdmarkdown9.3 KB
orchestration/debate-protocol.mdmarkdown11.2 KB
orchestration/integration-logic.mdmarkdown10.3 KB
orchestration/parallel-tracks.mdmarkdown8.9 KB
references/artifact-layout.mdexecutablescript22.4 KB
references/audit-invariants.mdmarkdown10.6 KB
references/champion-common-patterns.mdmarkdown3.4 KB
references/crossover-probing.mdexecutablescript4.7 KB
references/cudagraph-safety.mdmarkdown2.7 KB
references/debate-rules.mdmarkdown4.7 KB
references/debate-scoring-rubric.mdexecutablescript16.7 KB
references/e2e-delta-math.mdexecutablescript17.9 KB
references/e2e-latency-guide.mdexecutablescript9.6 KB
references/fusion-feasibility-heuristics.mdmarkdown5.6 KB
references/gpu-pool.mdexecutablescript5.4 KB
references/impl-track-rules.mdexecutablescript13.8 KB
references/nsys-profiling-guide.mdmarkdown9.8 KB
references/optimization-categories.mdexecutablescript11.0 KB
references/shutdown-protocol.mdexecutablescript3.1 KB
references/technology-selection.mdmarkdown5.0 KB
references/torch-compile-contract.mdmarkdown13.8 KB
references/validation-defaults.mdexecutablescript34.2 KB
references/workload-invocation.mdexecutablescript4.9 KB
references/writing-style.mdexecutablescript6.9 KB
scripts/ammo_state.pyexecutablescript142.0 KB
scripts/artifact_layout.jsonconfig16.3 KB
scripts/check_projection_accuracy.pyexecutablescript11.7 KB
scripts/create_evidence_template.pyexecutablescript9.7 KB
scripts/create_worktree_with_build.shexecutablescript10.2 KB
scripts/generate_validation_report.pyexecutablescript35.7 KB
scripts/gpu_reservation.pyexecutablescript27.1 KB
scripts/hook_cmd_classify.pyscript13.3 KB
scripts/mine_trace.pyscript29.4 KB
scripts/monitor_queue_ack.pyexecutablescript6.1 KB
scripts/ncu_sanity.shexecutablescript5.2 KB
scripts/ncu_sanity_driver.pyexecutablescript8.5 KB
scripts/new_target.pyexecutablescript22.0 KB
scripts/preflight_vllm_api.pyexecutablescript12.3 KB
scripts/reconcile_track_state.pyexecutablescript15.8 KB
scripts/remove_worktree_cleanup.shexecutablescript1.1 KB
scripts/resolve_codex_transcript.pyexecutablescript3.6 KB
scripts/run_vllm_bench_latency_sweep.pyexecutablescript249.6 KB
scripts/transcript_filter.pyexecutablescript38.4 KB
scripts/transitions.jsonexecutablescript1.5 KB
scripts/verify_validation_gates.pyexecutablescript88.8 KB
tests/agents/test-champion.mdexecutablescript20.4 KB
tests/agents/test-impl-champion.mdexecutablescript14.3 KB
tests/agents/test-implementer.mdexecutablescript15.8 KB
tests/agents/test-orchestrator.mdexecutablescript33.8 KB
tests/agents/test-researcher.mdexecutablescript21.0 KB
tests/agents/test-transcript-monitor.mdexecutablescript13.4 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.mdmarkdown1.1 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.summary.jsonconfig582 B
tests/fixtures/diluted_pass/legacy_no_phase.jsonconfig400 B
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.mdmarkdown2.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.summary.jsonconfig1.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.jsonconfig626 B
tests/fixtures/mining/real_campaign_bottleneck_analysis.mdmarkdown20.7 KB
tests/run_all.shexecutablescript1.6 KB
tests/scripts/test_transcript_filter.pyexecutablescript41.1 KB
tests/test_agent_runtime_contracts.pyexecutablescript1.8 KB
tests/test_ammo_state.pyexecutablescript71.7 KB
tests/test_artifact_layout_manifest.pyscript11.6 KB
tests/test_audit_started.pyscript18.2 KB
tests/test_correctness_comparator.pyscript12.6 KB
tests/test_diluted_pass.pyscript24.4 KB
tests/test_drift_compat.pyscript5.1 KB
tests/test_evidence_freshness.pyexecutablescript5.6 KB
tests/test_fail_closed.pyexecutablescript14.7 KB
tests/test_gpu_reservation.pyexecutablescript24.8 KB
tests/test_gpu_reservation_session_id.pyexecutablescript11.2 KB
tests/test_guidance_semantic_contract.pyscript24.9 KB
tests/test_hook_cmd_classify_parity.pyscript11.7 KB
tests/test_hook_semantics.pyexecutablescript95.3 KB
tests/test_mine_trace.pyscript24.2 KB
tests/test_mining_invalidated_schema.pyexecutablescript9.4 KB
tests/test_new_target_dp_ep.pyexecutablescript7.4 KB
tests/test_new_target_v2_scaffold.pyexecutablescript13.1 KB
tests/test_parse_artifacts.pyexecutablescript3.0 KB
tests/test_policy_convergence.pyscript797 B
tests/test_preflight_vllm_api.pyscript13.3 KB
tests/test_projection_accuracy_v2.pyexecutablescript4.1 KB
tests/test_reconcile_track_state.pyexecutablescript18.2 KB
tests/test_resolve_codex_transcript.pyexecutablescript3.1 KB
tests/test_state_schema_semantics.pyexecutablescript7.6 KB
tests/test_status_set_sync.pyexecutablescript9.1 KB
tests/test_sweep_baseline_from_arm.pyscript11.0 KB
tests/test_sweep_decode_capture.pyexecutablescript31.8 KB
tests/test_sweep_dp.pyexecutablescript45.4 KB
tests/test_sweep_fresh_cache.pyscript5.5 KB
tests/test_sweep_profiler_guard.pyscript22.6 KB
tests/test_sweep_real_prompts.pyscript20.1 KB
tests/test_sweep_refactor.pyexecutablescript37.6 KB
tests/test_sweep_round_slot.pyexecutablescript7.4 KB
tests/test_validation_report_phase_triage.pyexecutablescript10.0 KB
tests/test_validation_report_v2.pyexecutablescript6.2 KB
tests/test_verify_validation_gates.pyexecutablescript52.9 KB
tests/test_workload_matrix.pyexecutablescript19.3 KB

references/nsys-profiling-guide.md

View on GitHub
# Nsight Systems Profiling Guide

This guide owns how AMMO captures GPU traces and how it reads them. Read it before you run a profiler in Stage 1, and again before you write a bottleneck claim in Stage 2. It gives the exact commands, the capture semantics, and the list of what counts as valid evidence.

## The Default Workflow

Stage 2 has one default profiling workflow:

1. Run a clean E2E baseline sweep with no profiler flags.
2. Run a short Nsight Systems node capture. The sweep resolves the CUDA trace backend for your hardware.
3. Mine the nsys report for kernel ranking, launch chains, per-rank/per-device skew, CUDA memcpy/NVLink activity, and timing shares.
4. Run targeted Nsight Compute only for physical-ceiling claims such as occupancy, achieved bandwidth counters, or SM utilization.

Do not use profiler-contaminated latency as the official E2E baseline.

## Required Stage 1 Commands

Run two separate sweeps, in this order.

Clean baseline, used for all speedup math:

```bash
.venv/bin/python .codex/skills/ammo/scripts/run_vllm_bench_latency_sweep.py \
  --artifact-dir {artifact_dir} \
  --target-json {artifact_dir}/target.json \
  --round {N} \
  --slot baseline \
  --labels baseline \
  --capture-golden-refs
```

Bounded selected-step nsys capture, used for bottleneck attribution on every platform:

```bash
.venv/bin/python .codex/skills/ammo/scripts/run_vllm_bench_latency_sweep.py \
  --artifact-dir {artifact_dir} \
  --target-json {artifact_dir}/target.json \
  --round {N} \
  --slot profiling \
  --labels baseline \
  --nsys-profile \
  --nsys-mode node \
  --nsys-capture-output-steps 2,50%,100% \
  --nsys-num-iters 1 \
  --nsys-timeout-s 1800
```

`--slot baseline` plus any profiler flag is invalid; the sweep script enforces this guard.

The trace backend is resolved by the sweep script — see § Trace Backend. Do not pass `--nsys-trace` unless you must override it.

Keep `--nsys-mode node` for the default Stage 2 path because node mode is the ranking source for CUDA graph workloads.

Outputs:

```text
{artifact_dir}/rounds/{N}/sweeps/baseline/e2e_latency_results.json
{artifact_dir}/rounds/{N}/profiling/nsys/*.nsys-rep
```

This produces one `.nsys-rep` per selected depth and batch bucket. The trace
sidecar records both the synthetic capture shape and source shape:
`source_input_len`, `source_output_len`, `capture_output_step`,
`capture_window_output_len`, and `capture_target_output_len`.

Use the bounded fallback `--nsys-output-len 2 --nsys-num-iters 1` only if
selected-step capture fails; document that depth coverage was omitted.

## Selected-Step Capture

`--nsys-capture-output-steps` accepts comma-separated integers and percentages.
`--nsys-capture-output-steps 2,50%,100%` resolves against each workload bucket's
`output_len`. Percentages resolve against `--nsys-output-len` when supplied,
otherwise against each bucket's workload `output_len`. `--nsys-output-len`
remains a horizon override for percentage resolution. Duplicate resolved steps
are removed in stable order.

The sweep then profiles short shape-equivalent windows that land on a genuine full-batch decode step, not a chunked-prefill chunk. For target step `k`, the sweep captures decode depth `input_len + k` (invariant of the window) by running `input_len + k - w_eff`, `output_len = w_eff`, and vLLM's CUDA profiler captures only the final worker step.

`w_eff` is an effective capture window that the script floors child-wide to clear chunked prefill: `w_eff = max(--nsys-capture-window-output-len, max over (bucket, step) of ceil((input_len + step - requested_window) * batch_size / 16384) + 6)`, where `requested_window = --nsys-capture-window-output-len`. The `ceil(...)` term is the number of chunked-prefill worker-steps (chunk = 16384 tokens) the capture must arm past, evaluated at the shifted prompt length `input_len + step - requested_window` (longest at the deepest requested step).

`--nsys-capture-window-output-len` (default 2) is therefore only a LOWER BOUND — the script auto-raises it as needed (e.g. 2 → 11 for an 8192-token prompt at batch 8, where the deepest step 512 gives `ceil((8192+512-2)*8/16384)+6 = 5+6 = 11`) and you cannot force the effective window below the floor. You normally do not set it.

A step shallower than the effective capture window is still captured — the
script shifts `input_len` (`il_eff = input_len + step - w_eff`) so the trace
lands at the requested decode depth. The only step that is dropped is one where
`il_eff < 1` (the context is too short to host a steady-state decode); that
bucket/step logs a loud WARNING and the remaining steps continue. To recover a
dropped step, use a larger `input_len` or a shallower step — lowering
`--nsys-capture-window-output-len` will not help, because it is only a lower
bound that the script auto-raises to clear chunked prefill.

## Trace Backend

`_default_nsys_trace()` in the sweep script owns this rule: `--nsys-trace` defaults to `cuda-sw` when `target.hardware` names Blackwell (B200/B300/GB200/GB300, SM100/SM120), else `cuda`. The sweep prints the resolved value and its source. The script is the only place this rule lives; there is no hand-maintained hardware matrix to read or keep current.

On Blackwell, the hardware event system for CUDA tracing interacts poorly with CUDA graph replay, causing long warmup/profiling stalls followed by RPC or NCCL watchdog timeouts. This affects all workloads (not just MoE or TP>1). `cuda-sw` still captures CUDA API/software activity and graph node attribution while avoiding that hardware-tracing path. On a non-Blackwell target, override to `cuda-sw` only after logs match that replay/collective timeout failure.

## When a Capture Fails

If the command fails:

- Check the supervisor log under `rounds/{N}/sweeps/profiling/logs/`.
- Check the printed `nsys_trace:` line. It reports the resolved backend and whether it came from `target.hardware` or an explicit flag.
- Use the bounded fallback (`--nsys-output-len 2 --nsys-num-iters 1`) only when selected-step capture itself fails — see § Required Stage 1 Commands.
- Increase `--nsys-timeout-s` only if logs show useful forward progress.
- Reduce the profiled bucket set only as a last resort, and document the omitted buckets.

## Stage 2 Mining

Mine the existing nsys reports. Do not re-run the sweep just to analyze traces.

Minimum analysis:

- Export nsys reports to SQLite or stats tables.
- Rank kernels by total GPU time and count.
- Group kernel chains in timestamp order, not architecture order.
- Map top kernels to source paths or generated backends when possible.
- For TP > 1, compare all rank/device reports and report per-rank skew.
- Separate compute kernels from communication kernels and memcpy/P2P traffic.
- Compute `f_decode`, `decode_share_of_e2e`, and `f_e2e` using measured Stage 1 data.

Useful commands:

```bash
nsys stats --force-export=true --report cuda_gpu_kern_sum \
  {artifact_dir}/rounds/{N}/profiling/nsys/baseline_profile*.nsys-rep

nsys stats --force-export=true --report cuda_gpu_trace \
  {artifact_dir}/rounds/{N}/profiling/nsys/baseline_profile*.nsys-rep

nsys export --type sqlite --force-overwrite=true \
  --output {artifact_dir}/rounds/{N}/profiling/nsys/baseline.sqlite \
  {artifact_dir}/rounds/{N}/profiling/nsys/baseline_profile*.nsys-rep
```

Report approximate trace timings honestly. A value such as `~74 us` is valid when it comes from the nsys trace.

## Optional Graph Diagnostics

`--nsys-mode graph` is optional diagnostic enrichment. Use it only when node-mode results leave a specific open question about graph structure, launch grouping, CUDA graph replay, or metadata available only in graph view.

If node and graph mode disagree, use node-mode timing for bottleneck ranking (§ Required Stage 1 Commands) and explain why graph mode was captured.

Graph diagnostic command:

```bash
.venv/bin/python .codex/skills/ammo/scripts/run_vllm_bench_latency_sweep.py \
  --artifact-dir {artifact_dir} \
  --target-json {artifact_dir}/target.json \
  --round {N} \
  --slot profiling \
  --labels baseline \
  --nsys-profile \
  --nsys-mode graph \
  --nsys-output-len 2 \
  --nsys-num-iters 1 \
  --nsys-timeout-s 1800
```

Graph diagnostics use the same hardware-derived trace backend as node mode.

## Targeted NCU

The physical-ceiling gate is disjunctive: back the claim with targeted Nsight Compute, or with an explicitly cited hardware spec plus the math. Either one alone is enough. Nsys alone is not.

These count as physical-ceiling claims:

- Occupancy or achieved occupancy.
- SM utilization or tensor-core utilization.
- Achieved memory bandwidth from hardware counters.
- Register pressure, shared-memory pressure, or stall-reason claims.
- "Kernel X can improve by at most Y%" where Y comes from a hardware ceiling.

Keep NCU narrow. Profile only the top kernels identified by nsys, with representative bucket sizes. Store results under:

```text
{artifact_dir}/rounds/{N}/profiling/ncu/
```

Stage 2 may rank bottlenecks from nsys without NCU.

## Valid Stage 2 Evidence

Acceptable:

- `rounds/{N}/profiling/nsys/*.nsys-rep`
- nsys stats/export tables derived from those reports
- sweep JSON from `rounds/{N}/sweeps/baseline/`
- targeted NCU CSV/report for hardware-counter claims
- source-code mapping used only to explain what a measured kernel is

Not acceptable:

- Profiler-run latency as official E2E timing.
- Architecture-inferred kernel chains without trace timestamps.
- Occupancy/bandwidth-counter claims without targeted NCU.
- Graph-mode timing as the primary bottleneck ranking source.

## Report Checklist

`rounds/{N}/mining/bottleneck_analysis.md` must include:

- Exact Stage 1 baseline and profiling commands.
- Artifact paths for every trace used.
- Top kernels/components by measured time.
- Per-rank/per-device comparison for TP/DP runs.
- `f_decode`, `decode_share_of_e2e`, and `f_e2e` tables.
- Technology Landscape entries for the top components.
- Any NCU-backed physical-ceiling claims with paths to the raw NCU artifacts.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall