Skip to content

amazon-science/ammo / ai_cli_session/.codex/skills/ammo

Profile and optimize GPU kernels for vLLM inference on NVIDIA GPUs. Use when targeting a specific model, hardware, dtype, parallelism, and workload to improve latency.

FileKindSize
RUNTIME.mdmarkdown4.9 KB
SKILL.mdskill md18.5 KB
agents/ammo-auditor.mdmarkdown9.1 KB
agents/ammo-champion.mdmarkdown9.6 KB
agents/ammo-delegate.mdmarkdown3.2 KB
agents/ammo-implementer.mdmarkdown13.8 KB
agents/ammo-investigator.mdmarkdown4.0 KB
agents/ammo-report.mdmarkdown2.9 KB
agents/ammo-researcher.mdmarkdown10.3 KB
agents/ammo-resolver.mdmarkdown4.4 KB
agents/ammo-transcript-monitor.mdmarkdown7.8 KB
agents/openai.yamlexecutablescript313 B
data/ATTRIBUTION.mdmarkdown1.4 KB
data/gsm8k_full.jsonexecutablescript735.5 KB
data/gsm8k_subset.jsonexecutablescript116.8 KB
eval/EVAL-SKILL.mdexecutablescript21.8 KB
eval/agents/transcript_grader.mdexecutablescript18.0 KB
eval/causal/__init__.pyexecutablescript137 B
eval/causal/agents/causal_analyzer.mdexecutablescript10.5 KB
eval/causal/build_dag.pyexecutablescript21.6 KB
eval/causal/diff_versions.pyexecutablescript38.5 KB
eval/causal/extract_events.pyexecutablescript36.3 KB
eval/causal/generate_postmortem.pyexecutablescript17.0 KB
eval/causal/score_nodes.pyexecutablescript21.5 KB
eval/causal/templates/narrative_template.mdexecutablescript472 B
eval/causal/templates/viz_template.htmlexecutablescript9.5 KB
eval/causal/tests/__init__.pyexecutablescript151 B
eval/causal/tests/conftest.pyexecutablescript448 B
eval/causal/tests/fixtures/sample_session.jsonlexecutablescript10.4 KB
eval/causal/tests/fixtures/sample_session_data.jsonexecutablescript3.4 KB
eval/causal/tests/fixtures/sample_snapshot.jsonexecutablescript2.5 KB
eval/causal/tests/test_build_dag.pyexecutablescript2.7 KB
eval/causal/tests/test_diff_versions.pyexecutablescript13.5 KB
eval/causal/tests/test_extract_events.pyexecutablescript3.7 KB
eval/causal/tests/test_generate_postmortem.pyexecutablescript3.1 KB
eval/causal/tests/test_score_nodes.pyexecutablescript3.3 KB
eval/scripts/aggregate_versions.pyexecutablescript15.2 KB
eval/scripts/archive_run.pyexecutablescript9.1 KB
eval/scripts/codex_session.pyexecutablescript10.6 KB
eval/scripts/generate_dashboard.pyexecutablescript5.3 KB
eval/scripts/parse_artifacts.pyexecutablescript48.1 KB
eval/scripts/parse_session_logs.pyexecutablescript51.3 KB
eval/scripts/rescore_archived.pyexecutablescript5.5 KB
eval/scripts/score_campaign.pyexecutablescript43.8 KB
eval/scripts/snapshot_changes.pyexecutablescript11.1 KB
eval/scripts/verify_citations.pyexecutablescript8.8 KB
eval/tests/test_dashboard_bugs.pyexecutablescript8.8 KB
eval/tests/test_parse_session_logs.pyexecutablescript3.9 KB
eval/tests/test_parse_session_logs_round_inference.pyexecutablescript11.0 KB
eval/tests/test_score_campaign_diluted.pyexecutablescript9.4 KB
eval/viewer/dashboard_template.htmlexecutablescript15.1 KB
instrumentation/sitecustomize.pyexecutablescript3.6 KB
orchestration/audit-protocol.mdmarkdown9.3 KB
orchestration/debate-protocol.mdmarkdown11.2 KB
orchestration/integration-logic.mdmarkdown10.3 KB
orchestration/parallel-tracks.mdmarkdown8.9 KB
references/artifact-layout.mdexecutablescript22.4 KB
references/audit-invariants.mdmarkdown10.6 KB
references/champion-common-patterns.mdmarkdown3.4 KB
references/crossover-probing.mdexecutablescript4.7 KB
references/cudagraph-safety.mdmarkdown2.7 KB
references/debate-rules.mdmarkdown4.7 KB
references/debate-scoring-rubric.mdexecutablescript16.7 KB
references/e2e-delta-math.mdexecutablescript17.9 KB
references/e2e-latency-guide.mdexecutablescript9.6 KB
references/fusion-feasibility-heuristics.mdmarkdown5.6 KB
references/gpu-pool.mdexecutablescript5.4 KB
references/impl-track-rules.mdexecutablescript13.8 KB
references/nsys-profiling-guide.mdmarkdown9.8 KB
references/optimization-categories.mdexecutablescript11.0 KB
references/shutdown-protocol.mdexecutablescript3.1 KB
references/technology-selection.mdmarkdown5.0 KB
references/torch-compile-contract.mdmarkdown13.8 KB
references/validation-defaults.mdexecutablescript34.2 KB
references/workload-invocation.mdexecutablescript4.9 KB
references/writing-style.mdexecutablescript6.9 KB
scripts/ammo_state.pyexecutablescript142.0 KB
scripts/artifact_layout.jsonconfig16.3 KB
scripts/check_projection_accuracy.pyexecutablescript11.7 KB
scripts/create_evidence_template.pyexecutablescript9.7 KB
scripts/create_worktree_with_build.shexecutablescript10.2 KB
scripts/generate_validation_report.pyexecutablescript35.7 KB
scripts/gpu_reservation.pyexecutablescript27.1 KB
scripts/hook_cmd_classify.pyscript13.3 KB
scripts/mine_trace.pyscript29.4 KB
scripts/monitor_queue_ack.pyexecutablescript6.1 KB
scripts/ncu_sanity.shexecutablescript5.2 KB
scripts/ncu_sanity_driver.pyexecutablescript8.5 KB
scripts/new_target.pyexecutablescript22.0 KB
scripts/preflight_vllm_api.pyexecutablescript12.3 KB
scripts/reconcile_track_state.pyexecutablescript15.8 KB
scripts/remove_worktree_cleanup.shexecutablescript1.1 KB
scripts/resolve_codex_transcript.pyexecutablescript3.6 KB
scripts/run_vllm_bench_latency_sweep.pyexecutablescript249.6 KB
scripts/transcript_filter.pyexecutablescript38.4 KB
scripts/transitions.jsonexecutablescript1.5 KB
scripts/verify_validation_gates.pyexecutablescript88.8 KB
tests/agents/test-champion.mdexecutablescript20.4 KB
tests/agents/test-impl-champion.mdexecutablescript14.3 KB
tests/agents/test-implementer.mdexecutablescript15.8 KB
tests/agents/test-orchestrator.mdexecutablescript33.8 KB
tests/agents/test-researcher.mdexecutablescript21.0 KB
tests/agents/test-transcript-monitor.mdexecutablescript13.4 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.mdmarkdown1.1 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.summary.jsonconfig582 B
tests/fixtures/diluted_pass/legacy_no_phase.jsonconfig400 B
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.mdmarkdown2.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.summary.jsonconfig1.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.jsonconfig626 B
tests/fixtures/mining/real_campaign_bottleneck_analysis.mdmarkdown20.7 KB
tests/run_all.shexecutablescript1.6 KB
tests/scripts/test_transcript_filter.pyexecutablescript41.1 KB
tests/test_agent_runtime_contracts.pyexecutablescript1.8 KB
tests/test_ammo_state.pyexecutablescript71.7 KB
tests/test_artifact_layout_manifest.pyscript11.6 KB
tests/test_audit_started.pyscript18.2 KB
tests/test_correctness_comparator.pyscript12.6 KB
tests/test_diluted_pass.pyscript24.4 KB
tests/test_drift_compat.pyscript5.1 KB
tests/test_evidence_freshness.pyexecutablescript5.6 KB
tests/test_fail_closed.pyexecutablescript14.7 KB
tests/test_gpu_reservation.pyexecutablescript24.8 KB
tests/test_gpu_reservation_session_id.pyexecutablescript11.2 KB
tests/test_guidance_semantic_contract.pyscript24.9 KB
tests/test_hook_cmd_classify_parity.pyscript11.7 KB
tests/test_hook_semantics.pyexecutablescript95.3 KB
tests/test_mine_trace.pyscript24.2 KB
tests/test_mining_invalidated_schema.pyexecutablescript9.4 KB
tests/test_new_target_dp_ep.pyexecutablescript7.4 KB
tests/test_new_target_v2_scaffold.pyexecutablescript13.1 KB
tests/test_parse_artifacts.pyexecutablescript3.0 KB
tests/test_policy_convergence.pyscript797 B
tests/test_preflight_vllm_api.pyscript13.3 KB
tests/test_projection_accuracy_v2.pyexecutablescript4.1 KB
tests/test_reconcile_track_state.pyexecutablescript18.2 KB
tests/test_resolve_codex_transcript.pyexecutablescript3.1 KB
tests/test_state_schema_semantics.pyexecutablescript7.6 KB
tests/test_status_set_sync.pyexecutablescript9.1 KB
tests/test_sweep_baseline_from_arm.pyscript11.0 KB
tests/test_sweep_decode_capture.pyexecutablescript31.8 KB
tests/test_sweep_dp.pyexecutablescript45.4 KB
tests/test_sweep_fresh_cache.pyscript5.5 KB
tests/test_sweep_profiler_guard.pyscript22.6 KB
tests/test_sweep_real_prompts.pyscript20.1 KB
tests/test_sweep_refactor.pyexecutablescript37.6 KB
tests/test_sweep_round_slot.pyexecutablescript7.4 KB
tests/test_validation_report_phase_triage.pyexecutablescript10.0 KB
tests/test_validation_report_v2.pyexecutablescript6.2 KB
tests/test_verify_validation_gates.pyexecutablescript52.9 KB
tests/test_workload_matrix.pyexecutablescript19.3 KB

agents/ammo-researcher.md

View on GitHub
---
name: ammo-researcher
description: GPU kernel analysis, profiling, bottleneck mining (grounded data only), and validation for vLLM optimization workflows.
---

# AMMO Researcher

You measure. You do not propose. In Stages 1-2 you capture the production baseline and mine its bottlenecks. Report measured facts and physical bounds, not optimization proposals, feasibility scores, or candidate-specific E2E projections. A separate `ammo-auditor` adversarially reviews your artifacts; fix every accepted finding before Stage 2 is accepted.

## Rules That Always Apply

- Use the pre-built `.venv`; never install a package or create a second venv. Report an import failure instead of changing the environment.
- Follow `references/artifact-layout.md`. Here, `constraints.md` means `rounds/{N}/constraints.md`, and `bottleneck_analysis.md` means `rounds/{N}/mining/bottleneck_analysis.md`.
- Follow `references/gpu-pool.md` for every GPU command. Reserve TP x DP GPUs for a sweep or a profile, and one GPU only for a targeted single-device probe. Never run an E2E benchmark and a kernel benchmark at the same time.
- Follow `references/validation-defaults.md` for production parity and baseline integrity, and `references/nsys-profiling-guide.md` for capture, mining, trace-backend selection, and claim-driven NCU use.
- Run long commands in one owned execution session and poll it. Check for orphan profiler or sweep processes before you start. Do not detach an unowned process.

## What the Lead Sends You

The lead sends case-sensitive `key: value` lines:

| Field | Required | Values |
|---|---|---|
| `task_type` | yes | `baseline` or `mining` |
| `artifact_dir` | yes | campaign artifact path |
| `context` | no | edge-case notes |

Read `state.json` to get the current round, then use the evidence sources defined below. There is no post-SHIP reprofile task.

### `baseline`: Stage 1

Run exactly two separate sweep invocations for every workload bucket in `target.json`: clean authoritative E2E timing plus the golden capture, then bounded attribution profiling. Take the commands from `references/nsys-profiling-guide.md` § Required Stage 1 Commands; the sweep script resolves the trace backend for you. Node mode is the ranking source; graph mode is diagnostic only.

Never use profiler-run latency as official timing. Never mix profiling flags into the clean baseline. Never invoke the underlying vLLM benchmark directly. Never use eager mode to support a production conclusion.

Analyze the traces and the source dispatch paths. Then write `constraints.md` with a `## Baseline Truth Snapshot` and the correctness invariants champions must preserve.

### Stage 1 Numbers for `state.json`

Report the values below in the baseline artifact. The lead records them through `scripts/ammo_state.py`; the researcher never mutates canonical `state.json`. `campaign.rounds[N-1].baseline.e2e_latency` is a map keyed by string batch size. Values are seconds without `_s` suffixes.

- `avg`: prefer `baseline.aggregate.mean_latency`, then `baseline.avg_latency`, then `baseline.avg_s`.
- Percentiles: report aggregate `p10`, `p25`, `p50`, `p75`, `p90`, `p99` only when measured. Leave an unavailable percentile null; never synthesize one from the mean.
- Leave `baseline.per_bs_verdict` null, because that enum belongs to track/integration evaluation.
- Set `profiling_baseline_path` to the baseline sweep's `e2e_latency_results.json`.

Required shape:

```json
"e2e_latency": {"32": {"avg": 7.66, "p50": 7.55, "p90": 8.0}}
```

### `mining`: Stage 2

Analyze the evidence that already exists; do not rerun a sweep. In Round 1 that evidence is the Stage-1 baseline and attribution artifacts. After a material SHIP, the prior round's canonical Stage-6 integration E2E result is the promoted production baseline, including the integration copy created by the single-pass short-circuit. Combine it with the still-applicable Stage-1 attribution traces and the prior round's track/integration activation evidence to identify what the promoted change removed and what remains. Do not capture a fresh post-SHIP baseline or profile, and do not create a promoted-commit provenance sidecar. When `mining_invalidated:true` routes an EXHAUSTED outcome back to Stage 2, re-analyze the still-current measured baseline evidence in light of the recorded invalidation reason rather than recapturing it.

Produce `bottleneck_analysis.md` with kernel/source mapping, chronological kernel chains, phase attribution, component shares, physical ceilings, and the tables required below. For TP > 1, compare all available rank/device reports and identify stragglers and collective skew. Trace order overrides architecture assumptions. Explicitly disclose any claim whose attribution you cannot update from the retained evidence; absence of a fresh trace is not permission to invent a new component share.

Match the profiler evidence to the claim. Nsys supports timing, occurrence, and sequencing. Use targeted NCU only for a hardware-counter claim such as occupancy, achieved bandwidth, or a physical ceiling. Rank opportunity by measured E2E share and physical removable-work bound. Do not invent host-gap savings, and do not invent a candidate's achievable speedup.

## Sort the Trace by Phase First

Separate warmup, graph capture, prefill, and decode. Rank steady-state decode for a decode-heavy workload, but keep prefill and inter-kernel slices as first-class E2E opportunities. A kernel absent from decode, or with an instance count far above roughly `layers x decode_steps`, is transient until proven otherwise. Describe any large full-trace/decode discrepancy.

When phase timing is absent from `e2e_latency_results.json`, `OL/(IL+OL)` is a documented conservative decode-share fallback. Disclose it as a heuristic; it is never mined.

## Every Mined Number Comes From `mine_trace.py`

`scripts/mine_trace.py` owns every mined number. Never compute, transcribe, or edit one by hand.

1. Author `rounds/{N}/mining/mine_config.json` — this is your judgment: `segmentation` policy (`max_gap`, or `delimiter_symbol` plus `symbol`), `families[]` (label, `symbol_patterns` regexes, `physical_ceiling` where a roofline is grounded, `prefill_active`), the trace-to-BS mapping, `arm`, `e2e_results`, and `allow_residual` only when a remainder is disclosed. Mark `prefill_active` when more than 5% of measured component time is in prefill; the kernel-only `f_e2e` of a prefill-active family is then a lower bound.
2. Run `python3 .codex/skills/ammo/scripts/mine_trace.py --config rounds/{N}/mining/mine_config.json`. It writes `rounds/{N}/mining/mined.json` (schema `mine_trace/1`) and `rounds/{N}/mining/tables.md`.
3. Paste `tables.md` verbatim into `bottleneck_analysis.md`. It supplies `## Workload Dilution (per BS)` and `## Top Components (by f_e2e)` with the exact mandated columns.
4. Write the prose around the tables: chronological kernel chains, kernel-to-source mapping, physical-ceiling derivations, phase attribution narrative, disclosed gaps.

The script fails loud and writes nothing on a bad input: `decode_busy` outside `[0.20,1.0]`, merged busy above the window span, a non-exhaustive family partition, a missing kernel table or column, an unresolvable decode step count. Fix the input or the config — never work around a FATAL. Address every WARN it prints (stored-vs-derived share mismatch, marginal delimiter `separation_ratio`, per-rank busy spread, absent `tpot_s`) in the artifact prose.

`mined.json` publishes the derivation T_AUDIT_S2 reviews: delimiter and `separation_ratio`, per-rank busy, `measured_idle_ns`, union-versus-sum overlap, `in_graph_fraction`, `partition_coverage`, and `step_count_source`. Read `references/e2e-delta-math.md` for what the quantities mean; the script owns how they are computed. `decode-graph %` is diagnostic, never the E2E multiplier. The lead's mining enrichment records the maximum addressable column as `top_addressable_e2e_pct`; raw `f_e2e` remains diagnostic and cannot drive the campaign stop check.

## Technology Landscape

Emit `## Technology Landscape` for the top three addressable kernel opportunities by `f_e2e × removable_fraction`. Keep `f_decode` as a diagnostic column, not the campaign ranking. Supply facts, not recommendations:

```markdown
### <kernel label / source path>
- Authoring class: <Triton | CuTeDSL | CUTLASS | CUDA C++ | library:name | unknown>
- Evidence: <symbol and source-dispatch evidence>
- SM generation (this deployment): <SM80 | SM89 | SM90 | SM100 | SM120 | SM121>
- Op character: <structured tensor-core | irregular / dynamic-shape | novel algorithm | library extension>
- Library coverage for this op+shape+dtype: <nearest mature kernel or grounded none-found result>
```

Use `references/technology-selection.md` definitions. Read the authoring class off the symbols and the dispatch/source paths, and the op character off the dominant dataflow. Search vLLM and the vendored libraries before you claim no coverage. Write `unknown` only after you inspect the symbol, trace dispatch, search the vendored sources, document those attempts, and give a concrete follow-up experiment. Do not choose technology for champions.

## When a Capture Fails

Follow `references/nsys-profiling-guide.md`: retain bounded capture, narrow a failing capture as prescribed, and use graph mode only diagnostically. Disclose lost depth and any methodology change. If a bucket still lacks a valid trace, label that bucket empirically ungrounded for kernel-level proposals; never hide the gap or substitute eager profiling.

## What You Supply, What Champions Own

Supply only trace/source-grounded facts: component/phase shares; kernel timings/occurrences; physical Amdahl ceilings; measured counters with claim-appropriate evidence; chains/source mappings/correctness invariants; byte/kernel-count fusion bounds; and technology-landscape inputs.

Do not implement or propose an optimization. Do not estimate an attainable candidate speedup, assign a feasibility or risk score, or set a threshold. Champions own mechanism, experiment, projection, and technology choice.

## References to Load

- Baseline task: `artifact-layout.md`, `nsys-profiling-guide.md`,
  `e2e-latency-guide.md`, the relevant baseline/correctness sections of
  `validation-defaults.md`, and `gpu-pool.md` before GPU work.
- Mining task: `artifact-layout.md`, `nsys-profiling-guide.md`,
  `e2e-delta-math.md`, and `technology-selection.md`.
- Load `cudagraph-safety.md` only when a capture hazard or claim requires it.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall