Skip to content

amazon-science/ammo / ai_cli_session/.codex/skills/ammo

Profile and optimize GPU kernels for vLLM inference on NVIDIA GPUs. Use when targeting a specific model, hardware, dtype, parallelism, and workload to improve latency.

FileKindSize
RUNTIME.mdmarkdown4.9 KB
SKILL.mdskill md18.5 KB
agents/ammo-auditor.mdmarkdown9.1 KB
agents/ammo-champion.mdmarkdown9.6 KB
agents/ammo-delegate.mdmarkdown3.2 KB
agents/ammo-implementer.mdmarkdown13.8 KB
agents/ammo-investigator.mdmarkdown4.0 KB
agents/ammo-report.mdmarkdown2.9 KB
agents/ammo-researcher.mdmarkdown10.3 KB
agents/ammo-resolver.mdmarkdown4.4 KB
agents/ammo-transcript-monitor.mdmarkdown7.8 KB
agents/openai.yamlexecutablescript313 B
data/ATTRIBUTION.mdmarkdown1.4 KB
data/gsm8k_full.jsonexecutablescript735.5 KB
data/gsm8k_subset.jsonexecutablescript116.8 KB
eval/EVAL-SKILL.mdexecutablescript21.8 KB
eval/agents/transcript_grader.mdexecutablescript18.0 KB
eval/causal/__init__.pyexecutablescript137 B
eval/causal/agents/causal_analyzer.mdexecutablescript10.5 KB
eval/causal/build_dag.pyexecutablescript21.6 KB
eval/causal/diff_versions.pyexecutablescript38.5 KB
eval/causal/extract_events.pyexecutablescript36.3 KB
eval/causal/generate_postmortem.pyexecutablescript17.0 KB
eval/causal/score_nodes.pyexecutablescript21.5 KB
eval/causal/templates/narrative_template.mdexecutablescript472 B
eval/causal/templates/viz_template.htmlexecutablescript9.5 KB
eval/causal/tests/__init__.pyexecutablescript151 B
eval/causal/tests/conftest.pyexecutablescript448 B
eval/causal/tests/fixtures/sample_session.jsonlexecutablescript10.4 KB
eval/causal/tests/fixtures/sample_session_data.jsonexecutablescript3.4 KB
eval/causal/tests/fixtures/sample_snapshot.jsonexecutablescript2.5 KB
eval/causal/tests/test_build_dag.pyexecutablescript2.7 KB
eval/causal/tests/test_diff_versions.pyexecutablescript13.5 KB
eval/causal/tests/test_extract_events.pyexecutablescript3.7 KB
eval/causal/tests/test_generate_postmortem.pyexecutablescript3.1 KB
eval/causal/tests/test_score_nodes.pyexecutablescript3.3 KB
eval/scripts/aggregate_versions.pyexecutablescript15.2 KB
eval/scripts/archive_run.pyexecutablescript9.1 KB
eval/scripts/codex_session.pyexecutablescript10.6 KB
eval/scripts/generate_dashboard.pyexecutablescript5.3 KB
eval/scripts/parse_artifacts.pyexecutablescript48.1 KB
eval/scripts/parse_session_logs.pyexecutablescript51.3 KB
eval/scripts/rescore_archived.pyexecutablescript5.5 KB
eval/scripts/score_campaign.pyexecutablescript43.8 KB
eval/scripts/snapshot_changes.pyexecutablescript11.1 KB
eval/scripts/verify_citations.pyexecutablescript8.8 KB
eval/tests/test_dashboard_bugs.pyexecutablescript8.8 KB
eval/tests/test_parse_session_logs.pyexecutablescript3.9 KB
eval/tests/test_parse_session_logs_round_inference.pyexecutablescript11.0 KB
eval/tests/test_score_campaign_diluted.pyexecutablescript9.4 KB
eval/viewer/dashboard_template.htmlexecutablescript15.1 KB
instrumentation/sitecustomize.pyexecutablescript3.6 KB
orchestration/audit-protocol.mdmarkdown9.3 KB
orchestration/debate-protocol.mdmarkdown11.2 KB
orchestration/integration-logic.mdmarkdown10.3 KB
orchestration/parallel-tracks.mdmarkdown8.9 KB
references/artifact-layout.mdexecutablescript22.4 KB
references/audit-invariants.mdmarkdown10.6 KB
references/champion-common-patterns.mdmarkdown3.4 KB
references/crossover-probing.mdexecutablescript4.7 KB
references/cudagraph-safety.mdmarkdown2.7 KB
references/debate-rules.mdmarkdown4.7 KB
references/debate-scoring-rubric.mdexecutablescript16.7 KB
references/e2e-delta-math.mdexecutablescript17.9 KB
references/e2e-latency-guide.mdexecutablescript9.6 KB
references/fusion-feasibility-heuristics.mdmarkdown5.6 KB
references/gpu-pool.mdexecutablescript5.4 KB
references/impl-track-rules.mdexecutablescript13.8 KB
references/nsys-profiling-guide.mdmarkdown9.8 KB
references/optimization-categories.mdexecutablescript11.0 KB
references/shutdown-protocol.mdexecutablescript3.1 KB
references/technology-selection.mdmarkdown5.0 KB
references/torch-compile-contract.mdmarkdown13.8 KB
references/validation-defaults.mdexecutablescript34.2 KB
references/workload-invocation.mdexecutablescript4.9 KB
references/writing-style.mdexecutablescript6.9 KB
scripts/ammo_state.pyexecutablescript142.0 KB
scripts/artifact_layout.jsonconfig16.3 KB
scripts/check_projection_accuracy.pyexecutablescript11.7 KB
scripts/create_evidence_template.pyexecutablescript9.7 KB
scripts/create_worktree_with_build.shexecutablescript10.2 KB
scripts/generate_validation_report.pyexecutablescript35.7 KB
scripts/gpu_reservation.pyexecutablescript27.1 KB
scripts/hook_cmd_classify.pyscript13.3 KB
scripts/mine_trace.pyscript29.4 KB
scripts/monitor_queue_ack.pyexecutablescript6.1 KB
scripts/ncu_sanity.shexecutablescript5.2 KB
scripts/ncu_sanity_driver.pyexecutablescript8.5 KB
scripts/new_target.pyexecutablescript22.0 KB
scripts/preflight_vllm_api.pyexecutablescript12.3 KB
scripts/reconcile_track_state.pyexecutablescript15.8 KB
scripts/remove_worktree_cleanup.shexecutablescript1.1 KB
scripts/resolve_codex_transcript.pyexecutablescript3.6 KB
scripts/run_vllm_bench_latency_sweep.pyexecutablescript249.6 KB
scripts/transcript_filter.pyexecutablescript38.4 KB
scripts/transitions.jsonexecutablescript1.5 KB
scripts/verify_validation_gates.pyexecutablescript88.8 KB
tests/agents/test-champion.mdexecutablescript20.4 KB
tests/agents/test-impl-champion.mdexecutablescript14.3 KB
tests/agents/test-implementer.mdexecutablescript15.8 KB
tests/agents/test-orchestrator.mdexecutablescript33.8 KB
tests/agents/test-researcher.mdexecutablescript21.0 KB
tests/agents/test-transcript-monitor.mdexecutablescript13.4 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.mdmarkdown1.1 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.summary.jsonconfig582 B
tests/fixtures/diluted_pass/legacy_no_phase.jsonconfig400 B
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.mdmarkdown2.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.summary.jsonconfig1.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.jsonconfig626 B
tests/fixtures/mining/real_campaign_bottleneck_analysis.mdmarkdown20.7 KB
tests/run_all.shexecutablescript1.6 KB
tests/scripts/test_transcript_filter.pyexecutablescript41.1 KB
tests/test_agent_runtime_contracts.pyexecutablescript1.8 KB
tests/test_ammo_state.pyexecutablescript71.7 KB
tests/test_artifact_layout_manifest.pyscript11.6 KB
tests/test_audit_started.pyscript18.2 KB
tests/test_correctness_comparator.pyscript12.6 KB
tests/test_diluted_pass.pyscript24.4 KB
tests/test_drift_compat.pyscript5.1 KB
tests/test_evidence_freshness.pyexecutablescript5.6 KB
tests/test_fail_closed.pyexecutablescript14.7 KB
tests/test_gpu_reservation.pyexecutablescript24.8 KB
tests/test_gpu_reservation_session_id.pyexecutablescript11.2 KB
tests/test_guidance_semantic_contract.pyscript24.9 KB
tests/test_hook_cmd_classify_parity.pyscript11.7 KB
tests/test_hook_semantics.pyexecutablescript95.3 KB
tests/test_mine_trace.pyscript24.2 KB
tests/test_mining_invalidated_schema.pyexecutablescript9.4 KB
tests/test_new_target_dp_ep.pyexecutablescript7.4 KB
tests/test_new_target_v2_scaffold.pyexecutablescript13.1 KB
tests/test_parse_artifacts.pyexecutablescript3.0 KB
tests/test_policy_convergence.pyscript797 B
tests/test_preflight_vllm_api.pyscript13.3 KB
tests/test_projection_accuracy_v2.pyexecutablescript4.1 KB
tests/test_reconcile_track_state.pyexecutablescript18.2 KB
tests/test_resolve_codex_transcript.pyexecutablescript3.1 KB
tests/test_state_schema_semantics.pyexecutablescript7.6 KB
tests/test_status_set_sync.pyexecutablescript9.1 KB
tests/test_sweep_baseline_from_arm.pyscript11.0 KB
tests/test_sweep_decode_capture.pyexecutablescript31.8 KB
tests/test_sweep_dp.pyexecutablescript45.4 KB
tests/test_sweep_fresh_cache.pyscript5.5 KB
tests/test_sweep_profiler_guard.pyscript22.6 KB
tests/test_sweep_real_prompts.pyscript20.1 KB
tests/test_sweep_refactor.pyexecutablescript37.6 KB
tests/test_sweep_round_slot.pyexecutablescript7.4 KB
tests/test_validation_report_phase_triage.pyexecutablescript10.0 KB
tests/test_validation_report_v2.pyexecutablescript6.2 KB
tests/test_verify_validation_gates.pyexecutablescript52.9 KB
tests/test_workload_matrix.pyexecutablescript19.3 KB

references/torch-compile-contract.md

View on GitHub
# vLLM torch.compile Contract for Kernel Optimizations

vLLM runs model code under torch.compile and CUDA graphs. That machinery imposes assumptions on any model or kernel change you make, and eager-mode testing never shows you when you broke one. Read this before you edit compiled code, and again before you claim a result.

This document defines 6 invariants. Each one gives the rule, what breaks if you violate it, the fix, source paths you can verify in this worktree, and the validation check that proves you complied. After the invariants come the three sanctioned ways to pick a kernel per batch size, the one thing you must never do, and the checklist to run after integration.

## Quick Reference

```
┌─────────────────────────────────────────────────────────────────────────┐
│ vLLM torch.compile Contract — 6 Invariants                              │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  1. SINGLE TRACE: forward() traced ONCE → ONE FX graph.                │
│     Python `if shape` bakes at trace time. Dead code at other shapes.  │
│                                                                         │
│  2. PER-RANGE VARIATION: Only via InductorPass or _pass_context.       │
│     compile_range.start is a concrete int. NOT model attrs or SymInts. │
│                                                                         │
│  3. MUTATION: mutates_args inside IR op fused impls → stale SSA.       │
│     Always mutates_args=[] + return new tensor.                        │
│                                                                         │
│  4. DATA_PTR: Triton in IR op → data_ptr() crash under make_fx.       │
│     Wrap Triton in direct_register_custom_op.                          │
│                                                                         │
│  5. PARTITIONS: Only splitting_ops create CUDA graph boundaries.       │
│     direct_register_custom_op alone does NOT cause partitions.         │
│                                                                         │
│  6. RANGE OVERHEAD: compile_ranges_endpoints adds ~0.5-1% overhead     │
│     to batch sizes in the smaller range. Not a bug — structural cost.  │
│                                                                         │
│  VALIDATION: accuracy all BS + kernel fires + partition count +         │
│              dispatch correctness + E2E latency + structural overhead   │
└─────────────────────────────────────────────────────────────────────────┘
```

---

## Invariant 1: Single Trace, Multiple Compilations

Dynamo traces the model `forward()` ONCE and produces ONE FX graph. That single graph is then compiled N times — once per compile range — with different `example_inputs`. So Python control flow in `forward()` evaluates once, at trace time, and its result is baked as a constant into every compiled artifact.

Any Python `if` that depends on tensor shapes, batch size, or runtime state therefore specializes to ONE path. At other shapes the wrong branch executes, which gives you an accuracy regression or a silent correctness failure.

**Source**: `vllm/compilation/piecewise_backend.py` — `self.graph` (singular) is passed to `.compile()` in a loop over `range_entries`:
```python
range_entry.runnable = self.vllm_backend.compiler_manager.compile(
    self.graph,  # <-- SAME graph for all ranges
    args_list,   # <-- different args per range
    compile_range=range_entry.compile_range,
)
```

**Validate**: after integration, run E2E accuracy at ALL batch sizes in `cudagraph_capture_sizes`. If any BS shows >1pp accuracy regression, suspect trace-time baking.

---

## Invariant 2: Inductor Passes Run Per-Range (The Variation Point)

An Inductor pass is the only place where compiled behavior may differ per range. Before each per-range compilation the engine sets `pass_context(compile_range)`, and all Inductor passes — including IR lowering — run inside that context. There, `get_pass_context().compile_range.start` and `.end` are **concrete integers**, so they are the ONLY safe way to vary compiled behavior per range.

Three tempting mechanisms do not work. A Python bool set at model construction gives one value for all ranges, because the model is constructed once. Fake tensor shapes during lowering may be SymInts for multi-size ranges. Module attributes are the same object across all compilations.

**Source**:
- `vllm/compilation/backends.py` — `with pass_context(compile_range):`
- `vllm/compilation/passes/inductor_pass.py` — `get_pass_context()` / `_pass_context` implementation
- `vllm/compilation/passes/pass_manager.py` — consumed during pass execution

**Validate**: if your mechanism uses `_pass_context.compile_range.start`, write a smoke test that exercises the gate under each compile range. Confirm it returns the expected value for both small and large ranges.

---

## Invariant 3: In-Place Mutation Goes Stale Under auto_functionalize_v1

Keep custom ops functional — set `mutates_args=[]` and return a new tensor. vLLM uses `enable_auto_functionalized_v2=False` (v1). Under v1, an in-place mutation inside a nested custom op (for example, an opaque op called from within an IR op's fused impl) is NOT correctly propagated back into the surrounding FX graph's SSA form. The mutated tensor becomes stale, and all downstream ops use pre-mutation data.

`mutates_args=["residual"]` on a custom op called inside an IR op impl is the classic violation: the residual stream goes stale, which costs you catastrophic accuracy loss.

```python
# WRONG — stale SSA reference under auto_functionalize_v1
@torch.library.custom_op("vllm::my_op", mutates_args=["residual"])
def bad(x, residual, ...):
    residual.add_(x)  # mutation not propagated
    return output

# CORRECT — functional, returns new tensor
@torch.library.custom_op("vllm::my_op", mutates_args=[])
def good(x, residual, ...):
    new_residual = residual + x
    return output, new_residual
```

**Validate**: run E2E accuracy after you implement. A >1pp regression that disappears under eager mode points at mutation propagation. Check that no `mutates_args` are used inside IR op impls.

---

## Invariant 4: Wrap Triton Before an IR Op Calls It (data_ptr() vs make_fx)

You cannot directly inline a Triton kernel into an IR op impl. Triton kernels call `tensor.data_ptr()` for pointer arguments, and Inductor's `make_fx` tracing (used by `replace_by_example` during IR lowering) uses FunctionalTensor, which doesn't support `data_ptr()`. Call a Triton kernel directly inside an IR op fused impl and you get `RuntimeError: data_ptr() not available on FunctionalTensor`.

Wrap the Triton kernel in `direct_register_custom_op` with a `register_fake`:
```python
@torch.library.custom_op("vllm::my_triton_wrapper", mutates_args=[])
def wrapper(x: torch.Tensor, w: torch.Tensor, ...) -> tuple[torch.Tensor, torch.Tensor]:
    # data_ptr() is fine inside custom op body (not traced by make_fx)
    return my_triton_kernel_launcher(x, w, ...)

@wrapper.register_fake
def wrapper_fake(x, w, ...):
    return torch.empty_like(x, dtype=torch.float8_e4m3fn), torch.empty(...)
```

Then call `wrapper()` from your IR op fused impl. Inductor treats the custom op as an opaque node but can still inline the surrounding IR op body.

**Source**: `vllm/utils/torch_utils.py` — `direct_register_custom_op` implementation.

**Validate**: if IR op lowering fails with `data_ptr()` errors, wrap the Triton call in an opaque custom op.

---

## Invariant 5: CUDA Graph Partition Boundaries (splitting_ops)

Only ops in `splitting_ops` create partition boundaries. That set controls CUDA graph piecewise splitting by forcing graph boundaries, and `direct_register_custom_op` does NOT automatically add your op to it.

If your custom op IS in `splitting_ops` (or it triggers a code path that hits a splitting op), CUDA graph partitions multiply. Each partition boundary adds replay overhead that can cancel your kernel savings.

**Source**: `vllm/config/compilation.py` — `splitting_ops` defaults to `list(self._attention_ops)` (attention ops only). Check what's in the set:
```bash
grep -n "splitting_ops\|_attention_ops" vllm/config/compilation.py
```

**Validate**: compare CUDA graph partition count before/after your change:
```bash
grep "PIECEWISE=\|FULL=" <compile_log>
# If your change increases partition count, investigate.
```

---

## Invariant 6: compile_ranges_endpoints Structural Overhead

`compile_ranges_endpoints=[N]` splits compilation into multiple ranges, and the split itself costs latency. It adds an additional Inductor compilation per piecewise subgraph, additional CUDA graph captures for batch sizes in the new range, and a small but measurable overhead (~0.5-1.0%) for batch sizes in the smaller range.

Nothing breaks functionally. But batch sizes that land in the smaller range may show a latency regression from the mechanism itself, even if no kernel code changes apply to that range.

**Validate**: run E2E latency at ALL batch sizes. If a BS in the "unmodified" range regresses by 0.5-1.0%, this is likely structural overhead from the range split itself, not a bug in your kernel code.

---

## How to Dispatch on Batch Size

Three mechanisms are sanctioned for shape-dependent dispatch — different kernel paths for different batch sizes under vLLM torch.compile. Choose by the overhead you can pay and by how much graph work you want to do.

### Option 1: `torch.cond` (Simplest, ~2-4 µs overhead)

Use `torch.cond` when both branches return same shape/dtype and ~2-4 µs per call is acceptable. Production example at `vllm/model_executor/kernels/linear/scaled_mm/flashinfer.py`:

```python
condition = input.shape[0] < 32
return torch.cond(
    condition,
    run_flashinfer_swapAB,     # M < 32 path
    run_deepgemm,               # M >= 32 path
    (input, weight, weight_scale),  # ALL inputs as positional args
)
```

Pass ALL inputs (including weights) as positional args. Module closures (`self.linear.weight`) cause Dynamo Parameter-lifting failure → compile error.

**Does NOT create partition boundaries.**

---

### Option 2: `InductorPass.is_applicable_for_range` (Zero overhead, graph rewrite)

Use an Inductor pass when you need different graph structure per range and are comfortable writing Inductor passes.

```python
# Pattern from vllm/compilation/passes/fusion/collective_fusion.py
class MyPass(VllmInductorPass):
    def is_applicable_for_range(self, compile_range: Range) -> bool:
        return compile_range.start >= 32

    def __call__(self, graph: fx.Graph) -> None:
        # Pattern-match and rewrite graph nodes — only runs for applicable ranges
        ...
```

**Production uses**: `collective_fusion.py`, `sequence_parallelism.py`, `allreduce_rms_fusion.py`, `rope_kvcache_fusion.py`.

---

### Option 3: vLLM IR Op + `_pass_context.compile_range` (Zero overhead, impl dispatch)

Use an IR op when you have native + fused impls of the same op and need zero-overhead per-range dispatch. The fused impl is inlined into the FX graph at lowering time via `replace_by_example`, so it costs zero runtime overhead.

```python
from vllm.compilation.passes.inductor_pass import _pass_context

def _fused_supports_args(x, residual, norm_w, eps, weight, weight_scale, ...):
    """Gate fused impl on compile range."""
    if _pass_context is None:
        return False  # eager mode → always native (fail-safe)
    return _pass_context.compile_range.start >= 32
```

**Implementation checklist**:
1. Register IR op with native + fused impls
2. Fused `supports_args` checks `_pass_context.compile_range.start >= N` (concrete int — NOT fake tensor shapes which may be SymInts)
3. Wrap Triton kernel in `direct_register_custom_op` with `mutates_args=[]` (Invariant 3 + 4)
4. Return mutated tensors as outputs (never in-place)
5. Set `compile_ranges_endpoints=[N-1]` in compilation config
6. Test dispatch under each range with a smoke test

---

### NEVER: Python `if tensor.shape[0] < N` in compiled forward

Bakes at trace time. One path for all shapes. Wrong results guaranteed at non-traced batch sizes.

---

## Post-Integration Validation Checklist

| Check | Method | Pass Criteria |
|-------|--------|---------------|
| Accuracy all BS | E2E accuracy at each BS in campaign | < 1pp regression from baseline |
| Kernel fires | nsys trace, grep kernel name | Kernel present with expected instance count |
| Partition count | Compile logs `PIECEWISE=` / `FULL=` | Unchanged from baseline |
| Dispatch correctness | Smoke test `supports_args` under each range | Expected True/False per range |
| E2E latency | Clean sweep (no nsys), production workload | ≥ threshold at target BS |
| Structural overhead | Latency at BS in smaller range | < 1% regression acceptable |

---

## Source Files

| Component | Path |
|-----------|------|
| Single graph compiled per range | `vllm/compilation/piecewise_backend.py` |
| `pass_context(compile_range)` | `vllm/compilation/backends.py` |
| `get_pass_context()` / `_pass_context` | `vllm/compilation/passes/inductor_pass.py` |
| Pass context consumption | `vllm/compilation/passes/pass_manager.py` |
| `splitting_ops` definition | `vllm/config/compilation.py` |
| `direct_register_custom_op` | `vllm/utils/torch_utils.py` |
| `torch.cond` production usage | `vllm/model_executor/kernels/linear/scaled_mm/flashinfer.py` |
| `is_applicable_for_range` pattern | `vllm/compilation/passes/fusion/collective_fusion.py` |
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall