Skip to content

amazon-science/ammo / ai_cli_session/.codex/skills/ammo

Profile and optimize GPU kernels for vLLM inference on NVIDIA GPUs. Use when targeting a specific model, hardware, dtype, parallelism, and workload to improve latency.

FileKindSize
RUNTIME.mdmarkdown4.9 KB
SKILL.mdskill md18.5 KB
agents/ammo-auditor.mdmarkdown9.1 KB
agents/ammo-champion.mdmarkdown9.6 KB
agents/ammo-delegate.mdmarkdown3.2 KB
agents/ammo-implementer.mdmarkdown13.8 KB
agents/ammo-investigator.mdmarkdown4.0 KB
agents/ammo-report.mdmarkdown2.9 KB
agents/ammo-researcher.mdmarkdown10.3 KB
agents/ammo-resolver.mdmarkdown4.4 KB
agents/ammo-transcript-monitor.mdmarkdown7.8 KB
agents/openai.yamlexecutablescript313 B
data/ATTRIBUTION.mdmarkdown1.4 KB
data/gsm8k_full.jsonexecutablescript735.5 KB
data/gsm8k_subset.jsonexecutablescript116.8 KB
eval/EVAL-SKILL.mdexecutablescript21.8 KB
eval/agents/transcript_grader.mdexecutablescript18.0 KB
eval/causal/__init__.pyexecutablescript137 B
eval/causal/agents/causal_analyzer.mdexecutablescript10.5 KB
eval/causal/build_dag.pyexecutablescript21.6 KB
eval/causal/diff_versions.pyexecutablescript38.5 KB
eval/causal/extract_events.pyexecutablescript36.3 KB
eval/causal/generate_postmortem.pyexecutablescript17.0 KB
eval/causal/score_nodes.pyexecutablescript21.5 KB
eval/causal/templates/narrative_template.mdexecutablescript472 B
eval/causal/templates/viz_template.htmlexecutablescript9.5 KB
eval/causal/tests/__init__.pyexecutablescript151 B
eval/causal/tests/conftest.pyexecutablescript448 B
eval/causal/tests/fixtures/sample_session.jsonlexecutablescript10.4 KB
eval/causal/tests/fixtures/sample_session_data.jsonexecutablescript3.4 KB
eval/causal/tests/fixtures/sample_snapshot.jsonexecutablescript2.5 KB
eval/causal/tests/test_build_dag.pyexecutablescript2.7 KB
eval/causal/tests/test_diff_versions.pyexecutablescript13.5 KB
eval/causal/tests/test_extract_events.pyexecutablescript3.7 KB
eval/causal/tests/test_generate_postmortem.pyexecutablescript3.1 KB
eval/causal/tests/test_score_nodes.pyexecutablescript3.3 KB
eval/scripts/aggregate_versions.pyexecutablescript15.2 KB
eval/scripts/archive_run.pyexecutablescript9.1 KB
eval/scripts/codex_session.pyexecutablescript10.6 KB
eval/scripts/generate_dashboard.pyexecutablescript5.3 KB
eval/scripts/parse_artifacts.pyexecutablescript48.1 KB
eval/scripts/parse_session_logs.pyexecutablescript51.3 KB
eval/scripts/rescore_archived.pyexecutablescript5.5 KB
eval/scripts/score_campaign.pyexecutablescript43.8 KB
eval/scripts/snapshot_changes.pyexecutablescript11.1 KB
eval/scripts/verify_citations.pyexecutablescript8.8 KB
eval/tests/test_dashboard_bugs.pyexecutablescript8.8 KB
eval/tests/test_parse_session_logs.pyexecutablescript3.9 KB
eval/tests/test_parse_session_logs_round_inference.pyexecutablescript11.0 KB
eval/tests/test_score_campaign_diluted.pyexecutablescript9.4 KB
eval/viewer/dashboard_template.htmlexecutablescript15.1 KB
instrumentation/sitecustomize.pyexecutablescript3.6 KB
orchestration/audit-protocol.mdmarkdown9.3 KB
orchestration/debate-protocol.mdmarkdown11.2 KB
orchestration/integration-logic.mdmarkdown10.3 KB
orchestration/parallel-tracks.mdmarkdown8.9 KB
references/artifact-layout.mdexecutablescript22.4 KB
references/audit-invariants.mdmarkdown10.6 KB
references/champion-common-patterns.mdmarkdown3.4 KB
references/crossover-probing.mdexecutablescript4.7 KB
references/cudagraph-safety.mdmarkdown2.7 KB
references/debate-rules.mdmarkdown4.7 KB
references/debate-scoring-rubric.mdexecutablescript16.7 KB
references/e2e-delta-math.mdexecutablescript17.9 KB
references/e2e-latency-guide.mdexecutablescript9.6 KB
references/fusion-feasibility-heuristics.mdmarkdown5.6 KB
references/gpu-pool.mdexecutablescript5.4 KB
references/impl-track-rules.mdexecutablescript13.8 KB
references/nsys-profiling-guide.mdmarkdown9.8 KB
references/optimization-categories.mdexecutablescript11.0 KB
references/shutdown-protocol.mdexecutablescript3.1 KB
references/technology-selection.mdmarkdown5.0 KB
references/torch-compile-contract.mdmarkdown13.8 KB
references/validation-defaults.mdexecutablescript34.2 KB
references/workload-invocation.mdexecutablescript4.9 KB
references/writing-style.mdexecutablescript6.9 KB
scripts/ammo_state.pyexecutablescript142.0 KB
scripts/artifact_layout.jsonconfig16.3 KB
scripts/check_projection_accuracy.pyexecutablescript11.7 KB
scripts/create_evidence_template.pyexecutablescript9.7 KB
scripts/create_worktree_with_build.shexecutablescript10.2 KB
scripts/generate_validation_report.pyexecutablescript35.7 KB
scripts/gpu_reservation.pyexecutablescript27.1 KB
scripts/hook_cmd_classify.pyscript13.3 KB
scripts/mine_trace.pyscript29.4 KB
scripts/monitor_queue_ack.pyexecutablescript6.1 KB
scripts/ncu_sanity.shexecutablescript5.2 KB
scripts/ncu_sanity_driver.pyexecutablescript8.5 KB
scripts/new_target.pyexecutablescript22.0 KB
scripts/preflight_vllm_api.pyexecutablescript12.3 KB
scripts/reconcile_track_state.pyexecutablescript15.8 KB
scripts/remove_worktree_cleanup.shexecutablescript1.1 KB
scripts/resolve_codex_transcript.pyexecutablescript3.6 KB
scripts/run_vllm_bench_latency_sweep.pyexecutablescript249.6 KB
scripts/transcript_filter.pyexecutablescript38.4 KB
scripts/transitions.jsonexecutablescript1.5 KB
scripts/verify_validation_gates.pyexecutablescript88.8 KB
tests/agents/test-champion.mdexecutablescript20.4 KB
tests/agents/test-impl-champion.mdexecutablescript14.3 KB
tests/agents/test-implementer.mdexecutablescript15.8 KB
tests/agents/test-orchestrator.mdexecutablescript33.8 KB
tests/agents/test-researcher.mdexecutablescript21.0 KB
tests/agents/test-transcript-monitor.mdexecutablescript13.4 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.mdmarkdown1.1 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.summary.jsonconfig582 B
tests/fixtures/diluted_pass/legacy_no_phase.jsonconfig400 B
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.mdmarkdown2.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.summary.jsonconfig1.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.jsonconfig626 B
tests/fixtures/mining/real_campaign_bottleneck_analysis.mdmarkdown20.7 KB
tests/run_all.shexecutablescript1.6 KB
tests/scripts/test_transcript_filter.pyexecutablescript41.1 KB
tests/test_agent_runtime_contracts.pyexecutablescript1.8 KB
tests/test_ammo_state.pyexecutablescript71.7 KB
tests/test_artifact_layout_manifest.pyscript11.6 KB
tests/test_audit_started.pyscript18.2 KB
tests/test_correctness_comparator.pyscript12.6 KB
tests/test_diluted_pass.pyscript24.4 KB
tests/test_drift_compat.pyscript5.1 KB
tests/test_evidence_freshness.pyexecutablescript5.6 KB
tests/test_fail_closed.pyexecutablescript14.7 KB
tests/test_gpu_reservation.pyexecutablescript24.8 KB
tests/test_gpu_reservation_session_id.pyexecutablescript11.2 KB
tests/test_guidance_semantic_contract.pyscript24.9 KB
tests/test_hook_cmd_classify_parity.pyscript11.7 KB
tests/test_hook_semantics.pyexecutablescript95.3 KB
tests/test_mine_trace.pyscript24.2 KB
tests/test_mining_invalidated_schema.pyexecutablescript9.4 KB
tests/test_new_target_dp_ep.pyexecutablescript7.4 KB
tests/test_new_target_v2_scaffold.pyexecutablescript13.1 KB
tests/test_parse_artifacts.pyexecutablescript3.0 KB
tests/test_policy_convergence.pyscript797 B
tests/test_preflight_vllm_api.pyscript13.3 KB
tests/test_projection_accuracy_v2.pyexecutablescript4.1 KB
tests/test_reconcile_track_state.pyexecutablescript18.2 KB
tests/test_resolve_codex_transcript.pyexecutablescript3.1 KB
tests/test_state_schema_semantics.pyexecutablescript7.6 KB
tests/test_status_set_sync.pyexecutablescript9.1 KB
tests/test_sweep_baseline_from_arm.pyscript11.0 KB
tests/test_sweep_decode_capture.pyexecutablescript31.8 KB
tests/test_sweep_dp.pyexecutablescript45.4 KB
tests/test_sweep_fresh_cache.pyscript5.5 KB
tests/test_sweep_profiler_guard.pyscript22.6 KB
tests/test_sweep_real_prompts.pyscript20.1 KB
tests/test_sweep_refactor.pyexecutablescript37.6 KB
tests/test_sweep_round_slot.pyexecutablescript7.4 KB
tests/test_validation_report_phase_triage.pyexecutablescript10.0 KB
tests/test_validation_report_v2.pyexecutablescript6.2 KB
tests/test_verify_validation_gates.pyexecutablescript52.9 KB
tests/test_workload_matrix.pyexecutablescript19.3 KB

scripts/ncu_sanity_driver.py

View on GitHub
#!/usr/bin/env python3
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: CC-BY-NC-4.0
"""Optional claim-driven NCU driver — minimal steady-state replay target.

Use this only for a specific unresolved hardware-counter claim. The wrapper
runs it under `ncu --replay-mode application`. The driver loads
vLLM once (paying the ~3-5 min cold start / torch.compile / CUDA-graph capture
just once), warms up for a handful of decode steps, then runs the exact number
of steady-state decode steps the caller asks for. Under
`--replay-mode application`, ncu re-runs this entire process once per metric
pass, so the driver does NOT loop internally on a per-pass basis — it runs
ONE benchmark from start to finish, ncu handles pass orchestration externally.

Why a dedicated driver instead of `vllm bench latency`:

`vllm bench latency` takes `--num-iters N` and runs a full warmup + N iters.
Under ncu application-replay, that launches ncu's workload once per pass and
ncu captures EVERY kernel launch in the target. With L decoder layers,
default warmup=10, and a broad kernel filter, a single ncu run easily replays
~400 kernel launches per pass × 18+ passes for default `--set` metrics =
tens of minutes just on the cold-start-adjacent launches. Adding
`--launch-count 30` to ncu helps, but only bounds what ncu *records* — the
underlying vLLM process still runs the full warmup every pass.

This driver fixes that by making warmup tiny (default 3) and steady-state
decode steps explicit (default 10). A caller who wants a 30-launch ncu
window picks `--warmup 3 --iters 10` and sets ncu's `--launch-count 30`;
the two numbers multiply to a predictable kernel-launch budget.

Target invocation (called by ncu_sanity.sh):
  ncu ... --launch-count 30 -- \\
    python ncu_sanity_driver.py \\
      --target-json <path> --batch-size 8 --warmup 3 --iters 10
"""

from __future__ import annotations

import argparse
import json
import os
import sys
import time
from pathlib import Path

# vLLM V1 launches the engine in a subprocess by default. ncu's
# --profile-from-start off gates capture via cudaProfilerStart(), but the call
# only affects the calling process's CUDA context — the engine subprocess keeps
# capture disabled, so ncu records "No kernels were profiled." Force V1 into
# single-process mode so the driver's cudaProfilerStart() gates the same
# context that launches kernels. Must be set before any vllm import.
os.environ.setdefault("VLLM_ENABLE_V1_MULTIPROCESSING", "0")


def _read_target(target_json: Path) -> dict:
    with target_json.open() as f:
        return json.load(f)


def _pick_bucket(target: dict, batch_size: int) -> dict:
    workload = target.get("workload", {})
    matrix = workload.get("workload_matrix")
    if matrix:
        for b in matrix:
            if int(b["batch_size"]) == batch_size:
                return {
                    "input_len": int(b["input_len"]),
                    "output_len": int(b["output_len"]),
                    "batch_size": batch_size,
                }
        raise SystemExit(
            f"batch_size={batch_size} not in workload_matrix; "
            f"available: {[b['batch_size'] for b in matrix]}"
        )
    return {
        "input_len": int(workload.get("input_len", 64)),
        "output_len": int(workload.get("output_len", 512)),
        "batch_size": batch_size,
    }


def main() -> int:
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("--target-json", type=Path, required=True,
                   help="target.json — used for model_id, tp, max_model_len, workload shape.")
    p.add_argument("--batch-size", type=int, required=True,
                   help="Which decode bucket to replay. Must be present in workload.")
    p.add_argument("--warmup", type=int, default=3,
                   help="Steady-state warmup iterations before the measurement window. "
                        "Small by design — ncu application-replay pays this per pass.")
    p.add_argument("--iters", type=int, default=10,
                   help="Steady-state iterations to time. ncu will capture the first "
                        "`--launch-count` kernel launches within these iters; 10 iters × "
                        "L decoder layers is the upper bound on captured launches per pass.")
    p.add_argument("--output-len", type=int, default=None,
                   help="Override workload.output_len. Defaults to the bucket's output_len. "
                        "Short OL (e.g. 8) is fine for ncu — we're measuring kernel counters, "
                        "not latency — and it keeps each iter cheap.")
    args = p.parse_args()

    target = _read_target(args.target_json)
    bucket = _pick_bucket(target, args.batch_size)
    if args.output_len is not None:
        bucket["output_len"] = args.output_len

    model_id = target["target"]["model_id"]
    tp = int(target["target"].get("tp", 1))
    max_model_len = int(target["target"].get("max_model_len", 4096))

    # Import inside main so `--help` doesn't require torch/vllm.
    import torch
    from vllm import LLM, SamplingParams

    print(f"[driver] model={model_id} tp={tp} bs={bucket['batch_size']} "
          f"il={bucket['input_len']} ol={bucket['output_len']} "
          f"warmup={args.warmup} iters={args.iters}", flush=True)

    llm_kwargs = dict(
        model=model_id,
        tensor_parallel_size=tp,
        max_model_len=max_model_len,
        enforce_eager=False,
        enable_prefix_caching=False,
        max_num_seqs=bucket["batch_size"],
        compilation_config={
            "cudagraph_capture_sizes": [bucket["batch_size"]],
        },
        # Pinned explicitly because the two construction paths below default it
        # differently: LLM(**kwargs) injects True, EngineArgs defaults to False.
        # Stat loggers inside an ncu-profiled process add launches we do not
        # want, so both paths must get True.
        disable_log_stats=True,
    )
    # vLLM builds that expose LLM.from_engine_args want an EngineArgs object;
    # older builds take the same fields as constructor kwargs. Every field is
    # passed explicitly, so both paths build the same engine.
    if hasattr(LLM, "from_engine_args"):
        from vllm.engine.arg_utils import EngineArgs

        llm = LLM.from_engine_args(EngineArgs(**llm_kwargs))
    else:
        llm = LLM(**llm_kwargs)

    sampling = SamplingParams(
        max_tokens=bucket["output_len"],
        temperature=0.0,
        ignore_eos=True,
    )

    # Synthetic prompts of approximately input_len tokens — content doesn't
    # matter for kernel-counter capture, and BPE merges mean the actual token
    # count lands within ~20% of input_len. Exact length doesn't affect
    # steady-state decode counters; we just need a cheap-to-tokenize prompt.
    prompt = "hi " * bucket["input_len"]
    prompts = [prompt] * bucket["batch_size"]

    print("[driver] warmup...", flush=True)
    for _ in range(args.warmup):
        llm.generate(prompts, sampling, use_tqdm=False)

    # Application-replay: ncu runs this whole process multiple times, capturing
    # a bounded kernel window per pass. Gate the measured region with
    # torch.cuda.cudart().cudaProfilerStart/Stop so that when ncu is launched
    # with `--profile-from-start off` the cold start + warmup launches stay
    # out of the captured launch-count window.
    print("[driver] measure...", flush=True)
    profiler_started = False
    try:
        torch.cuda.cudart().cudaProfilerStart()
        profiler_started = True
    except Exception as exc:  # noqa: BLE001 — we need to surface any failure mode
        # Without Start(), `--profile-from-start off` means ncu captures nothing.
        # Warn loudly — silent degradation here produces an empty CSV that
        # looks superficially valid and wastes a full rerun to diagnose.
        print(f"[driver] WARNING: cudaProfilerStart failed ({exc!r}); "
              f"ncu capture window may be empty if --profile-from-start off is set.",
              file=sys.stderr, flush=True)

    t0 = time.perf_counter()
    for _ in range(args.iters):
        llm.generate(prompts, sampling, use_tqdm=False)
    dt = time.perf_counter() - t0

    if profiler_started:
        try:
            torch.cuda.cudart().cudaProfilerStop()
        except Exception as exc:  # noqa: BLE001
            print(f"[driver] WARNING: cudaProfilerStop failed ({exc!r})",
                  file=sys.stderr, flush=True)

    print(f"[driver] done: {args.iters} iters in {dt:.3f}s "
          f"({dt / args.iters:.3f}s/iter avg)", flush=True)
    return 0


if __name__ == "__main__":
    sys.exit(main())
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall