Skip to content

amazon-science/ammo / ai_cli_session/.codex/skills/ammo

Profile and optimize GPU kernels for vLLM inference on NVIDIA GPUs. Use when targeting a specific model, hardware, dtype, parallelism, and workload to improve latency.

FileKindSize
RUNTIME.mdmarkdown4.9 KB
SKILL.mdskill md18.5 KB
agents/ammo-auditor.mdmarkdown9.1 KB
agents/ammo-champion.mdmarkdown9.6 KB
agents/ammo-delegate.mdmarkdown3.2 KB
agents/ammo-implementer.mdmarkdown13.8 KB
agents/ammo-investigator.mdmarkdown4.0 KB
agents/ammo-report.mdmarkdown2.9 KB
agents/ammo-researcher.mdmarkdown10.3 KB
agents/ammo-resolver.mdmarkdown4.4 KB
agents/ammo-transcript-monitor.mdmarkdown7.8 KB
agents/openai.yamlexecutablescript313 B
data/ATTRIBUTION.mdmarkdown1.4 KB
data/gsm8k_full.jsonexecutablescript735.5 KB
data/gsm8k_subset.jsonexecutablescript116.8 KB
eval/EVAL-SKILL.mdexecutablescript21.8 KB
eval/agents/transcript_grader.mdexecutablescript18.0 KB
eval/causal/__init__.pyexecutablescript137 B
eval/causal/agents/causal_analyzer.mdexecutablescript10.5 KB
eval/causal/build_dag.pyexecutablescript21.6 KB
eval/causal/diff_versions.pyexecutablescript38.5 KB
eval/causal/extract_events.pyexecutablescript36.3 KB
eval/causal/generate_postmortem.pyexecutablescript17.0 KB
eval/causal/score_nodes.pyexecutablescript21.5 KB
eval/causal/templates/narrative_template.mdexecutablescript472 B
eval/causal/templates/viz_template.htmlexecutablescript9.5 KB
eval/causal/tests/__init__.pyexecutablescript151 B
eval/causal/tests/conftest.pyexecutablescript448 B
eval/causal/tests/fixtures/sample_session.jsonlexecutablescript10.4 KB
eval/causal/tests/fixtures/sample_session_data.jsonexecutablescript3.4 KB
eval/causal/tests/fixtures/sample_snapshot.jsonexecutablescript2.5 KB
eval/causal/tests/test_build_dag.pyexecutablescript2.7 KB
eval/causal/tests/test_diff_versions.pyexecutablescript13.5 KB
eval/causal/tests/test_extract_events.pyexecutablescript3.7 KB
eval/causal/tests/test_generate_postmortem.pyexecutablescript3.1 KB
eval/causal/tests/test_score_nodes.pyexecutablescript3.3 KB
eval/scripts/aggregate_versions.pyexecutablescript15.2 KB
eval/scripts/archive_run.pyexecutablescript9.1 KB
eval/scripts/codex_session.pyexecutablescript10.6 KB
eval/scripts/generate_dashboard.pyexecutablescript5.3 KB
eval/scripts/parse_artifacts.pyexecutablescript48.1 KB
eval/scripts/parse_session_logs.pyexecutablescript51.3 KB
eval/scripts/rescore_archived.pyexecutablescript5.5 KB
eval/scripts/score_campaign.pyexecutablescript43.8 KB
eval/scripts/snapshot_changes.pyexecutablescript11.1 KB
eval/scripts/verify_citations.pyexecutablescript8.8 KB
eval/tests/test_dashboard_bugs.pyexecutablescript8.8 KB
eval/tests/test_parse_session_logs.pyexecutablescript3.9 KB
eval/tests/test_parse_session_logs_round_inference.pyexecutablescript11.0 KB
eval/tests/test_score_campaign_diluted.pyexecutablescript9.4 KB
eval/viewer/dashboard_template.htmlexecutablescript15.1 KB
instrumentation/sitecustomize.pyexecutablescript3.6 KB
orchestration/audit-protocol.mdmarkdown9.3 KB
orchestration/debate-protocol.mdmarkdown11.2 KB
orchestration/integration-logic.mdmarkdown10.3 KB
orchestration/parallel-tracks.mdmarkdown8.9 KB
references/artifact-layout.mdexecutablescript22.4 KB
references/audit-invariants.mdmarkdown10.6 KB
references/champion-common-patterns.mdmarkdown3.4 KB
references/crossover-probing.mdexecutablescript4.7 KB
references/cudagraph-safety.mdmarkdown2.7 KB
references/debate-rules.mdmarkdown4.7 KB
references/debate-scoring-rubric.mdexecutablescript16.7 KB
references/e2e-delta-math.mdexecutablescript17.9 KB
references/e2e-latency-guide.mdexecutablescript9.6 KB
references/fusion-feasibility-heuristics.mdmarkdown5.6 KB
references/gpu-pool.mdexecutablescript5.4 KB
references/impl-track-rules.mdexecutablescript13.8 KB
references/nsys-profiling-guide.mdmarkdown9.8 KB
references/optimization-categories.mdexecutablescript11.0 KB
references/shutdown-protocol.mdexecutablescript3.1 KB
references/technology-selection.mdmarkdown5.0 KB
references/torch-compile-contract.mdmarkdown13.8 KB
references/validation-defaults.mdexecutablescript34.2 KB
references/workload-invocation.mdexecutablescript4.9 KB
references/writing-style.mdexecutablescript6.9 KB
scripts/ammo_state.pyexecutablescript142.0 KB
scripts/artifact_layout.jsonconfig16.3 KB
scripts/check_projection_accuracy.pyexecutablescript11.7 KB
scripts/create_evidence_template.pyexecutablescript9.7 KB
scripts/create_worktree_with_build.shexecutablescript10.2 KB
scripts/generate_validation_report.pyexecutablescript35.7 KB
scripts/gpu_reservation.pyexecutablescript27.1 KB
scripts/hook_cmd_classify.pyscript13.3 KB
scripts/mine_trace.pyscript29.4 KB
scripts/monitor_queue_ack.pyexecutablescript6.1 KB
scripts/ncu_sanity.shexecutablescript5.2 KB
scripts/ncu_sanity_driver.pyexecutablescript8.5 KB
scripts/new_target.pyexecutablescript22.0 KB
scripts/preflight_vllm_api.pyexecutablescript12.3 KB
scripts/reconcile_track_state.pyexecutablescript15.8 KB
scripts/remove_worktree_cleanup.shexecutablescript1.1 KB
scripts/resolve_codex_transcript.pyexecutablescript3.6 KB
scripts/run_vllm_bench_latency_sweep.pyexecutablescript249.6 KB
scripts/transcript_filter.pyexecutablescript38.4 KB
scripts/transitions.jsonexecutablescript1.5 KB
scripts/verify_validation_gates.pyexecutablescript88.8 KB
tests/agents/test-champion.mdexecutablescript20.4 KB
tests/agents/test-impl-champion.mdexecutablescript14.3 KB
tests/agents/test-implementer.mdexecutablescript15.8 KB
tests/agents/test-orchestrator.mdexecutablescript33.8 KB
tests/agents/test-researcher.mdexecutablescript21.0 KB
tests/agents/test-transcript-monitor.mdexecutablescript13.4 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.mdmarkdown1.1 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.summary.jsonconfig582 B
tests/fixtures/diluted_pass/legacy_no_phase.jsonconfig400 B
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.mdmarkdown2.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.summary.jsonconfig1.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.jsonconfig626 B
tests/fixtures/mining/real_campaign_bottleneck_analysis.mdmarkdown20.7 KB
tests/run_all.shexecutablescript1.6 KB
tests/scripts/test_transcript_filter.pyexecutablescript41.1 KB
tests/test_agent_runtime_contracts.pyexecutablescript1.8 KB
tests/test_ammo_state.pyexecutablescript71.7 KB
tests/test_artifact_layout_manifest.pyscript11.6 KB
tests/test_audit_started.pyscript18.2 KB
tests/test_correctness_comparator.pyscript12.6 KB
tests/test_diluted_pass.pyscript24.4 KB
tests/test_drift_compat.pyscript5.1 KB
tests/test_evidence_freshness.pyexecutablescript5.6 KB
tests/test_fail_closed.pyexecutablescript14.7 KB
tests/test_gpu_reservation.pyexecutablescript24.8 KB
tests/test_gpu_reservation_session_id.pyexecutablescript11.2 KB
tests/test_guidance_semantic_contract.pyscript24.9 KB
tests/test_hook_cmd_classify_parity.pyscript11.7 KB
tests/test_hook_semantics.pyexecutablescript95.3 KB
tests/test_mine_trace.pyscript24.2 KB
tests/test_mining_invalidated_schema.pyexecutablescript9.4 KB
tests/test_new_target_dp_ep.pyexecutablescript7.4 KB
tests/test_new_target_v2_scaffold.pyexecutablescript13.1 KB
tests/test_parse_artifacts.pyexecutablescript3.0 KB
tests/test_policy_convergence.pyscript797 B
tests/test_preflight_vllm_api.pyscript13.3 KB
tests/test_projection_accuracy_v2.pyexecutablescript4.1 KB
tests/test_reconcile_track_state.pyexecutablescript18.2 KB
tests/test_resolve_codex_transcript.pyexecutablescript3.1 KB
tests/test_state_schema_semantics.pyexecutablescript7.6 KB
tests/test_status_set_sync.pyexecutablescript9.1 KB
tests/test_sweep_baseline_from_arm.pyscript11.0 KB
tests/test_sweep_decode_capture.pyexecutablescript31.8 KB
tests/test_sweep_dp.pyexecutablescript45.4 KB
tests/test_sweep_fresh_cache.pyscript5.5 KB
tests/test_sweep_profiler_guard.pyscript22.6 KB
tests/test_sweep_real_prompts.pyscript20.1 KB
tests/test_sweep_refactor.pyexecutablescript37.6 KB
tests/test_sweep_round_slot.pyexecutablescript7.4 KB
tests/test_validation_report_phase_triage.pyexecutablescript10.0 KB
tests/test_validation_report_v2.pyexecutablescript6.2 KB
tests/test_verify_validation_gates.pyexecutablescript52.9 KB
tests/test_workload_matrix.pyexecutablescript19.3 KB

tests/test_sweep_baseline_from_arm.py

View on GitHub
#!/usr/bin/env python3
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: CC-BY-NC-4.0
"""Tests for the --baseline-from-arm sweep flag.

`--baseline-from` imports the comparator by LITERAL filename
`{prefix}_{tag}.json`, so the comparator identity is the filename prefix, not
the directory. Before this flag, a round >= 2 integration sweep pointed at a
pre-SHIP directory silently imported that directory's bf16 baseline arm and the
cumulative delta was read as incremental.

Covers:
- default is 'baseline' (byte-identical to the pre-flag behavior)
- --baseline-from-arm opt imports the 'opt' filename prefix instead
- an unknown arm is rejected by argparse
- a non-default arm that the run cannot honor is disclosed as IGNORED
- a honored non-default arm names the imported comparator in the .md report
"""

from __future__ import annotations

import json
import sys
from pathlib import Path
from typing import Any, Dict, List, Tuple
from unittest import mock

import pytest

_SCRIPTS_DIR = Path(__file__).resolve().parent.parent / "scripts"
if str(_SCRIPTS_DIR) not in sys.path:
    sys.path.insert(0, str(_SCRIPTS_DIR))

BASELINE_AVG = 0.050
IMPORTED_OPT_AVG = 0.030
CHILD_OPT_AVG = 0.020


# ---------------------------------------------------------------------------
# CLI argument parsing
# ---------------------------------------------------------------------------


def _parse_args(argv: List[str]):
    """Build the sweep argparser and parse argv (excluding program name)."""
    from run_vllm_bench_latency_sweep import main
    import argparse as _ap

    captured: Dict[str, Any] = {}

    def intercept(self, args=None, namespace=None):
        captured["parser"] = self
        raise SystemExit(0)

    with mock.patch.object(_ap.ArgumentParser, "parse_args", intercept):
        try:
            main()
        except SystemExit:
            pass

    parser = captured.get("parser")
    assert parser is not None, "Could not capture parser"
    return parser.parse_args(argv)


class TestBaselineFromArmArg:
    def test_default_is_baseline(self):
        ns = _parse_args(["--artifact-dir", "/tmp/x"])
        assert ns.baseline_from_arm == "baseline"

    def test_opt_is_accepted(self):
        ns = _parse_args(
            ["--artifact-dir", "/tmp/x", "--baseline-from-arm", "opt"]
        )
        assert ns.baseline_from_arm == "opt"

    def test_unknown_arm_is_rejected(self):
        with pytest.raises(SystemExit):
            _parse_args(
                ["--artifact-dir", "/tmp/x", "--baseline-from-arm", "mainline"]
            )


# ---------------------------------------------------------------------------
# Import behavior: which filename prefix is read
# ---------------------------------------------------------------------------


def _bench_json(avg_latency: float) -> Dict[str, Any]:
    return {
        "avg_latency": avg_latency,
        "latencies": [avg_latency, avg_latency],
        "percentiles": {
            "10": avg_latency,
            "50": avg_latency,
            "90": avg_latency,
            "99": avg_latency,
        },
    }


def _runner_json() -> Dict[str, Any]:
    return {
        "ok": True,
        "label": "test",
        "batch_size": 1,
        "start_time": "2026-01-01T00:00:00Z",
        "end_time": "2026-01-01T00:01:00Z",
        "duration_s": 60.0,
    }


def _write_target_json(artifact_dir: Path) -> None:
    target = {
        "artifact_dir": str(artifact_dir),
        "target": {
            "model_id": "test-model/test",
            "dtype": "fp16",
            "tp": 1,
            "ep": 1,
            "max_model_len": 4096,
        },
        "workload": {
            "input_len": 64,
            "output_len": 128,
            "batch_sizes": [1],
            "num_iters": 2,
        },
        "bench": {
            "runner": "vllm_bench_latency",
            "vllm_cmd": "vllm",
            "extra_args": [],
            "baseline_extra_args": [],
            "opt_extra_args": ["--some-opt-flag"],
            "baseline_env": {},
            "opt_env": {"ENABLE_OPT": "1"},
            "baseline_label": "baseline",
            "opt_label": "opt",
            "fastpath_evidence": {
                "baseline": {"require_patterns": [], "forbid_patterns": []},
                "opt": {"require_patterns": [], "forbid_patterns": []},
            },
        },
    }
    (artifact_dir / "target.json").write_text(json.dumps(target, indent=2))


def _write_stage1_dir(artifact_dir: Path) -> Path:
    """A pre-SHIP sweep dir: it holds BOTH arms, exactly as the live bug did."""
    stage1 = artifact_dir / "e2e_baseline"
    json_dir = stage1 / "json"
    json_dir.mkdir(parents=True, exist_ok=True)
    for prefix, avg in (
        ("baseline", BASELINE_AVG),
        ("opt", IMPORTED_OPT_AVG),
    ):
        (json_dir / f"{prefix}_bs1.json").write_text(json.dumps(_bench_json(avg)))
        (json_dir / f"{prefix}_bs1.runner.json").write_text(
            json.dumps(_runner_json())
        )
    return stage1


def _child_flag(cmd: List[str], flag: str) -> str | None:
    for i, arg in enumerate(cmd):
        if arg == flag and i + 1 < len(cmd):
            return cmd[i + 1]
    return None


def _child_side_effect(cmd, *, env, cwd, timeout_s, log_path, heartbeat_s,
                       status_path=None):
    """Write only the opt child's artifacts; the comparator must come from the
    import, so a mistaken re-run of a baseline arm shows up as a missing row."""
    label = _child_flag(cmd, "--_child-label")
    out_root = _child_flag(cmd, "--_out-root")
    if label and out_root:
        json_dir = Path(out_root) / "json"
        json_dir.mkdir(parents=True, exist_ok=True)
        (json_dir / f"{label}_bs1.json").write_text(
            json.dumps(_bench_json(CHILD_OPT_AVG))
        )
        (json_dir / f"{label}_bs1.runner.json").write_text(json.dumps(_runner_json()))
    return {"ok": True, "returncode": 0}


def _run_sweep(
    tmp_path: Path, extra_args: List[str], labels: str = "opt"
) -> Tuple[Dict[str, Any], str]:
    """Run one sweep and return (results JSON, results .md text)."""
    import run_vllm_bench_latency_sweep as sweep

    artifact_dir = tmp_path / "artifacts"
    artifact_dir.mkdir()
    _write_target_json(artifact_dir)
    stage1 = _write_stage1_dir(artifact_dir)

    argv = [
        "run_vllm_bench_latency_sweep.py",
        "--artifact-dir", str(artifact_dir),
        "--labels", labels,
        "--overwrite",
        "--baseline-from", str(stage1),
        # No GSM8K corpus is bundled next to the canonical script copy, and the
        # import path under test never reads prompts.
        "--dummy-prompt-source", "random",
    ] + extra_args

    with (
        mock.patch.object(sys, "argv", argv),
        mock.patch.object(sweep, "_run_cmd_streaming", side_effect=_child_side_effect),
        mock.patch("shutil.which", return_value="/usr/bin/vllm"),
    ):
        try:
            sweep.main()
        except SystemExit as exc:
            if exc.code:
                raise

    results = sorted(artifact_dir.rglob("e2e_latency_results.json"))
    assert results, "sweep wrote no e2e_latency_results.json"
    md_path = results[0].parent / "e2e_latency_results.md"
    assert md_path.exists(), "sweep wrote no e2e_latency_results.md"
    return json.loads(results[0].read_text()), md_path.read_text()


def _run_gate_sweep(tmp_path: Path, extra_args: List[str]) -> Dict[str, Any]:
    return _run_sweep(tmp_path, extra_args)[0]


class TestBaselineFromArmImport:
    def test_default_imports_the_baseline_prefix(self, tmp_path):
        """Absent the flag, the comparator is the {baseline_label}_ prefix."""
        results = _run_gate_sweep(tmp_path, [])
        row = results["results"][0]
        assert row["baseline"]["avg_s"] == pytest.approx(BASELINE_AVG)
        # Backward compat: the default run's header gains no new key.
        assert "baseline_source_arm" not in results

    def test_arm_opt_imports_the_opt_prefix(self, tmp_path):
        """--baseline-from-arm opt reads the promoted arm's files instead."""
        results = _run_gate_sweep(tmp_path, ["--baseline-from-arm", "opt"])
        row = results["results"][0]
        assert row["baseline"]["avg_s"] == pytest.approx(IMPORTED_OPT_AVG)
        assert row["baseline"]["avg_s"] != pytest.approx(BASELINE_AVG)
        # A non-default comparator arm is disclosed in the artifact.
        assert results["baseline_source_arm"] == "opt"

    def test_imported_rows_still_land_under_baseline_label(self, tmp_path):
        """The import writes under baseline_label whichever arm it read, so the
        results collection and every downstream gate keep reading one key."""
        results = _run_gate_sweep(tmp_path, ["--baseline-from-arm", "opt"])
        row = results["results"][0]
        assert row["opt"]["avg_s"] == pytest.approx(CHILD_OPT_AVG)
        assert results["bench"]["baseline_label"] == "baseline"


# ---------------------------------------------------------------------------
# Disclosure: the flag is never silently dead, and never silently honored
# ---------------------------------------------------------------------------


class TestBaselineFromArmDisclosure:
    """The .md report is what an auditor reads. Both the ignored case and the
    honored case must be visible there, not only in the JSON header."""

    def test_honored_arm_is_named_in_the_md_report(self, tmp_path):
        results, md = _run_sweep(tmp_path, ["--baseline-from-arm", "opt"])
        warnings = results.get("warnings") or []
        provenance = [w for w in warnings if "comparator imported from arm" in w]
        assert provenance, f"no comparator-provenance warning in {warnings}"
        assert "'opt'" in provenance[0]
        assert "--baseline-from-arm" in provenance[0]
        # Visible to a reader of the .md alone.
        assert "## Warnings" in md
        assert "comparator imported from arm 'opt'" in md

    def test_ignored_arm_is_disclosed_when_the_import_is_gated_off(self, tmp_path):
        """--labels baseline,opt gates the import block off, so the flag cannot
        act. The run must say so instead of implying a promoted comparator."""
        results, md = _run_sweep(
            tmp_path, ["--baseline-from-arm", "opt"], labels="baseline,opt"
        )
        warnings = results.get("warnings") or []
        ignored = [w for w in warnings if "was IGNORED" in w]
        assert ignored, f"no ignored-flag warning in {warnings}"
        assert "--baseline-from-arm opt" in ignored[0]
        assert "## Warnings" in md
        assert "--baseline-from-arm opt was IGNORED" in md
        # The gated-off run imports nothing, so it must not also claim provenance.
        assert not [w for w in warnings if "comparator imported from arm" in w]

    def test_default_run_gains_no_warning_and_no_warnings_section(self, tmp_path):
        """Backward compat: the default arm adds neither warning nor .md section."""
        results, md = _run_sweep(tmp_path, [])
        warnings = results.get("warnings") or []
        assert not [w for w in warnings if "--baseline-from-arm" in w]
        assert not [w for w in warnings if "comparator imported from arm" in w]
        assert "## Warnings" not in md
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall