Skip to content

amazon-science/ammo / ai_cli_session/.codex/skills/ammo

Profile and optimize GPU kernels for vLLM inference on NVIDIA GPUs. Use when targeting a specific model, hardware, dtype, parallelism, and workload to improve latency.

FileKindSize
RUNTIME.mdmarkdown4.9 KB
SKILL.mdskill md18.5 KB
agents/ammo-auditor.mdmarkdown9.1 KB
agents/ammo-champion.mdmarkdown9.6 KB
agents/ammo-delegate.mdmarkdown3.2 KB
agents/ammo-implementer.mdmarkdown13.8 KB
agents/ammo-investigator.mdmarkdown4.0 KB
agents/ammo-report.mdmarkdown2.9 KB
agents/ammo-researcher.mdmarkdown10.3 KB
agents/ammo-resolver.mdmarkdown4.4 KB
agents/ammo-transcript-monitor.mdmarkdown7.8 KB
agents/openai.yamlexecutablescript313 B
data/ATTRIBUTION.mdmarkdown1.4 KB
data/gsm8k_full.jsonexecutablescript735.5 KB
data/gsm8k_subset.jsonexecutablescript116.8 KB
eval/EVAL-SKILL.mdexecutablescript21.8 KB
eval/agents/transcript_grader.mdexecutablescript18.0 KB
eval/causal/__init__.pyexecutablescript137 B
eval/causal/agents/causal_analyzer.mdexecutablescript10.5 KB
eval/causal/build_dag.pyexecutablescript21.6 KB
eval/causal/diff_versions.pyexecutablescript38.5 KB
eval/causal/extract_events.pyexecutablescript36.3 KB
eval/causal/generate_postmortem.pyexecutablescript17.0 KB
eval/causal/score_nodes.pyexecutablescript21.5 KB
eval/causal/templates/narrative_template.mdexecutablescript472 B
eval/causal/templates/viz_template.htmlexecutablescript9.5 KB
eval/causal/tests/__init__.pyexecutablescript151 B
eval/causal/tests/conftest.pyexecutablescript448 B
eval/causal/tests/fixtures/sample_session.jsonlexecutablescript10.4 KB
eval/causal/tests/fixtures/sample_session_data.jsonexecutablescript3.4 KB
eval/causal/tests/fixtures/sample_snapshot.jsonexecutablescript2.5 KB
eval/causal/tests/test_build_dag.pyexecutablescript2.7 KB
eval/causal/tests/test_diff_versions.pyexecutablescript13.5 KB
eval/causal/tests/test_extract_events.pyexecutablescript3.7 KB
eval/causal/tests/test_generate_postmortem.pyexecutablescript3.1 KB
eval/causal/tests/test_score_nodes.pyexecutablescript3.3 KB
eval/scripts/aggregate_versions.pyexecutablescript15.2 KB
eval/scripts/archive_run.pyexecutablescript9.1 KB
eval/scripts/codex_session.pyexecutablescript10.6 KB
eval/scripts/generate_dashboard.pyexecutablescript5.3 KB
eval/scripts/parse_artifacts.pyexecutablescript48.1 KB
eval/scripts/parse_session_logs.pyexecutablescript51.3 KB
eval/scripts/rescore_archived.pyexecutablescript5.5 KB
eval/scripts/score_campaign.pyexecutablescript43.8 KB
eval/scripts/snapshot_changes.pyexecutablescript11.1 KB
eval/scripts/verify_citations.pyexecutablescript8.8 KB
eval/tests/test_dashboard_bugs.pyexecutablescript8.8 KB
eval/tests/test_parse_session_logs.pyexecutablescript3.9 KB
eval/tests/test_parse_session_logs_round_inference.pyexecutablescript11.0 KB
eval/tests/test_score_campaign_diluted.pyexecutablescript9.4 KB
eval/viewer/dashboard_template.htmlexecutablescript15.1 KB
instrumentation/sitecustomize.pyexecutablescript3.6 KB
orchestration/audit-protocol.mdmarkdown9.3 KB
orchestration/debate-protocol.mdmarkdown11.2 KB
orchestration/integration-logic.mdmarkdown10.3 KB
orchestration/parallel-tracks.mdmarkdown8.9 KB
references/artifact-layout.mdexecutablescript22.4 KB
references/audit-invariants.mdmarkdown10.6 KB
references/champion-common-patterns.mdmarkdown3.4 KB
references/crossover-probing.mdexecutablescript4.7 KB
references/cudagraph-safety.mdmarkdown2.7 KB
references/debate-rules.mdmarkdown4.7 KB
references/debate-scoring-rubric.mdexecutablescript16.7 KB
references/e2e-delta-math.mdexecutablescript17.9 KB
references/e2e-latency-guide.mdexecutablescript9.6 KB
references/fusion-feasibility-heuristics.mdmarkdown5.6 KB
references/gpu-pool.mdexecutablescript5.4 KB
references/impl-track-rules.mdexecutablescript13.8 KB
references/nsys-profiling-guide.mdmarkdown9.8 KB
references/optimization-categories.mdexecutablescript11.0 KB
references/shutdown-protocol.mdexecutablescript3.1 KB
references/technology-selection.mdmarkdown5.0 KB
references/torch-compile-contract.mdmarkdown13.8 KB
references/validation-defaults.mdexecutablescript34.2 KB
references/workload-invocation.mdexecutablescript4.9 KB
references/writing-style.mdexecutablescript6.9 KB
scripts/ammo_state.pyexecutablescript142.0 KB
scripts/artifact_layout.jsonconfig16.3 KB
scripts/check_projection_accuracy.pyexecutablescript11.7 KB
scripts/create_evidence_template.pyexecutablescript9.7 KB
scripts/create_worktree_with_build.shexecutablescript10.2 KB
scripts/generate_validation_report.pyexecutablescript35.7 KB
scripts/gpu_reservation.pyexecutablescript27.1 KB
scripts/hook_cmd_classify.pyscript13.3 KB
scripts/mine_trace.pyscript29.4 KB
scripts/monitor_queue_ack.pyexecutablescript6.1 KB
scripts/ncu_sanity.shexecutablescript5.2 KB
scripts/ncu_sanity_driver.pyexecutablescript8.5 KB
scripts/new_target.pyexecutablescript22.0 KB
scripts/preflight_vllm_api.pyexecutablescript12.3 KB
scripts/reconcile_track_state.pyexecutablescript15.8 KB
scripts/remove_worktree_cleanup.shexecutablescript1.1 KB
scripts/resolve_codex_transcript.pyexecutablescript3.6 KB
scripts/run_vllm_bench_latency_sweep.pyexecutablescript249.6 KB
scripts/transcript_filter.pyexecutablescript38.4 KB
scripts/transitions.jsonexecutablescript1.5 KB
scripts/verify_validation_gates.pyexecutablescript88.8 KB
tests/agents/test-champion.mdexecutablescript20.4 KB
tests/agents/test-impl-champion.mdexecutablescript14.3 KB
tests/agents/test-implementer.mdexecutablescript15.8 KB
tests/agents/test-orchestrator.mdexecutablescript33.8 KB
tests/agents/test-researcher.mdexecutablescript21.0 KB
tests/agents/test-transcript-monitor.mdexecutablescript13.4 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.mdmarkdown1.1 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.summary.jsonconfig582 B
tests/fixtures/diluted_pass/legacy_no_phase.jsonconfig400 B
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.mdmarkdown2.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.summary.jsonconfig1.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.jsonconfig626 B
tests/fixtures/mining/real_campaign_bottleneck_analysis.mdmarkdown20.7 KB
tests/run_all.shexecutablescript1.6 KB
tests/scripts/test_transcript_filter.pyexecutablescript41.1 KB
tests/test_agent_runtime_contracts.pyexecutablescript1.8 KB
tests/test_ammo_state.pyexecutablescript71.7 KB
tests/test_artifact_layout_manifest.pyscript11.6 KB
tests/test_audit_started.pyscript18.2 KB
tests/test_correctness_comparator.pyscript12.6 KB
tests/test_diluted_pass.pyscript24.4 KB
tests/test_drift_compat.pyscript5.1 KB
tests/test_evidence_freshness.pyexecutablescript5.6 KB
tests/test_fail_closed.pyexecutablescript14.7 KB
tests/test_gpu_reservation.pyexecutablescript24.8 KB
tests/test_gpu_reservation_session_id.pyexecutablescript11.2 KB
tests/test_guidance_semantic_contract.pyscript24.9 KB
tests/test_hook_cmd_classify_parity.pyscript11.7 KB
tests/test_hook_semantics.pyexecutablescript95.3 KB
tests/test_mine_trace.pyscript24.2 KB
tests/test_mining_invalidated_schema.pyexecutablescript9.4 KB
tests/test_new_target_dp_ep.pyexecutablescript7.4 KB
tests/test_new_target_v2_scaffold.pyexecutablescript13.1 KB
tests/test_parse_artifacts.pyexecutablescript3.0 KB
tests/test_policy_convergence.pyscript797 B
tests/test_preflight_vllm_api.pyscript13.3 KB
tests/test_projection_accuracy_v2.pyexecutablescript4.1 KB
tests/test_reconcile_track_state.pyexecutablescript18.2 KB
tests/test_resolve_codex_transcript.pyexecutablescript3.1 KB
tests/test_state_schema_semantics.pyexecutablescript7.6 KB
tests/test_status_set_sync.pyexecutablescript9.1 KB
tests/test_sweep_baseline_from_arm.pyscript11.0 KB
tests/test_sweep_decode_capture.pyexecutablescript31.8 KB
tests/test_sweep_dp.pyexecutablescript45.4 KB
tests/test_sweep_fresh_cache.pyscript5.5 KB
tests/test_sweep_profiler_guard.pyscript22.6 KB
tests/test_sweep_real_prompts.pyscript20.1 KB
tests/test_sweep_refactor.pyexecutablescript37.6 KB
tests/test_sweep_round_slot.pyexecutablescript7.4 KB
tests/test_validation_report_phase_triage.pyexecutablescript10.0 KB
tests/test_validation_report_v2.pyexecutablescript6.2 KB
tests/test_verify_validation_gates.pyexecutablescript52.9 KB
tests/test_workload_matrix.pyexecutablescript19.3 KB

tests/test_workload_matrix.py

View on GitHub
#!/usr/bin/env python3
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: CC-BY-NC-4.0
"""TDD tests for multi-dimensional workload matrix sweeps.

Plan: .codex/plans/workload-matrix-sweep.md

Tests cover:
  Phase 1: _expand_workload_to_buckets
  Phase 2: _bucket_file_tag
  Phase 3: _render_md_table (heterogeneous columns)
  Phase 4: _validate_buckets_model_len
  Phase 7: generate_validation_report._render_e2e_section (heterogeneous rows)
"""

from __future__ import annotations

import sys
from pathlib import Path

import pytest

# Ensure the scripts directory is importable.
_SCRIPTS_DIR = Path(__file__).resolve().parent.parent / "scripts"
if str(_SCRIPTS_DIR) not in sys.path:
    sys.path.insert(0, str(_SCRIPTS_DIR))


# ---------------------------------------------------------------------------
# Phase 1: _expand_workload_to_buckets
# ---------------------------------------------------------------------------

class TestExpandWorkloadToBuckets:
    """Tests for converting workload spec → list of bucket dicts."""

    def _call(self, workload):
        from run_vllm_bench_latency_sweep import _expand_workload_to_buckets
        return _expand_workload_to_buckets(workload)

    def test_legacy_flat_format(self):
        """Legacy flat {input_len, output_len, batch_sizes} → correct buckets."""
        wl = {"input_len": 64, "output_len": 512, "batch_sizes": [1, 4, 8], "num_iters": 5}
        buckets = self._call(wl)
        assert buckets == [
            {"input_len": 64, "output_len": 512, "batch_size": 1},
            {"input_len": 64, "output_len": 512, "batch_size": 4},
            {"input_len": 64, "output_len": 512, "batch_size": 8},
        ]

    def test_matrix_format(self):
        """New workload_matrix format → correct buckets in order."""
        wl = {
            "input_len": 64,
            "output_len": 512,
            "batch_sizes": [1],
            "num_iters": 5,
            "workload_matrix": [
                {"input_len": 128, "output_len": 256, "batch_size": 2},
                {"input_len": 64, "output_len": 512, "batch_size": 8},
            ],
        }
        buckets = self._call(wl)
        assert buckets == [
            {"input_len": 128, "output_len": 256, "batch_size": 2},
            {"input_len": 64, "output_len": 512, "batch_size": 8},
        ]

    def test_matrix_overrides_flat(self):
        """When workload_matrix is present, flat batch_sizes are ignored."""
        wl = {
            "input_len": 64,
            "output_len": 512,
            "batch_sizes": [1, 4, 8],
            "num_iters": 5,
            "workload_matrix": [
                {"input_len": 128, "output_len": 256, "batch_size": 2},
            ],
        }
        buckets = self._call(wl)
        assert len(buckets) == 1
        assert buckets[0]["batch_size"] == 2

    def test_matrix_missing_keys_raises(self):
        """Each matrix entry must have input_len, output_len, batch_size."""
        wl = {
            "input_len": 64,
            "output_len": 512,
            "batch_sizes": [1],
            "num_iters": 5,
            "workload_matrix": [
                {"input_len": 128, "batch_size": 2},  # missing output_len
            ],
        }
        with pytest.raises(SystemExit):
            self._call(wl)

    def test_empty_matrix_raises(self):
        """An empty workload_matrix should raise."""
        wl = {
            "input_len": 64,
            "output_len": 512,
            "batch_sizes": [1],
            "num_iters": 5,
            "workload_matrix": [],
        }
        with pytest.raises(SystemExit):
            self._call(wl)

    def test_duplicate_tuples_raises(self):
        """Duplicate (input_len, output_len, batch_size) tuples should raise."""
        wl = {
            "input_len": 64,
            "output_len": 512,
            "batch_sizes": [1],
            "num_iters": 5,
            "workload_matrix": [
                {"input_len": 128, "output_len": 256, "batch_size": 2},
                {"input_len": 128, "output_len": 256, "batch_size": 2},
            ],
        }
        with pytest.raises(SystemExit):
            self._call(wl)

    def test_legacy_flat_no_matrix_key(self):
        """Legacy format works even when workload_matrix key is absent."""
        wl = {"input_len": 32, "output_len": 128, "batch_sizes": [16], "num_iters": 3}
        buckets = self._call(wl)
        assert len(buckets) == 1
        assert buckets[0] == {"input_len": 32, "output_len": 128, "batch_size": 16}


# ---------------------------------------------------------------------------
# Phase 2: _bucket_file_tag
# ---------------------------------------------------------------------------

class TestBucketFileTag:
    """Tests for file naming helper."""

    def _call(self, bucket, all_buckets):
        from run_vllm_bench_latency_sweep import _bucket_file_tag
        return _bucket_file_tag(bucket, all_buckets)

    def test_homogeneous_short_form(self):
        """All buckets share same IL/OL → short form bs{BS}."""
        buckets = [
            {"input_len": 64, "output_len": 512, "batch_size": 1},
            {"input_len": 64, "output_len": 512, "batch_size": 4},
            {"input_len": 64, "output_len": 512, "batch_size": 8},
        ]
        assert self._call(buckets[0], buckets) == "bs1"
        assert self._call(buckets[1], buckets) == "bs4"
        assert self._call(buckets[2], buckets) == "bs8"

    def test_heterogeneous_long_form(self):
        """Mixed IL/OL across buckets → long form il{IL}_ol{OL}_bs{BS}."""
        buckets = [
            {"input_len": 64, "output_len": 512, "batch_size": 1},
            {"input_len": 128, "output_len": 256, "batch_size": 4},
        ]
        assert self._call(buckets[0], buckets) == "il64_ol512_bs1"
        assert self._call(buckets[1], buckets) == "il128_ol256_bs4"

    def test_single_bucket_short_form(self):
        """A single bucket is always homogeneous → short form."""
        buckets = [{"input_len": 64, "output_len": 512, "batch_size": 1}]
        assert self._call(buckets[0], buckets) == "bs1"


# ---------------------------------------------------------------------------
# Phase 3: _render_md_table (heterogeneous IL/OL columns)
# ---------------------------------------------------------------------------

class TestRenderMdTable:
    """Tests for MD table rendering with optional IL/OL columns."""

    def _call(self, rows, baseline_label="baseline", opt_label="opt"):
        from run_vllm_bench_latency_sweep import _render_md_table
        return _render_md_table(rows, baseline_label, opt_label)

    def test_homogeneous_omits_il_ol(self):
        """When all rows share the same IL/OL (or have no IL/OL), no IL/OL columns."""
        rows = [
            {
                "batch_size": 1,
                "input_len": 64,
                "output_len": 512,
                "baseline": {"avg_s": 1.0, "fastpath_evidence": {"status": "unknown"}},
                "opt": {"avg_s": 0.8, "fastpath_evidence": {"status": "pass"}},
                "speedup": 1.25,
                "improvement_pct": 20.0,
            },
            {
                "batch_size": 4,
                "input_len": 64,
                "output_len": 512,
                "baseline": {"avg_s": 2.0, "fastpath_evidence": {"status": "unknown"}},
                "opt": {"avg_s": 1.5, "fastpath_evidence": {"status": "pass"}},
                "speedup": 1.333,
                "improvement_pct": 25.0,
            },
        ]
        md = self._call(rows)
        assert "Input Len" not in md
        assert "Output Len" not in md
        # Batch size column should still be present.
        assert "Batch Size" in md

    def test_heterogeneous_includes_il_ol(self):
        """When rows have different IL/OL, add Input Len and Output Len columns."""
        rows = [
            {
                "batch_size": 1,
                "input_len": 64,
                "output_len": 512,
                "baseline": {"avg_s": 1.0, "fastpath_evidence": {"status": "unknown"}},
                "opt": {"avg_s": 0.8, "fastpath_evidence": {"status": "pass"}},
                "speedup": 1.25,
                "improvement_pct": 20.0,
            },
            {
                "batch_size": 4,
                "input_len": 128,
                "output_len": 256,
                "baseline": {"avg_s": 2.0, "fastpath_evidence": {"status": "unknown"}},
                "opt": {"avg_s": 1.5, "fastpath_evidence": {"status": "pass"}},
                "speedup": 1.333,
                "improvement_pct": 25.0,
            },
        ]
        md = self._call(rows)
        assert "Input Len" in md
        assert "Output Len" in md

    def test_legacy_rows_without_il_ol_no_crash(self):
        """Legacy rows that don't have input_len/output_len keys should not crash."""
        rows = [
            {
                "batch_size": 1,
                "baseline": {"avg_s": 1.0, "fastpath_evidence": {"status": "unknown"}},
                "opt": {"avg_s": 0.8, "fastpath_evidence": {"status": "pass"}},
                "speedup": 1.25,
                "improvement_pct": 20.0,
            },
        ]
        md = self._call(rows)
        assert "Batch Size" in md
        # Should not crash — just omit IL/OL columns.
        assert "Input Len" not in md


# ---------------------------------------------------------------------------
# Phase 4: _validate_buckets_model_len
# ---------------------------------------------------------------------------

class TestValidateBucketsModelLen:
    """Tests for max_model_len validation against buckets."""

    def _call(self, buckets, max_model_len):
        from run_vllm_bench_latency_sweep import _validate_buckets_model_len
        return _validate_buckets_model_len(buckets, max_model_len)

    def test_invalid_bucket_raises(self):
        """Bucket with input_len + output_len > max_model_len should raise."""
        buckets = [{"input_len": 2048, "output_len": 2048, "batch_size": 1}]
        with pytest.raises(SystemExit):
            self._call(buckets, max_model_len=2048)

    def test_all_valid_passes(self):
        """Bucket with input_len + output_len <= max_model_len should pass."""
        buckets = [
            {"input_len": 64, "output_len": 512, "batch_size": 1},
            {"input_len": 128, "output_len": 256, "batch_size": 4},
        ]
        # Should not raise.
        self._call(buckets, max_model_len=4096)

    def test_exact_boundary_passes(self):
        """Bucket with input_len + output_len == max_model_len should pass."""
        buckets = [{"input_len": 2048, "output_len": 2048, "batch_size": 1}]
        self._call(buckets, max_model_len=4096)


# ---------------------------------------------------------------------------
# Phase 7: generate_validation_report._render_e2e_section
# ---------------------------------------------------------------------------

class TestRenderE2ESection:
    """Tests for the validation report E2E section with heterogeneous rows."""

    def _call(self, e2e):
        from generate_validation_report import _render_e2e_section
        return _render_e2e_section(e2e)

    def test_heterogeneous_rows_render(self):
        """Heterogeneous rows (different IL/OL per row) should render IL/OL columns."""
        e2e = {
            "model_id": "test-model",
            "tp": 1,
            "max_model_len": 4096,
            "bench": {"baseline_label": "baseline", "opt_label": "opt"},
            "workload": {"num_iters": 5},
            "results": [
                {
                    "batch_size": 1,
                    "input_len": 64,
                    "output_len": 512,
                    "baseline": {"avg_s": 1.0, "fastpath_evidence": {"status": "unknown"}},
                    "opt": {"avg_s": 0.8, "fastpath_evidence": {"status": "pass"}},
                    "speedup": 1.25,
                    "improvement_pct": 20.0,
                },
                {
                    "batch_size": 4,
                    "input_len": 128,
                    "output_len": 256,
                    "baseline": {"avg_s": 2.0, "fastpath_evidence": {"status": "unknown"}},
                    "opt": {"avg_s": 1.5, "fastpath_evidence": {"status": "pass"}},
                    "speedup": 1.333,
                    "improvement_pct": 25.0,
                },
            ],
        }
        md = self._call(e2e)
        assert "Input Len" in md
        assert "Output Len" in md

    def test_legacy_rows_still_work(self):
        """Legacy rows (same IL/OL or missing IL/OL) should still render fine."""
        e2e = {
            "model_id": "test-model",
            "tp": 1,
            "max_model_len": 4096,
            "bench": {"baseline_label": "baseline", "opt_label": "opt"},
            "workload": {"input_len": 64, "output_len": 512, "num_iters": 5},
            "results": [
                {
                    "batch_size": 1,
                    "baseline": {"avg_s": 1.0, "fastpath_evidence": {"status": "unknown"}},
                    "opt": {"avg_s": 0.8, "fastpath_evidence": {"status": "pass"}},
                    "speedup": 1.25,
                    "improvement_pct": 20.0,
                },
            ],
        }
        md = self._call(e2e)
        # Should not crash. Should not include IL/OL columns for legacy data.
        assert "Batch Size" in md
        assert "Input Len" not in md

    def test_gate_logic_unchanged(self):
        """Gate logic in main() still operates on batch_size — verify section renders."""
        e2e = {
            "model_id": "test-model",
            "tp": 1,
            "max_model_len": 4096,
            "bench": {"baseline_label": "baseline", "opt_label": "opt"},
            "workload": {"input_len": 64, "output_len": 512, "num_iters": 5},
            "results": [
                {
                    "batch_size": 1,
                    "baseline": {"avg_s": 1.0, "fastpath_evidence": {"status": "unknown"}},
                    "opt": {"avg_s": 0.8, "fastpath_evidence": {"status": "pass"}},
                    "speedup": 1.25,
                    "improvement_pct": 20.0,
                },
            ],
        }
        md = self._call(e2e)
        # The section should include a table row for BS=1.
        assert "| 1 |" in md


class TestNsysProfileIntegration:
    """Tests for nsys profiling integration (prefix construction, file rename, error handling)."""

    def _build_nsys_prefix(self, buckets, label="baseline", nsys_dir="/tmp/nsys", nsys_trace="cuda"):
        """Replicate the nsys prefix construction logic from main()."""
        return [
            "nsys", "profile",
            f"--trace={nsys_trace},nvtx",
            "--sample=none",
            "--capture-range=cudaProfilerApi",
            f"--capture-range-end=repeat:{len(buckets)}",
            "--cuda-graph-trace=node",
            "--trace-fork-before-exec=true",
            "--force-overwrite=true",
            "-o", f"{nsys_dir}/{label}_profile",
        ]

    def test_nsys_prefix_repeat_count_matches_buckets(self):
        """Verify repeat:N in constructed prefix matches bucket count."""
        buckets = [
            {"input_len": 64, "output_len": 512, "batch_size": 1},
            {"input_len": 64, "output_len": 512, "batch_size": 8},
            {"input_len": 64, "output_len": 512, "batch_size": 32},
        ]
        prefix = self._build_nsys_prefix(buckets)
        assert "--capture-range-end=repeat:3" in prefix
        assert "--capture-range=cudaProfilerApi" in prefix
        assert "--cuda-graph-trace=node" in prefix

    def test_nsys_prefix_single_bucket(self):
        """Single bucket should produce repeat:1."""
        buckets = [{"input_len": 64, "output_len": 512, "batch_size": 1}]
        prefix = self._build_nsys_prefix(buckets)
        assert "--capture-range-end=repeat:1" in prefix

    def test_nsys_prefix_supports_cuda_sw_trace_backend(self):
        """Blackwell (B200/B300) runs use software CUDA tracing."""
        buckets = [{"input_len": 64, "output_len": 2, "batch_size": 8}]
        prefix = self._build_nsys_prefix(buckets, nsys_trace="cuda-sw")
        assert "--trace=cuda-sw,nvtx" in prefix
        assert "--cuda-graph-trace=node" in prefix

    def test_nsys_file_rename_sequence(self, tmp_path):
        """Verify actual file rename from nsys sequential numbering to bucket tags."""
        from run_vllm_bench_latency_sweep import _bucket_file_tag

        nsys_dir = tmp_path / "nsys"
        nsys_dir.mkdir()
        buckets = [
            {"input_len": 64, "output_len": 512, "batch_size": 1},
            {"input_len": 64, "output_len": 512, "batch_size": 8},
        ]
        # Create fake nsys output files as nsys repeat mode would
        for i in range(1, len(buckets) + 1):
            (nsys_dir / f"baseline_profile.{i}.nsys-rep").write_text(f"fake-trace-{i}")

        # Replicate the parent's rename logic
        for i, bucket in enumerate(buckets, 1):
            src = nsys_dir / f"baseline_profile.{i}.nsys-rep"
            tag = _bucket_file_tag(bucket, buckets)
            dst = nsys_dir / f"baseline_{tag}.nsys-rep"
            if src.exists():
                src.rename(dst)

        # Verify renamed files exist with correct names and original content
        assert (nsys_dir / "baseline_bs1.nsys-rep").read_text() == "fake-trace-1"
        assert (nsys_dir / "baseline_bs8.nsys-rep").read_text() == "fake-trace-2"
        assert not (nsys_dir / "baseline_profile.1.nsys-rep").exists()
        assert not (nsys_dir / "baseline_profile.2.nsys-rep").exists()

    def test_nsys_file_rename_heterogeneous(self, tmp_path):
        """Verify rename uses long-form tags for heterogeneous buckets."""
        from run_vllm_bench_latency_sweep import _bucket_file_tag

        nsys_dir = tmp_path / "nsys"
        nsys_dir.mkdir()
        buckets = [
            {"input_len": 64, "output_len": 512, "batch_size": 1},
            {"input_len": 128, "output_len": 256, "batch_size": 4},
        ]
        for i in range(1, len(buckets) + 1):
            (nsys_dir / f"opt_profile.{i}.nsys-rep").write_text(f"data-{i}")

        for i, bucket in enumerate(buckets, 1):
            src = nsys_dir / f"opt_profile.{i}.nsys-rep"
            tag = _bucket_file_tag(bucket, buckets)
            dst = nsys_dir / f"opt_{tag}.nsys-rep"
            if src.exists():
                src.rename(dst)

        assert (nsys_dir / "opt_il64_ol512_bs1.nsys-rep").exists()
        assert (nsys_dir / "opt_il128_ol256_bs4.nsys-rep").exists()

    def test_nsys_rename_missing_files_no_crash(self, tmp_path):
        """Rename logic should silently skip missing nsys output files."""
        from run_vllm_bench_latency_sweep import _bucket_file_tag

        nsys_dir = tmp_path / "nsys"
        nsys_dir.mkdir()
        buckets = [
            {"input_len": 64, "output_len": 512, "batch_size": 1},
            {"input_len": 64, "output_len": 512, "batch_size": 8},
        ]
        # Only create file for first bucket (simulate nsys failure on second)
        (nsys_dir / "baseline_profile.1.nsys-rep").write_text("ok")

        for i, bucket in enumerate(buckets, 1):
            src = nsys_dir / f"baseline_profile.{i}.nsys-rep"
            tag = _bucket_file_tag(bucket, buckets)
            dst = nsys_dir / f"baseline_{tag}.nsys-rep"
            if src.exists():
                src.rename(dst)

        assert (nsys_dir / "baseline_bs1.nsys-rep").exists()
        assert not (nsys_dir / "baseline_bs8.nsys-rep").exists()  # gracefully missing


if __name__ == "__main__":
    pytest.main([__file__, "-v"])
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall