Skip to content

amazon-science/ammo / ai_cli_session/.codex/skills/ammo

Profile and optimize GPU kernels for vLLM inference on NVIDIA GPUs. Use when targeting a specific model, hardware, dtype, parallelism, and workload to improve latency.

FileKindSize
RUNTIME.mdmarkdown4.9 KB
SKILL.mdskill md18.5 KB
agents/ammo-auditor.mdmarkdown9.1 KB
agents/ammo-champion.mdmarkdown9.6 KB
agents/ammo-delegate.mdmarkdown3.2 KB
agents/ammo-implementer.mdmarkdown13.8 KB
agents/ammo-investigator.mdmarkdown4.0 KB
agents/ammo-report.mdmarkdown2.9 KB
agents/ammo-researcher.mdmarkdown10.3 KB
agents/ammo-resolver.mdmarkdown4.4 KB
agents/ammo-transcript-monitor.mdmarkdown7.8 KB
agents/openai.yamlexecutablescript313 B
data/ATTRIBUTION.mdmarkdown1.4 KB
data/gsm8k_full.jsonexecutablescript735.5 KB
data/gsm8k_subset.jsonexecutablescript116.8 KB
eval/EVAL-SKILL.mdexecutablescript21.8 KB
eval/agents/transcript_grader.mdexecutablescript18.0 KB
eval/causal/__init__.pyexecutablescript137 B
eval/causal/agents/causal_analyzer.mdexecutablescript10.5 KB
eval/causal/build_dag.pyexecutablescript21.6 KB
eval/causal/diff_versions.pyexecutablescript38.5 KB
eval/causal/extract_events.pyexecutablescript36.3 KB
eval/causal/generate_postmortem.pyexecutablescript17.0 KB
eval/causal/score_nodes.pyexecutablescript21.5 KB
eval/causal/templates/narrative_template.mdexecutablescript472 B
eval/causal/templates/viz_template.htmlexecutablescript9.5 KB
eval/causal/tests/__init__.pyexecutablescript151 B
eval/causal/tests/conftest.pyexecutablescript448 B
eval/causal/tests/fixtures/sample_session.jsonlexecutablescript10.4 KB
eval/causal/tests/fixtures/sample_session_data.jsonexecutablescript3.4 KB
eval/causal/tests/fixtures/sample_snapshot.jsonexecutablescript2.5 KB
eval/causal/tests/test_build_dag.pyexecutablescript2.7 KB
eval/causal/tests/test_diff_versions.pyexecutablescript13.5 KB
eval/causal/tests/test_extract_events.pyexecutablescript3.7 KB
eval/causal/tests/test_generate_postmortem.pyexecutablescript3.1 KB
eval/causal/tests/test_score_nodes.pyexecutablescript3.3 KB
eval/scripts/aggregate_versions.pyexecutablescript15.2 KB
eval/scripts/archive_run.pyexecutablescript9.1 KB
eval/scripts/codex_session.pyexecutablescript10.6 KB
eval/scripts/generate_dashboard.pyexecutablescript5.3 KB
eval/scripts/parse_artifacts.pyexecutablescript48.1 KB
eval/scripts/parse_session_logs.pyexecutablescript51.3 KB
eval/scripts/rescore_archived.pyexecutablescript5.5 KB
eval/scripts/score_campaign.pyexecutablescript43.8 KB
eval/scripts/snapshot_changes.pyexecutablescript11.1 KB
eval/scripts/verify_citations.pyexecutablescript8.8 KB
eval/tests/test_dashboard_bugs.pyexecutablescript8.8 KB
eval/tests/test_parse_session_logs.pyexecutablescript3.9 KB
eval/tests/test_parse_session_logs_round_inference.pyexecutablescript11.0 KB
eval/tests/test_score_campaign_diluted.pyexecutablescript9.4 KB
eval/viewer/dashboard_template.htmlexecutablescript15.1 KB
instrumentation/sitecustomize.pyexecutablescript3.6 KB
orchestration/audit-protocol.mdmarkdown9.3 KB
orchestration/debate-protocol.mdmarkdown11.2 KB
orchestration/integration-logic.mdmarkdown10.3 KB
orchestration/parallel-tracks.mdmarkdown8.9 KB
references/artifact-layout.mdexecutablescript22.4 KB
references/audit-invariants.mdmarkdown10.6 KB
references/champion-common-patterns.mdmarkdown3.4 KB
references/crossover-probing.mdexecutablescript4.7 KB
references/cudagraph-safety.mdmarkdown2.7 KB
references/debate-rules.mdmarkdown4.7 KB
references/debate-scoring-rubric.mdexecutablescript16.7 KB
references/e2e-delta-math.mdexecutablescript17.9 KB
references/e2e-latency-guide.mdexecutablescript9.6 KB
references/fusion-feasibility-heuristics.mdmarkdown5.6 KB
references/gpu-pool.mdexecutablescript5.4 KB
references/impl-track-rules.mdexecutablescript13.8 KB
references/nsys-profiling-guide.mdmarkdown9.8 KB
references/optimization-categories.mdexecutablescript11.0 KB
references/shutdown-protocol.mdexecutablescript3.1 KB
references/technology-selection.mdmarkdown5.0 KB
references/torch-compile-contract.mdmarkdown13.8 KB
references/validation-defaults.mdexecutablescript34.2 KB
references/workload-invocation.mdexecutablescript4.9 KB
references/writing-style.mdexecutablescript6.9 KB
scripts/ammo_state.pyexecutablescript142.0 KB
scripts/artifact_layout.jsonconfig16.3 KB
scripts/check_projection_accuracy.pyexecutablescript11.7 KB
scripts/create_evidence_template.pyexecutablescript9.7 KB
scripts/create_worktree_with_build.shexecutablescript10.2 KB
scripts/generate_validation_report.pyexecutablescript35.7 KB
scripts/gpu_reservation.pyexecutablescript27.1 KB
scripts/hook_cmd_classify.pyscript13.3 KB
scripts/mine_trace.pyscript29.4 KB
scripts/monitor_queue_ack.pyexecutablescript6.1 KB
scripts/ncu_sanity.shexecutablescript5.2 KB
scripts/ncu_sanity_driver.pyexecutablescript8.5 KB
scripts/new_target.pyexecutablescript22.0 KB
scripts/preflight_vllm_api.pyexecutablescript12.3 KB
scripts/reconcile_track_state.pyexecutablescript15.8 KB
scripts/remove_worktree_cleanup.shexecutablescript1.1 KB
scripts/resolve_codex_transcript.pyexecutablescript3.6 KB
scripts/run_vllm_bench_latency_sweep.pyexecutablescript249.6 KB
scripts/transcript_filter.pyexecutablescript38.4 KB
scripts/transitions.jsonexecutablescript1.5 KB
scripts/verify_validation_gates.pyexecutablescript88.8 KB
tests/agents/test-champion.mdexecutablescript20.4 KB
tests/agents/test-impl-champion.mdexecutablescript14.3 KB
tests/agents/test-implementer.mdexecutablescript15.8 KB
tests/agents/test-orchestrator.mdexecutablescript33.8 KB
tests/agents/test-researcher.mdexecutablescript21.0 KB
tests/agents/test-transcript-monitor.mdexecutablescript13.4 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.mdmarkdown1.1 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.summary.jsonconfig582 B
tests/fixtures/diluted_pass/legacy_no_phase.jsonconfig400 B
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.mdmarkdown2.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.summary.jsonconfig1.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.jsonconfig626 B
tests/fixtures/mining/real_campaign_bottleneck_analysis.mdmarkdown20.7 KB
tests/run_all.shexecutablescript1.6 KB
tests/scripts/test_transcript_filter.pyexecutablescript41.1 KB
tests/test_agent_runtime_contracts.pyexecutablescript1.8 KB
tests/test_ammo_state.pyexecutablescript71.7 KB
tests/test_artifact_layout_manifest.pyscript11.6 KB
tests/test_audit_started.pyscript18.2 KB
tests/test_correctness_comparator.pyscript12.6 KB
tests/test_diluted_pass.pyscript24.4 KB
tests/test_drift_compat.pyscript5.1 KB
tests/test_evidence_freshness.pyexecutablescript5.6 KB
tests/test_fail_closed.pyexecutablescript14.7 KB
tests/test_gpu_reservation.pyexecutablescript24.8 KB
tests/test_gpu_reservation_session_id.pyexecutablescript11.2 KB
tests/test_guidance_semantic_contract.pyscript24.9 KB
tests/test_hook_cmd_classify_parity.pyscript11.7 KB
tests/test_hook_semantics.pyexecutablescript95.3 KB
tests/test_mine_trace.pyscript24.2 KB
tests/test_mining_invalidated_schema.pyexecutablescript9.4 KB
tests/test_new_target_dp_ep.pyexecutablescript7.4 KB
tests/test_new_target_v2_scaffold.pyexecutablescript13.1 KB
tests/test_parse_artifacts.pyexecutablescript3.0 KB
tests/test_policy_convergence.pyscript797 B
tests/test_preflight_vllm_api.pyscript13.3 KB
tests/test_projection_accuracy_v2.pyexecutablescript4.1 KB
tests/test_reconcile_track_state.pyexecutablescript18.2 KB
tests/test_resolve_codex_transcript.pyexecutablescript3.1 KB
tests/test_state_schema_semantics.pyexecutablescript7.6 KB
tests/test_status_set_sync.pyexecutablescript9.1 KB
tests/test_sweep_baseline_from_arm.pyscript11.0 KB
tests/test_sweep_decode_capture.pyexecutablescript31.8 KB
tests/test_sweep_dp.pyexecutablescript45.4 KB
tests/test_sweep_fresh_cache.pyscript5.5 KB
tests/test_sweep_profiler_guard.pyscript22.6 KB
tests/test_sweep_real_prompts.pyscript20.1 KB
tests/test_sweep_refactor.pyexecutablescript37.6 KB
tests/test_sweep_round_slot.pyexecutablescript7.4 KB
tests/test_validation_report_phase_triage.pyexecutablescript10.0 KB
tests/test_validation_report_v2.pyexecutablescript6.2 KB
tests/test_verify_validation_gates.pyexecutablescript52.9 KB
tests/test_workload_matrix.pyexecutablescript19.3 KB

eval/tests/test_dashboard_bugs.py

View on GitHub
#!/usr/bin/env python3
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: CC-BY-NC-4.0
"""Tests that reproduce dashboard bugs found in the eval pipeline.

Each test targets a specific bug:
- test_version_ordering_by_date: Bug 1 - versions sorted by created_at, not dir name
- test_slug_normalization: Bug 2 - bf16/bfloat16 treated as same target
- test_delta_computation_cross_target: Bug 3 - deltas null when adjacent versions differ
- test_delta_finds_same_target_predecessor: Bug 6 - delta should find most recent same-target version
- test_chart_canvas_not_destroyed: Bug 5 - Chart.js fallback preserves <canvas>
"""

import json
import sys
import tempfile
from pathlib import Path

import pytest

# Add scripts to path
SCRIPTS_DIR = Path(__file__).parent.parent / "scripts"
sys.path.insert(0, str(SCRIPTS_DIR))

from aggregate_versions import rebuild_index  # noqa: E402
from generate_dashboard import _collect_dashboard_data  # noqa: E402


def _make_scorecard(score: float, speedup: float, shipped: int) -> dict:
    """Create a minimal scorecard for testing."""
    return {
        "overall_score": score,
        "overall_score_without_transcript": score * 0.8,
        "dimensions": {
            "e2e_outcome": {"score": score * 0.4, "weight": 0.4, "weighted_contribution": score * 0.16},
            "gate_pass_rates": {"score": 5.0, "weight": 0.15, "weighted_contribution": 0.75},
            "debate_quality": {"score": 7.0, "weight": 0.15, "weighted_contribution": 1.05},
            "campaign_efficiency": {"score": 6.0, "weight": 0.15, "weighted_contribution": 0.9},
            "transcript_quality": {"score": 7.0, "weight": 0.15, "weighted_contribution": 1.05},
        },
        "raw_metrics": {
            "cumulative_speedup_vs_round1": speedup,
            "shipped_optimizations": shipped,
            "total_rounds": 1,
            "total_proposals": 3,
            "total_tracks": 3,
            "tracks_passed": shipped,
            "tracks_failed": 3 - shipped,
            "campaign_status": "converged",
        },
        "timing": {"total_tracked_seconds": 3600},
        "agent_costs": {"total_tokens": 100000, "total_duration_ms": 60000, "total_agent_invocations": 10},
    }


def _make_version(repo: Path, version_id: str, target_slug: str,
                  created_at: str, description: str,
                  score: float, speedup: float, shipped: int = 1):
    """Create a version directory with meta.json and one run."""
    vdir = repo / "versions" / version_id
    run_dir = vdir / "runs" / target_slug / "run_1"
    run_dir.mkdir(parents=True, exist_ok=True)

    meta = {
        "version_id": version_id,
        "git_commit": version_id[:9],
        "description": description,
        "created_at": created_at,
    }
    (vdir / "meta.json").write_text(json.dumps(meta))

    sc = _make_scorecard(score, speedup, shipped)
    (run_dir / "scorecard.json").write_text(json.dumps(sc))


@pytest.fixture
def repo_with_misordered_versions(tmp_path):
    """Reproduces Bug 1+2+3: versions that sort alphabetically != chronologically,
    with inconsistent slug naming."""
    repo = tmp_path / "ammo-eval"
    repo.mkdir()

    # Version A: created Mar 10, high score, git hash starts with 'f'
    _make_version(repo, "faaac90ef_campaign-A", "qwen3-5-4b_l40s_bf16_tp1",
                  "2026-03-10T20:00:00+00:00", "Campaign A", 8.99, 1.262, 2)

    # Version B: created Mar 11, mid score, git hash starts with '0'
    _make_version(repo, "06f309c83_campaign-B", "qwen3-5-4b_l40s_bf16_tp1",
                  "2026-03-11T21:00:00+00:00", "Campaign B", 6.27, 1.06, 1)

    # Version C: created Mar 13, low score, different slug (bfloat16 vs bf16)
    _make_version(repo, "99a0fbed6_campaign-C", "qwen3-5-4b_l40s_bfloat16_tp1",
                  "2026-03-13T13:00:00+00:00", "Campaign C", 3.31, 1.018, 2)

    return repo


class TestVersionOrdering:
    """Bug 1: Versions must be sorted by created_at, not directory name."""

    def test_version_ordering_by_date(self, repo_with_misordered_versions):
        repo = repo_with_misordered_versions
        index = rebuild_index(repo)
        versions = index["versions"]

        dates = [v["created_at"] for v in versions]
        assert dates == sorted(dates), (
            f"Versions not sorted chronologically: {dates}"
        )

        # Specifically: Campaign A (Mar 10) should be first
        assert "faaac90ef" in versions[0]["version_id"]
        assert "06f309c83" in versions[1]["version_id"]
        assert "99a0fbed6" in versions[2]["version_id"]


class TestSlugNormalization:
    """Bug 2: bf16 and bfloat16 should be treated as the same target."""

    def test_slug_normalization(self, repo_with_misordered_versions):
        repo = repo_with_misordered_versions
        index = rebuild_index(repo)

        target_slugs = {t["slug"] for t in index["reference_targets"]}
        # Should be 1 target, not 2
        assert len(target_slugs) == 1, (
            f"Expected 1 normalized target, got {len(target_slugs)}: {target_slugs}"
        )

    def test_all_versions_visible_for_target(self, repo_with_misordered_versions):
        repo = repo_with_misordered_versions
        index = rebuild_index(repo)

        # Pick the single normalized target slug
        target_slugs = [t["slug"] for t in index["reference_targets"]]
        assert len(target_slugs) == 1
        slug = target_slugs[0]

        # All 3 versions should have data for this target
        versions_with_data = [
            v for v in index["versions"]
            if slug in v.get("summary", {})
        ]
        assert len(versions_with_data) == 3, (
            f"Expected 3 versions with data for '{slug}', got {len(versions_with_data)}"
        )


class TestDeltaComputation:
    """Bugs 3+6: Deltas should compare same-target versions correctly."""

    def test_delta_computation_cross_target(self, repo_with_misordered_versions):
        repo = repo_with_misordered_versions
        index = rebuild_index(repo)
        data = _collect_dashboard_data(repo)

        # After slug normalization and ordering, deltas should not be null
        # for versions that evaluate the same (normalized) target
        for v in data["versions"][1:]:
            deltas = v.get("deltas", {})
            for target, d in deltas.items():
                # At least one delta should be non-null
                assert d.get("delta_score") is not None or d.get("delta_speedup") is not None, (
                    f"Delta is null for version {v['version_id']} target {target}: {d}"
                )

    def test_delta_finds_same_target_predecessor(self):
        """When versions alternate targets, delta should find the most recent
        version that evaluated the SAME target, not just version i-1."""
        with tempfile.TemporaryDirectory() as tmp:
            repo = Path(tmp) / "ammo-eval"
            repo.mkdir()

            # v1: target A
            _make_version(repo, "aaa_v1", "target_a",
                          "2026-03-01T00:00:00+00:00", "v1", 5.0, 1.05)
            # v2: target B (different target)
            _make_version(repo, "bbb_v2", "target_b",
                          "2026-03-02T00:00:00+00:00", "v2", 6.0, 1.10)
            # v3: target A again
            _make_version(repo, "ccc_v3", "target_a",
                          "2026-03-03T00:00:00+00:00", "v3", 7.0, 1.15)

            rebuild_index(repo)
            data = _collect_dashboard_data(repo)

            # v3's delta for target_a should compare against v1 (not v2)
            v3 = data["versions"][2]
            delta_a = v3.get("deltas", {}).get("target_a", {})
            assert delta_a.get("delta_score") is not None, (
                f"v3 should have a non-null delta for target_a vs v1: {delta_a}"
            )
            # Delta should be 7.0 - 5.0 = 2.0
            assert abs(delta_a["delta_score"] - 2.0) < 0.1, (
                f"Expected delta_score ~2.0, got {delta_a['delta_score']}"
            )


class TestChartCanvasPreservation:
    """Bug 5: The <canvas> element should not be destroyed by Chart.js fallback."""

    def test_chart_canvas_not_destroyed(self, repo_with_misordered_versions):
        """The dashboard template should not replace canvas innerHTML on fallback."""
        repo = repo_with_misordered_versions
        rebuild_index(repo)

        template_path = Path(__file__).parent.parent / "viewer" / "dashboard_template.html"
        template = template_path.read_text()

        # The renderTrendChart function should NOT replace parentElement.innerHTML
        # when Chart.js is unavailable. It should just return early.
        assert "parentElement.innerHTML" not in template or "return" in template.split("parentElement.innerHTML")[0].split("renderTrendChart")[-1], (
            "renderTrendChart should not destroy canvas parentElement innerHTML. "
            "Use early return instead of innerHTML replacement when Chart.js unavailable."
        )
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall