Skip to content

amazon-science/ammo / ai_cli_session/.codex/skills/ammo

Profile and optimize GPU kernels for vLLM inference on NVIDIA GPUs. Use when targeting a specific model, hardware, dtype, parallelism, and workload to improve latency.

FileKindSize
RUNTIME.mdmarkdown4.9 KB
SKILL.mdskill md18.5 KB
agents/ammo-auditor.mdmarkdown9.1 KB
agents/ammo-champion.mdmarkdown9.6 KB
agents/ammo-delegate.mdmarkdown3.2 KB
agents/ammo-implementer.mdmarkdown13.8 KB
agents/ammo-investigator.mdmarkdown4.0 KB
agents/ammo-report.mdmarkdown2.9 KB
agents/ammo-researcher.mdmarkdown10.3 KB
agents/ammo-resolver.mdmarkdown4.4 KB
agents/ammo-transcript-monitor.mdmarkdown7.8 KB
agents/openai.yamlexecutablescript313 B
data/ATTRIBUTION.mdmarkdown1.4 KB
data/gsm8k_full.jsonexecutablescript735.5 KB
data/gsm8k_subset.jsonexecutablescript116.8 KB
eval/EVAL-SKILL.mdexecutablescript21.8 KB
eval/agents/transcript_grader.mdexecutablescript18.0 KB
eval/causal/__init__.pyexecutablescript137 B
eval/causal/agents/causal_analyzer.mdexecutablescript10.5 KB
eval/causal/build_dag.pyexecutablescript21.6 KB
eval/causal/diff_versions.pyexecutablescript38.5 KB
eval/causal/extract_events.pyexecutablescript36.3 KB
eval/causal/generate_postmortem.pyexecutablescript17.0 KB
eval/causal/score_nodes.pyexecutablescript21.5 KB
eval/causal/templates/narrative_template.mdexecutablescript472 B
eval/causal/templates/viz_template.htmlexecutablescript9.5 KB
eval/causal/tests/__init__.pyexecutablescript151 B
eval/causal/tests/conftest.pyexecutablescript448 B
eval/causal/tests/fixtures/sample_session.jsonlexecutablescript10.4 KB
eval/causal/tests/fixtures/sample_session_data.jsonexecutablescript3.4 KB
eval/causal/tests/fixtures/sample_snapshot.jsonexecutablescript2.5 KB
eval/causal/tests/test_build_dag.pyexecutablescript2.7 KB
eval/causal/tests/test_diff_versions.pyexecutablescript13.5 KB
eval/causal/tests/test_extract_events.pyexecutablescript3.7 KB
eval/causal/tests/test_generate_postmortem.pyexecutablescript3.1 KB
eval/causal/tests/test_score_nodes.pyexecutablescript3.3 KB
eval/scripts/aggregate_versions.pyexecutablescript15.2 KB
eval/scripts/archive_run.pyexecutablescript9.1 KB
eval/scripts/codex_session.pyexecutablescript10.6 KB
eval/scripts/generate_dashboard.pyexecutablescript5.3 KB
eval/scripts/parse_artifacts.pyexecutablescript48.1 KB
eval/scripts/parse_session_logs.pyexecutablescript51.3 KB
eval/scripts/rescore_archived.pyexecutablescript5.5 KB
eval/scripts/score_campaign.pyexecutablescript43.8 KB
eval/scripts/snapshot_changes.pyexecutablescript11.1 KB
eval/scripts/verify_citations.pyexecutablescript8.8 KB
eval/tests/test_dashboard_bugs.pyexecutablescript8.8 KB
eval/tests/test_parse_session_logs.pyexecutablescript3.9 KB
eval/tests/test_parse_session_logs_round_inference.pyexecutablescript11.0 KB
eval/tests/test_score_campaign_diluted.pyexecutablescript9.4 KB
eval/viewer/dashboard_template.htmlexecutablescript15.1 KB
instrumentation/sitecustomize.pyexecutablescript3.6 KB
orchestration/audit-protocol.mdmarkdown9.3 KB
orchestration/debate-protocol.mdmarkdown11.2 KB
orchestration/integration-logic.mdmarkdown10.3 KB
orchestration/parallel-tracks.mdmarkdown8.9 KB
references/artifact-layout.mdexecutablescript22.4 KB
references/audit-invariants.mdmarkdown10.6 KB
references/champion-common-patterns.mdmarkdown3.4 KB
references/crossover-probing.mdexecutablescript4.7 KB
references/cudagraph-safety.mdmarkdown2.7 KB
references/debate-rules.mdmarkdown4.7 KB
references/debate-scoring-rubric.mdexecutablescript16.7 KB
references/e2e-delta-math.mdexecutablescript17.9 KB
references/e2e-latency-guide.mdexecutablescript9.6 KB
references/fusion-feasibility-heuristics.mdmarkdown5.6 KB
references/gpu-pool.mdexecutablescript5.4 KB
references/impl-track-rules.mdexecutablescript13.8 KB
references/nsys-profiling-guide.mdmarkdown9.8 KB
references/optimization-categories.mdexecutablescript11.0 KB
references/shutdown-protocol.mdexecutablescript3.1 KB
references/technology-selection.mdmarkdown5.0 KB
references/torch-compile-contract.mdmarkdown13.8 KB
references/validation-defaults.mdexecutablescript34.2 KB
references/workload-invocation.mdexecutablescript4.9 KB
references/writing-style.mdexecutablescript6.9 KB
scripts/ammo_state.pyexecutablescript142.0 KB
scripts/artifact_layout.jsonconfig16.3 KB
scripts/check_projection_accuracy.pyexecutablescript11.7 KB
scripts/create_evidence_template.pyexecutablescript9.7 KB
scripts/create_worktree_with_build.shexecutablescript10.2 KB
scripts/generate_validation_report.pyexecutablescript35.7 KB
scripts/gpu_reservation.pyexecutablescript27.1 KB
scripts/hook_cmd_classify.pyscript13.3 KB
scripts/mine_trace.pyscript29.4 KB
scripts/monitor_queue_ack.pyexecutablescript6.1 KB
scripts/ncu_sanity.shexecutablescript5.2 KB
scripts/ncu_sanity_driver.pyexecutablescript8.5 KB
scripts/new_target.pyexecutablescript22.0 KB
scripts/preflight_vllm_api.pyexecutablescript12.3 KB
scripts/reconcile_track_state.pyexecutablescript15.8 KB
scripts/remove_worktree_cleanup.shexecutablescript1.1 KB
scripts/resolve_codex_transcript.pyexecutablescript3.6 KB
scripts/run_vllm_bench_latency_sweep.pyexecutablescript249.6 KB
scripts/transcript_filter.pyexecutablescript38.4 KB
scripts/transitions.jsonexecutablescript1.5 KB
scripts/verify_validation_gates.pyexecutablescript88.8 KB
tests/agents/test-champion.mdexecutablescript20.4 KB
tests/agents/test-impl-champion.mdexecutablescript14.3 KB
tests/agents/test-implementer.mdexecutablescript15.8 KB
tests/agents/test-orchestrator.mdexecutablescript33.8 KB
tests/agents/test-researcher.mdexecutablescript21.0 KB
tests/agents/test-transcript-monitor.mdexecutablescript13.4 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.mdmarkdown1.1 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.summary.jsonconfig582 B
tests/fixtures/diluted_pass/legacy_no_phase.jsonconfig400 B
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.mdmarkdown2.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.summary.jsonconfig1.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.jsonconfig626 B
tests/fixtures/mining/real_campaign_bottleneck_analysis.mdmarkdown20.7 KB
tests/run_all.shexecutablescript1.6 KB
tests/scripts/test_transcript_filter.pyexecutablescript41.1 KB
tests/test_agent_runtime_contracts.pyexecutablescript1.8 KB
tests/test_ammo_state.pyexecutablescript71.7 KB
tests/test_artifact_layout_manifest.pyscript11.6 KB
tests/test_audit_started.pyscript18.2 KB
tests/test_correctness_comparator.pyscript12.6 KB
tests/test_diluted_pass.pyscript24.4 KB
tests/test_drift_compat.pyscript5.1 KB
tests/test_evidence_freshness.pyexecutablescript5.6 KB
tests/test_fail_closed.pyexecutablescript14.7 KB
tests/test_gpu_reservation.pyexecutablescript24.8 KB
tests/test_gpu_reservation_session_id.pyexecutablescript11.2 KB
tests/test_guidance_semantic_contract.pyscript24.9 KB
tests/test_hook_cmd_classify_parity.pyscript11.7 KB
tests/test_hook_semantics.pyexecutablescript95.3 KB
tests/test_mine_trace.pyscript24.2 KB
tests/test_mining_invalidated_schema.pyexecutablescript9.4 KB
tests/test_new_target_dp_ep.pyexecutablescript7.4 KB
tests/test_new_target_v2_scaffold.pyexecutablescript13.1 KB
tests/test_parse_artifacts.pyexecutablescript3.0 KB
tests/test_policy_convergence.pyscript797 B
tests/test_preflight_vllm_api.pyscript13.3 KB
tests/test_projection_accuracy_v2.pyexecutablescript4.1 KB
tests/test_reconcile_track_state.pyexecutablescript18.2 KB
tests/test_resolve_codex_transcript.pyexecutablescript3.1 KB
tests/test_state_schema_semantics.pyexecutablescript7.6 KB
tests/test_status_set_sync.pyexecutablescript9.1 KB
tests/test_sweep_baseline_from_arm.pyscript11.0 KB
tests/test_sweep_decode_capture.pyexecutablescript31.8 KB
tests/test_sweep_dp.pyexecutablescript45.4 KB
tests/test_sweep_fresh_cache.pyscript5.5 KB
tests/test_sweep_profiler_guard.pyscript22.6 KB
tests/test_sweep_real_prompts.pyscript20.1 KB
tests/test_sweep_refactor.pyexecutablescript37.6 KB
tests/test_sweep_round_slot.pyexecutablescript7.4 KB
tests/test_validation_report_phase_triage.pyexecutablescript10.0 KB
tests/test_validation_report_v2.pyexecutablescript6.2 KB
tests/test_verify_validation_gates.pyexecutablescript52.9 KB
tests/test_workload_matrix.pyexecutablescript19.3 KB

eval/scripts/verify_citations.py

View on GitHub
#!/usr/bin/env python3
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: CC-BY-NC-4.0
"""Verify structured citations in AMMO eval agent outputs.

Checks that every citation in an agent's JSON output:
  1. References a file that exists (relative to artifact dir or absolute)
  2. Has a line range within the file's actual line count
  3. Has quoted_content that fuzzy-matches the actual content (>70% similarity)

Usage:
  python verify_citations.py --input /tmp/ammo_eval_deep_analysis.json --artifact-dir kernel_opt_artifacts/...
  python verify_citations.py --input /tmp/ammo_eval_transcript_grading.json --artifact-dir kernel_opt_artifacts/...
  python verify_citations.py --input /tmp/ammo_eval_deep_analysis.json --artifact-dir kernel_opt_artifacts/... --strict
"""

from __future__ import annotations

import argparse
import json
import sys
from difflib import SequenceMatcher
from pathlib import Path
from typing import Any


def _fuzzy_match(quoted: str, actual: str) -> float:
    """Return similarity ratio between quoted and actual content (0.0–1.0).

    The rubric tells agents to truncate quoted_content to ~200 chars, so the
    quote is often a prefix/substring of the full line range content. We handle
    this by: (1) if the normalized quote appears verbatim in the normalized
    actual, return 1.0; (2) otherwise, compare the quote against a window of
    the actual text of equal length (sliding window best match); (3) fall back
    to SequenceMatcher on the full strings.
    """
    quoted_norm = " ".join(quoted.split())
    actual_norm = " ".join(actual.split())

    if not quoted_norm:
        return 0.0

    # Exact substring match
    if quoted_norm in actual_norm:
        return 1.0

    # Sliding window: find the best-matching window in actual of same length
    # as quoted (handles truncation + minor reformatting)
    qlen = len(quoted_norm)
    if len(actual_norm) > qlen:
        best = 0.0
        # Sample windows at stride of max(1, qlen//4) for performance
        stride = max(1, qlen // 4)
        for i in range(0, len(actual_norm) - qlen + 1, stride):
            window = actual_norm[i : i + qlen]
            score = SequenceMatcher(None, quoted_norm, window).ratio()
            if score > best:
                best = score
                if best > 0.95:
                    break  # close enough
        return best

    # Same length or quoted is longer — direct comparison
    return SequenceMatcher(None, quoted_norm, actual_norm).ratio()


def _resolve_file(file_path: str, artifact_dir: Path) -> Path | None:
    """Resolve a citation file_path to an actual filesystem path."""
    p = Path(file_path)

    # Absolute path — use directly
    if p.is_absolute():
        return p if p.exists() else None

    # Relative to artifact dir
    candidate = artifact_dir / p
    if candidate.exists():
        return candidate

    # Try with investigation/ prefix (common for bottleneck_analysis.md)
    candidate = artifact_dir / "investigation" / p
    if candidate.exists():
        return candidate

    return None


def _extract_citations(obj: Any) -> list[dict]:
    """Recursively extract all citation objects from a JSON structure.

    A citation object has all of: file_path, line_start, line_end, claim,
    quoted_content. This walks dicts and lists to find them regardless of
    nesting depth.
    """
    REQUIRED = {"file_path", "line_start", "line_end", "claim", "quoted_content"}
    results = []

    if isinstance(obj, dict):
        if REQUIRED.issubset(obj.keys()):
            results.append(obj)
        for v in obj.values():
            results.extend(_extract_citations(v))
    elif isinstance(obj, list):
        for item in obj:
            results.extend(_extract_citations(item))

    return results


def verify_citations(
    data: Any, artifact_dir: Path, similarity_threshold: float = 0.70
) -> dict:
    """Verify all citations in a parsed JSON structure.

    Returns a summary dict with per-citation results and aggregate stats.
    """
    citations = _extract_citations(data)

    if not citations:
        return {
            "total": 0,
            "passed": 0,
            "failed": 0,
            "failure_rate": 0.0,
            "results": [],
        }

    results = []
    passed = 0
    failed = 0

    for cit in citations:
        file_path = cit["file_path"]
        line_start = cit["line_start"]
        line_end = cit["line_end"]
        claim = cit["claim"]
        quoted = cit["quoted_content"]

        errors = []

        # Check 1: File exists
        resolved = _resolve_file(file_path, artifact_dir)
        if resolved is None:
            errors.append(f"file_not_found: {file_path}")
        else:
            # Check 2: Line range valid
            try:
                lines = resolved.read_text(encoding="utf-8", errors="replace").splitlines()
            except Exception as e:
                errors.append(f"file_read_error: {e}")
                lines = []

            if lines:
                max_line = len(lines)
                if line_start < 1 or line_end < line_start:
                    errors.append(
                        f"invalid_line_range: {line_start}-{line_end}"
                    )
                elif line_end > max_line:
                    errors.append(
                        f"line_range_exceeds_file: {line_start}-{line_end} "
                        f"but file has {max_line} lines"
                    )
                else:
                    # Check 3: Content similarity
                    actual_lines = lines[line_start - 1 : line_end]
                    actual_content = "\n".join(actual_lines)
                    similarity = _fuzzy_match(quoted, actual_content)

                    if similarity < similarity_threshold:
                        errors.append(
                            f"content_mismatch: similarity={similarity:.2f} "
                            f"(threshold={similarity_threshold:.2f})"
                        )

        result = {
            "file_path": file_path,
            "line_range": f"{line_start}-{line_end}",
            "claim": claim,
            "verified": len(errors) == 0,
        }
        if errors:
            result["errors"] = errors
            failed += 1
        else:
            passed += 1
        results.append(result)

    total = len(results)
    return {
        "total": total,
        "passed": passed,
        "failed": failed,
        "failure_rate": round(failed / total, 3) if total > 0 else 0.0,
        "results": results,
    }


def main() -> int:
    parser = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    parser.add_argument(
        "--input",
        required=True,
        help="Path to JSON file containing agent output with citations",
    )
    parser.add_argument(
        "--artifact-dir",
        required=True,
        help="Path to the campaign artifact directory (citations are relative to this)",
    )
    parser.add_argument(
        "--threshold",
        type=float,
        default=0.70,
        help="Minimum fuzzy-match similarity for quoted_content (default: 0.70)",
    )
    parser.add_argument(
        "--strict",
        action="store_true",
        help="Exit with error code if any citation fails (for pipeline gating)",
    )
    parser.add_argument(
        "--output",
        type=str,
        default=None,
        help="Write verification results JSON to this path (default: stdout summary only)",
    )

    args = parser.parse_args()

    input_path = Path(args.input)
    if not input_path.exists():
        print(f"ERROR: Input file not found: {input_path}", file=sys.stderr)
        return 1

    artifact_dir = Path(args.artifact_dir).resolve()
    if not artifact_dir.exists():
        print(f"ERROR: Artifact directory not found: {artifact_dir}", file=sys.stderr)
        return 1

    data = json.loads(input_path.read_text(encoding="utf-8"))
    report = verify_citations(data, artifact_dir, args.threshold)

    # Print summary
    print(f"\nCitation Verification: {report['passed']}/{report['total']} passed "
          f"({report['failure_rate']:.1%} failure rate)")

    if report["failed"] > 0:
        print(f"\nFailed citations:")
        for r in report["results"]:
            if not r["verified"]:
                print(f"  - {r['file_path']}:{r['line_range']} — {r['claim']}")
                for err in r.get("errors", []):
                    print(f"    {err}")

    # Write output if requested
    if args.output:
        out_path = Path(args.output)
        out_path.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
        print(f"\nDetailed results written to: {out_path}")

    if args.strict and report["failed"] > 0:
        print(f"\n--strict mode: failing due to {report['failed']} broken citation(s)")
        return 1

    return 0


if __name__ == "__main__":
    sys.exit(main())
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall