Skip to content

amazon-science/ammo / ai_cli_session/.codex/skills/ammo

Profile and optimize GPU kernels for vLLM inference on NVIDIA GPUs. Use when targeting a specific model, hardware, dtype, parallelism, and workload to improve latency.

FileKindSize
RUNTIME.mdmarkdown4.9 KB
SKILL.mdskill md18.5 KB
agents/ammo-auditor.mdmarkdown9.1 KB
agents/ammo-champion.mdmarkdown9.6 KB
agents/ammo-delegate.mdmarkdown3.2 KB
agents/ammo-implementer.mdmarkdown13.8 KB
agents/ammo-investigator.mdmarkdown4.0 KB
agents/ammo-report.mdmarkdown2.9 KB
agents/ammo-researcher.mdmarkdown10.3 KB
agents/ammo-resolver.mdmarkdown4.4 KB
agents/ammo-transcript-monitor.mdmarkdown7.8 KB
agents/openai.yamlexecutablescript313 B
data/ATTRIBUTION.mdmarkdown1.4 KB
data/gsm8k_full.jsonexecutablescript735.5 KB
data/gsm8k_subset.jsonexecutablescript116.8 KB
eval/EVAL-SKILL.mdexecutablescript21.8 KB
eval/agents/transcript_grader.mdexecutablescript18.0 KB
eval/causal/__init__.pyexecutablescript137 B
eval/causal/agents/causal_analyzer.mdexecutablescript10.5 KB
eval/causal/build_dag.pyexecutablescript21.6 KB
eval/causal/diff_versions.pyexecutablescript38.5 KB
eval/causal/extract_events.pyexecutablescript36.3 KB
eval/causal/generate_postmortem.pyexecutablescript17.0 KB
eval/causal/score_nodes.pyexecutablescript21.5 KB
eval/causal/templates/narrative_template.mdexecutablescript472 B
eval/causal/templates/viz_template.htmlexecutablescript9.5 KB
eval/causal/tests/__init__.pyexecutablescript151 B
eval/causal/tests/conftest.pyexecutablescript448 B
eval/causal/tests/fixtures/sample_session.jsonlexecutablescript10.4 KB
eval/causal/tests/fixtures/sample_session_data.jsonexecutablescript3.4 KB
eval/causal/tests/fixtures/sample_snapshot.jsonexecutablescript2.5 KB
eval/causal/tests/test_build_dag.pyexecutablescript2.7 KB
eval/causal/tests/test_diff_versions.pyexecutablescript13.5 KB
eval/causal/tests/test_extract_events.pyexecutablescript3.7 KB
eval/causal/tests/test_generate_postmortem.pyexecutablescript3.1 KB
eval/causal/tests/test_score_nodes.pyexecutablescript3.3 KB
eval/scripts/aggregate_versions.pyexecutablescript15.2 KB
eval/scripts/archive_run.pyexecutablescript9.1 KB
eval/scripts/codex_session.pyexecutablescript10.6 KB
eval/scripts/generate_dashboard.pyexecutablescript5.3 KB
eval/scripts/parse_artifacts.pyexecutablescript48.1 KB
eval/scripts/parse_session_logs.pyexecutablescript51.3 KB
eval/scripts/rescore_archived.pyexecutablescript5.5 KB
eval/scripts/score_campaign.pyexecutablescript43.8 KB
eval/scripts/snapshot_changes.pyexecutablescript11.1 KB
eval/scripts/verify_citations.pyexecutablescript8.8 KB
eval/tests/test_dashboard_bugs.pyexecutablescript8.8 KB
eval/tests/test_parse_session_logs.pyexecutablescript3.9 KB
eval/tests/test_parse_session_logs_round_inference.pyexecutablescript11.0 KB
eval/tests/test_score_campaign_diluted.pyexecutablescript9.4 KB
eval/viewer/dashboard_template.htmlexecutablescript15.1 KB
instrumentation/sitecustomize.pyexecutablescript3.6 KB
orchestration/audit-protocol.mdmarkdown9.3 KB
orchestration/debate-protocol.mdmarkdown11.2 KB
orchestration/integration-logic.mdmarkdown10.3 KB
orchestration/parallel-tracks.mdmarkdown8.9 KB
references/artifact-layout.mdexecutablescript22.4 KB
references/audit-invariants.mdmarkdown10.6 KB
references/champion-common-patterns.mdmarkdown3.4 KB
references/crossover-probing.mdexecutablescript4.7 KB
references/cudagraph-safety.mdmarkdown2.7 KB
references/debate-rules.mdmarkdown4.7 KB
references/debate-scoring-rubric.mdexecutablescript16.7 KB
references/e2e-delta-math.mdexecutablescript17.9 KB
references/e2e-latency-guide.mdexecutablescript9.6 KB
references/fusion-feasibility-heuristics.mdmarkdown5.6 KB
references/gpu-pool.mdexecutablescript5.4 KB
references/impl-track-rules.mdexecutablescript13.8 KB
references/nsys-profiling-guide.mdmarkdown9.8 KB
references/optimization-categories.mdexecutablescript11.0 KB
references/shutdown-protocol.mdexecutablescript3.1 KB
references/technology-selection.mdmarkdown5.0 KB
references/torch-compile-contract.mdmarkdown13.8 KB
references/validation-defaults.mdexecutablescript34.2 KB
references/workload-invocation.mdexecutablescript4.9 KB
references/writing-style.mdexecutablescript6.9 KB
scripts/ammo_state.pyexecutablescript142.0 KB
scripts/artifact_layout.jsonconfig16.3 KB
scripts/check_projection_accuracy.pyexecutablescript11.7 KB
scripts/create_evidence_template.pyexecutablescript9.7 KB
scripts/create_worktree_with_build.shexecutablescript10.2 KB
scripts/generate_validation_report.pyexecutablescript35.7 KB
scripts/gpu_reservation.pyexecutablescript27.1 KB
scripts/hook_cmd_classify.pyscript13.3 KB
scripts/mine_trace.pyscript29.4 KB
scripts/monitor_queue_ack.pyexecutablescript6.1 KB
scripts/ncu_sanity.shexecutablescript5.2 KB
scripts/ncu_sanity_driver.pyexecutablescript8.5 KB
scripts/new_target.pyexecutablescript22.0 KB
scripts/preflight_vllm_api.pyexecutablescript12.3 KB
scripts/reconcile_track_state.pyexecutablescript15.8 KB
scripts/remove_worktree_cleanup.shexecutablescript1.1 KB
scripts/resolve_codex_transcript.pyexecutablescript3.6 KB
scripts/run_vllm_bench_latency_sweep.pyexecutablescript249.6 KB
scripts/transcript_filter.pyexecutablescript38.4 KB
scripts/transitions.jsonexecutablescript1.5 KB
scripts/verify_validation_gates.pyexecutablescript88.8 KB
tests/agents/test-champion.mdexecutablescript20.4 KB
tests/agents/test-impl-champion.mdexecutablescript14.3 KB
tests/agents/test-implementer.mdexecutablescript15.8 KB
tests/agents/test-orchestrator.mdexecutablescript33.8 KB
tests/agents/test-researcher.mdexecutablescript21.0 KB
tests/agents/test-transcript-monitor.mdexecutablescript13.4 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.mdmarkdown1.1 KB
tests/fixtures/diluted_pass/legacy_no_phase.golden.summary.jsonconfig582 B
tests/fixtures/diluted_pass/legacy_no_phase.jsonconfig400 B
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.mdmarkdown2.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.golden.summary.jsonconfig1.0 KB
tests/fixtures/diluted_pass/legacy_scalar_phase.jsonconfig626 B
tests/fixtures/mining/real_campaign_bottleneck_analysis.mdmarkdown20.7 KB
tests/run_all.shexecutablescript1.6 KB
tests/scripts/test_transcript_filter.pyexecutablescript41.1 KB
tests/test_agent_runtime_contracts.pyexecutablescript1.8 KB
tests/test_ammo_state.pyexecutablescript71.7 KB
tests/test_artifact_layout_manifest.pyscript11.6 KB
tests/test_audit_started.pyscript18.2 KB
tests/test_correctness_comparator.pyscript12.6 KB
tests/test_diluted_pass.pyscript24.4 KB
tests/test_drift_compat.pyscript5.1 KB
tests/test_evidence_freshness.pyexecutablescript5.6 KB
tests/test_fail_closed.pyexecutablescript14.7 KB
tests/test_gpu_reservation.pyexecutablescript24.8 KB
tests/test_gpu_reservation_session_id.pyexecutablescript11.2 KB
tests/test_guidance_semantic_contract.pyscript24.9 KB
tests/test_hook_cmd_classify_parity.pyscript11.7 KB
tests/test_hook_semantics.pyexecutablescript95.3 KB
tests/test_mine_trace.pyscript24.2 KB
tests/test_mining_invalidated_schema.pyexecutablescript9.4 KB
tests/test_new_target_dp_ep.pyexecutablescript7.4 KB
tests/test_new_target_v2_scaffold.pyexecutablescript13.1 KB
tests/test_parse_artifacts.pyexecutablescript3.0 KB
tests/test_policy_convergence.pyscript797 B
tests/test_preflight_vllm_api.pyscript13.3 KB
tests/test_projection_accuracy_v2.pyexecutablescript4.1 KB
tests/test_reconcile_track_state.pyexecutablescript18.2 KB
tests/test_resolve_codex_transcript.pyexecutablescript3.1 KB
tests/test_state_schema_semantics.pyexecutablescript7.6 KB
tests/test_status_set_sync.pyexecutablescript9.1 KB
tests/test_sweep_baseline_from_arm.pyscript11.0 KB
tests/test_sweep_decode_capture.pyexecutablescript31.8 KB
tests/test_sweep_dp.pyexecutablescript45.4 KB
tests/test_sweep_fresh_cache.pyscript5.5 KB
tests/test_sweep_profiler_guard.pyscript22.6 KB
tests/test_sweep_real_prompts.pyscript20.1 KB
tests/test_sweep_refactor.pyexecutablescript37.6 KB
tests/test_sweep_round_slot.pyexecutablescript7.4 KB
tests/test_validation_report_phase_triage.pyexecutablescript10.0 KB
tests/test_validation_report_v2.pyexecutablescript6.2 KB
tests/test_verify_validation_gates.pyexecutablescript52.9 KB
tests/test_workload_matrix.pyexecutablescript19.3 KB

eval/causal/agents/causal_analyzer.md

View on GitHub
# Causal Post-Mortem Analyzer — LLM Rubric

## Role Description

You are a causal post-mortem analyzer for AMMO GPU kernel optimization campaigns. Your job is to analyze flagged anomalies in the campaign's causal DAG, identify root causes, trace causal chains, and recommend actionable fixes.

---

## Input Format

For each anomaly you receive, the following fields are provided:

| Field | Type | Description |
|---|---|---|
| `node_id` | string | Which DAG node this anomaly is in |
| `anomaly_type` | string | One of: `DATA_CORRUPTION`, `QUALITY_DROP`, `WASTED_EFFORT`, `THRASHING`, `LUCKY_WIN` |
| `triggered_by` | dict | Which metric(s) tripped the anomaly threshold |
| `upstream_data` | any | The node's input artifacts (file contents or excerpts) |
| `node_output` | any | The node's output artifacts |
| `downstream_output` | any | The immediate downstream consumer's output |
| `transcript_window` | string | ~2K tokens of agent conversation around the anomaly events |
| `events_window` | list | Structured events from the anomaly node (from events.json) |

---

## Required Output Fields

For every anomaly, produce a JSON object with **ALL 9 fields below**. Validation fails if any field is missing.

| Field | Type | Description |
|---|---|---|
| `node_id` | string | Echo back the node ID |
| `anomaly_type` | string | Echo back the classified type |
| `root_cause` | string | 1-2 sentence explanation of what went wrong (or right, for LUCKY_WIN) |
| `causal_chain` | list of strings | Ordered sequence of cause → effect steps |
| `counterfactual` | string | What would likely have happened if the root cause were different |
| `skill_attribution` | enum string | One of: `"skill_guidance"` \| `"agent_behavior"` \| `"input_data"` \| `"stochastic"` |
| `actionable_fix` | string | Specific change to skill text, agent prompt, or process |
| `confidence` | float 0-1 | How confident you are in this analysis, based on evidence quality |
| `evidence` | list of objects | Structured citations backing every factual claim (see below) |

### `evidence` field (REQUIRED)

Every factual claim in `root_cause`, `causal_chain`, and `counterfactual` must be backed by at least one citation. Each citation is an object with:

| Field | Required | Description |
|---|---|---|
| `file_path` | yes | Path relative to artifact dir, or absolute for JSONL/temp files |
| `line_start` | yes | First line number, 1-indexed |
| `line_end` | yes | Last line number, inclusive |
| `claim` | yes | The specific factual statement this citation supports |
| `quoted_content` | yes | Verbatim excerpt from the cited lines, truncated to about 200 characters if long |

Open the artifact file, find the exact supporting lines, and include the verbatim text. The citation validator checks that cited files exist and that `quoted_content` matches the cited lines.

### `skill_attribution` values

- **`skill_guidance`**: The skill text/prompt caused this behavior.
- **`agent_behavior`**: The agent deviated from what the skill instructed.
- **`input_data`**: Profiling data was ambiguous or wrong.
- **`stochastic`**: Random variance, not attributable to any single cause.

If `skill_attribution` is `"skill_guidance"`, the `actionable_fix` **must** reference the specific section or line of the skill that should change.

---

## Per-Anomaly-Type Reasoning Instructions

### DATA_CORRUPTION

Cross-reference every numeric claim in the node's output against `upstream_data`. Identify the exact value that diverges. Trace whether downstream nodes consumed the corrupt value.

Look for:
- f-values not present in `bottleneck_analysis.md`
- Timings that do not match nsys data
- Speedup claims without benchmark backing
- Values that appear plausible but shift between the node output and the source artifact

### QUALITY_DROP

Compare the quality of the upstream node's output with this node's output. Identify what information was available but not used, or what reasoning step went wrong.

Ask:
- Was there a framing effect (the prompt anchored on the wrong metric)?
- Did the agent ignore available data that contradicted its conclusion?
- Did the skill prompt guide the agent away from the right answer?
- Was a valid candidate discarded due to a reasoning shortcut?

### WASTED_EFFORT

Identify why downstream nodes did not consume this node's output.

Diagnose one of:
- **Inaccessible**: wrong file path, output written to unexpected location
- **Irrelevant**: addressed a non-bottleneck layer or an already-solved problem
- **Contradicted**: a higher-authority source superseded this node's findings
- **Duplicated**: another agent completed equivalent work in parallel

Estimate the token cost of the wasted work based on `transcript_window` length and node depth.

### THRASHING

Identify the retry loop pattern. Count distinct retry attempts visible in `events_window` and `transcript_window`.

Root cause must be one of:
- **Ambiguous skill instruction**: the skill allowed multiple valid interpretations, causing the agent to oscillate
- **Tooling failure**: wrong build flags, environment mismatch, or missing dependencies caused repeated failures
- **Agent reasoning error**: agent ignored a consistent error signal and retried without changing approach

Estimate wasted cycles (number of retries beyond the first) and approximate token cost.

### LUCKY_WIN

Identify three things:
1. What the agent predicted would happen
2. What actually happened
3. Why the outcome was positive despite any wrong reasoning

Then assess reproducibility: would this optimization succeed again on a fresh run? Is the success coincidental (e.g., the wrong kernel also happens to benefit from the optimization for an unrelated reason)?

Set `confidence` lower when the gap between prediction and outcome is large, because the causal mechanism is uncertain.

---

## Cross-Session Causal Patterns

In addition to the per-anomaly-type analysis above, proactively check for these patterns identified from cross-session analysis of 8+ campaigns. These are the highest-impact failure modes — if any are present, they likely dominate the campaign outcome.

### DOMINANT_COMPONENT_MISSED

Check whether the highest f_decode component (from `bottleneck_analysis.md`) had any optimization track. If the component contributing the most decode time was never targeted by any champion:
- Trace WHY: was it due to framing bias in the bottleneck analysis? A single negative experiment? All champions independently choosing a lower-f component?
- This pattern was the #1 predictor of poor E2E outcomes across 8 sessions. Sessions where all champions avoided the dominant component averaged +0.96% E2E vs +5.23% when at least one champion targeted it.

### FRAMING_BIAS

Check whether `bottleneck_analysis.md` framed a component with <85% BW or compute utilization as "near-optimal", "well-optimized", or "no red flags." Below 85% utilization, there is meaningful headroom — the top-performing session extracted +6.14% E2E from a component at 73% BW utilization that was initially described as having "no red flags."
- If this framing appears, trace whether champions cited it as justification for avoiding the component.

### SINGLE_NEGATIVE_DISMISSAL

Check whether any component contributing >30% of f_decode was dismissed based on a single micro-experiment. A valid dismissal requires two independent negative results with different approaches. Look for:
- One champion tests Triton kernel at a single batch size, shows cuBLAS wins → entire debate accepts this as definitive
- No one challenges the methodology or tests alternative approaches

### COLD_PRODUCTION_GAP

Check whether micro-benchmark speedups overpredicted E2E improvement by >2x. The empirical translation factor from isolated cold-cache kernel benchmarks to production E2E is 0.33-0.5x across all observed sessions. If a campaign's proposals predicted large E2E gains but the sweep showed much smaller improvements:
- Flag the gap and estimate the translation factor
- This is an expected physical effect (CUDA graphs, memory subsystem warming, kernel overlap), not necessarily an error — but proposals that don't account for it indicate overconfident projections

When any of these patterns are detected, include them in your analysis even if they weren't flagged as anomalies by `score_nodes.py`. Create an entry with `anomaly_type: "QUALITY_DROP"` and reference the specific pattern name in the `root_cause` field.

---

## Output Schema Example

```json
[
  {
    "node_id": "debate",
    "anomaly_type": "DATA_CORRUPTION",
    "root_cause": "Champion-3 cited f=22% for MoE dispatch but bottleneck_analysis.md shows 14.8%",
    "causal_chain": [
      "Champion-3 hallucinated f-value (22% vs actual 14.8%)",
      "Inflated feasibility score in debate selection (7.2 vs corrected ~5.0)",
      "op003 selected over more promising op004",
      "op003 failed correctness gate → implementation round wasted"
    ],
    "counterfactual": "Without hallucinated f-value, op004 (targeting 6.2% bottleneck with sound roofline) likely selected and had higher implementation success probability",
    "skill_attribution": "agent_behavior",
    "actionable_fix": "Add to champion prompt: 'For every f-value cited, include the exact line number from bottleneck_analysis.md. The orchestrator will cross-check these citations.'",
    "confidence": 0.85,
    "evidence": [
      {
        "file_path": "debate/campaign_round_1/proposals/champion-3.md",
        "line_start": 42,
        "line_end": 42,
        "claim": "Champion-3 cited f=22% for MoE dispatch",
        "quoted_content": "MoE dispatch accounts for ~22% of decode latency (f=0.22)"
      },
      {
        "file_path": "investigation/bottleneck_analysis.md",
        "line_start": 87,
        "line_end": 87,
        "claim": "Actual f-value for MoE dispatch is 14.8%",
        "quoted_content": "| MoE dispatch | 14.8% | 312 us |"
      }
    ]
  }
]
```

---

## Output Instructions

Write your output as a JSON array of per-anomaly objects to the path specified by the orchestrator (typically `/tmp/ammo_eval_deep_analysis.json`).

Rules:
- Every object must include **ALL 9 required fields**.
- If you cannot determine a field with high confidence, still include it with your best assessment and set `confidence` accordingly — **do not omit fields**.
- Process all anomalies in the input list; do not silently skip any.
- The `causal_chain` list must contain at least 2 entries and at most 8. Each entry is a single cause-or-effect step, written as a short declarative clause.
- `confidence` values above 0.9 require at least two independent corroborating signals from `upstream_data`, `transcript_window`, or `events_window`. If you have only one signal, cap `confidence` at 0.85.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall