anyshift-io/sre-skills / skills/kubectl-investigator
kubectl-investigator
UnverifiedInvestigate a live or recent incident in a Kubernetes cluster. Anchor the window, bisect the change surface (rollouts, ConfigMaps/Secrets, RBAC, HPA/cluster changes, CronJobs), classify against four reference failure paths (OOM, DNS, cascading-failure, deploy-correlator), confirm the hypothesis with three independent signals, quantify blast radius, and propose mitigation before root cause. Use whenever an agent is asked "what is breaking in the cluster right now", "why did this pod/Deployment just page", "did the rollout cause Z", or to triage an active Kubernetes incident. Vendor-neutral by default (works with kubectl, kube-state-metrics, and whatever telemetry you have); an opt-in Anyshift integration is documented separately.
tests/eval/run_eval.py
View on GitHub"""
Ablation eval for the kubectl-investigator skill.
For each fixture, runs N trials in each of two conditions:
- Control: the agent is given the telemetry and a generic "investigate this incident" prompt.
- Treatment: the agent is given the same telemetry plus SKILL.md as the methodology to follow.
Each agent output is graded against a 7-item rubric (rubric.md) by an LLM judge.
The judge uses the deterministic reference implementation's outputs (from _methodology.py)
as the source of truth.
Final report: per-fixture mean score (control vs treatment), lift (= treatment - control),
stdev across trials, and a "skill is valuable when lift is consistently positive" verdict.
Requirements:
- ANTHROPIC_API_KEY environment variable.
- `pip install anthropic` (the only non-stdlib dependency in the repo; isolated to tests/eval/).
Usage:
python tests/eval/run_eval.py --trials 5
python tests/eval/run_eval.py --trials 1 --fixtures 01,03,05 # smoke test
Cost note: with 11 fixtures, 2 conditions, 5 trials, plus the judge call per output,
the eval makes ~110 agent calls + ~110 judge calls = ~220 LLM calls. Expect $20-50
depending on model (Sonnet recommended for cost; Opus for highest agent quality).
"""
from __future__ import annotations
import argparse
import json
import os
import statistics
import sys
import time
from pathlib import Path
try:
from anthropic import Anthropic
except ImportError:
print("ERROR: anthropic SDK not installed. Run: pip install anthropic", file=sys.stderr)
sys.exit(1)
sys.path.insert(0, str(Path(__file__).parent))
from incidents import INCIDENTS, load_fixture_text # noqa: E402
REPO_ROOT = Path(__file__).resolve().parent.parent.parent
SKILL_MD = REPO_ROOT / "SKILL.md"
DEFAULT_AGENT_MODEL = os.environ.get("EVAL_AGENT_MODEL", "claude-sonnet-4-6")
DEFAULT_JUDGE_MODEL = os.environ.get("EVAL_JUDGE_MODEL", "claude-sonnet-4-6")
MAX_TOKENS = 4096
def build_control_prompt(incident: dict) -> str:
fixture_text = load_fixture_text(incident["fixture_dir"])
return f"""You are an SRE investigating an incident. Here is what you know:
- Service: {incident['service']}
- Alert fired at: {incident['alert_time']} ({incident['alert_message']})
- Investigation triggered at: {incident['tnow']}
All telemetry for the affected service:
{fixture_text}
Investigate the incident. Produce: a timeline, root-cause hypothesis with supporting evidence, blast-radius estimate, and recommended mitigation. Be specific about what you would do first."""
def build_treatment_prompt(incident: dict, skill_md_text: str) -> str:
return f"""You are an SRE investigating an incident, following the methodology below exactly.
METHODOLOGY (SKILL.md):
{skill_md_text}
INCIDENT CONTEXT:
- Service: {incident['service']}
- Alert fired at: {incident['alert_time']} ({incident['alert_message']})
- Investigation triggered at: {incident['tnow']}
TELEMETRY:
{load_fixture_text(incident['fixture_dir'])}
Apply the methodology end-to-end. Produce the structured output the methodology's "Output format" section prescribes (seven sections)."""
JUDGE_SYSTEM = """You are an expert SRE evaluator grading an incident investigation against a 7-item rubric. Each item is binary: 1 (pass) or 0 (fail). Be strict but fair; no partial credit.
You will be given:
1. A known-good answer generated by a deterministic reference implementation of the methodology.
2. The agent's investigation output.
3. The 7 rubric items.
Return JSON only (no prose), with this exact schema:
{
"scores": [<int>, <int>, <int>, <int>, <int>, <int>, <int>],
"notes": ["<one sentence>", ...]
}"""
def build_judge_prompt(incident: dict, agent_output: str) -> str:
return f"""INCIDENT: {incident['id']}
SERVICE: {incident['service']}
KNOWN-GOOD ANSWER (from deterministic methodology):
- T0: {incident['expected_t0']}
- Change surface: {incident['expected_change_count']} change(s) in window - {incident['expected_change_summary']}
- Classification: {incident['expected_path']}
- Top mitigation: {incident['expected_top_mitigation']}
- Should escalate to human: {incident['expected_escalate']}
AGENT INVESTIGATION OUTPUT:
{agent_output}
RUBRIC:
1. T0 anchor: agent picks T0 within +/-2 minutes of {incident['expected_t0']}.
2. Change surface: agent correctly enumerates {incident['expected_change_count']} change(s) in window ({incident['expected_change_summary']}).
3. Classification: agent classifies the failure as {incident['expected_path']}.
4. Independent signals: agent cites at least 3 signals from distinct source categories (logs, metrics, traces, change events, orchestrator events). Two signals from the same category count as one.
5. Blast radius: agent quantifies users affected + surfaces affected + business impact (all three required for pass).
6. Mitigation: agent's TOP recommended action matches {incident['expected_top_mitigation']}.
7. Handoff: agent produces a structured handoff including timeline, ranked hypothesis(es), recommended mitigation, and open questions (all four required for pass).
Score each item as 1 (pass) or 0 (fail). Return JSON only."""
RETRYABLE_STATUS = {408, 409, 429, 500, 502, 503, 529}
MAX_RETRIES = 6
def _with_retries(fn, *args, **kwargs):
"""Call fn with exponential backoff on transient API errors (429/5xx/529/overloaded).
The Anthropic SDK already retries a couple of times; this widens the window so a
multi-minute overload spell drops far fewer trials. Re-raises on non-retryable
errors or once retries are exhausted.
Returns (result, call_seconds) where call_seconds is the wall-time of the SUCCESSFUL
attempt only - backoff sleeps and failed attempts are excluded, so duration metrics
reflect real investigation latency, not how overloaded the API happened to be.
"""
delay = 2.0
last_exc = None
for attempt in range(MAX_RETRIES):
try:
t_call = time.time()
return fn(*args, **kwargs), time.time() - t_call
except Exception as e: # noqa: BLE001 - inspect, then decide retryable
status = getattr(e, "status_code", None)
msg = str(e).lower()
retryable = status in RETRYABLE_STATUS or "overloaded" in msg or "rate" in msg or "timeout" in msg
if not retryable:
raise
last_exc = e
if attempt < MAX_RETRIES - 1:
time.sleep(delay)
delay = min(delay * 2, 60.0)
raise last_exc
def run_agent(client: Anthropic, model: str, prompt: str) -> tuple[str, float]:
"""Returns (agent_output_text, investigation_seconds). Seconds excludes retry backoff."""
def _call():
return client.messages.create(
model=model,
max_tokens=MAX_TOKENS,
messages=[{"role": "user", "content": prompt}],
)
resp, call_s = _with_retries(_call)
return "".join(block.text for block in resp.content if block.type == "text"), call_s
def run_judge(client: Anthropic, model: str, incident: dict, agent_output: str) -> dict:
def _call():
return client.messages.create(
model=model,
max_tokens=1024,
system=JUDGE_SYSTEM,
messages=[{"role": "user", "content": build_judge_prompt(incident, agent_output)}],
)
resp, _ = _with_retries(_call)
raw = "".join(block.text for block in resp.content if block.type == "text")
# Strip code fences if the judge wrapped JSON in markdown.
raw = raw.strip()
if raw.startswith("```"):
raw = raw.split("```", 2)[1]
if raw.startswith("json"):
raw = raw[4:]
raw = raw.rsplit("```", 1)[0]
return json.loads(raw.strip())
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--trials", type=int, default=5, help="Trials per (fixture, condition) cell")
parser.add_argument("--fixtures", default="", help="Comma-separated fixture IDs (prefix match); empty = all")
parser.add_argument("--agent-model", default=DEFAULT_AGENT_MODEL)
parser.add_argument("--judge-model", default=DEFAULT_JUDGE_MODEL)
parser.add_argument("--output", default="eval_results.json", help="Where to write the raw results")
parser.add_argument("--fresh", action="store_true", help="Ignore an existing results file and start clean (default: resume/fill gaps)")
args = parser.parse_args()
if "ANTHROPIC_API_KEY" not in os.environ:
print("ERROR: ANTHROPIC_API_KEY not set", file=sys.stderr)
return 1
client = Anthropic()
skill_md_text = SKILL_MD.read_text()
fixtures_to_run = INCIDENTS
if args.fixtures:
filters = [f.strip() for f in args.fixtures.split(",")]
fixtures_to_run = [i for i in INCIDENTS if any(i["id"].startswith(f) for f in filters)]
print(f"Running {len(fixtures_to_run)} fixtures x 2 conditions x {args.trials} trials = {len(fixtures_to_run) * 2 * args.trials} agent calls")
print(f"Agent model: {args.agent_model}, Judge model: {args.judge_model}\n")
# Resume: reload any completed trials from a prior run so a re-run fills ONLY the
# gaps (e.g. trials dropped to a transient overload), never redoing finished work.
# Pass --fresh to ignore an existing results file and start clean.
results: list[dict] = []
completed: set[tuple[str, str, int]] = set()
out_path = Path(args.output)
if out_path.exists() and not args.fresh:
try:
results = json.loads(out_path.read_text())
completed = {(r["incident"], r["condition"], r["trial"]) for r in results}
print(f"Resuming from {args.output}: {len(completed)} trials already complete; filling gaps only.\n")
except (json.JSONDecodeError, KeyError, OSError):
results, completed = [], set()
for incident in fixtures_to_run:
for condition in ("control", "treatment"):
for trial in range(args.trials):
if (incident["id"], condition, trial) in completed:
continue # already have this cell from a prior run
t_start = time.time()
prompt = build_control_prompt(incident) if condition == "control" else build_treatment_prompt(incident, skill_md_text)
try:
agent_output, agent_s = run_agent(client, args.agent_model, prompt) # agent_s excludes retry backoff
judge_result = run_judge(client, args.judge_model, incident, agent_output)
score = sum(judge_result["scores"])
except Exception as e:
print(f" ERROR on {incident['id']} {condition} trial {trial}: {e}", file=sys.stderr)
continue
elapsed = time.time() - t_start # agent + judge, for cost/wall-clock accounting
results.append({
"incident": incident["id"],
"condition": condition,
"trial": trial,
"score": score,
"scores_by_item": judge_result["scores"],
"notes": judge_result.get("notes", []),
"agent_output": agent_output,
"agent_chars": len(agent_output),
"agent_s": round(agent_s, 1),
"elapsed_s": round(elapsed, 1),
})
# Crash-safe: persist after every trial via atomic temp+rename so an
# overload-induced death never throws away completed work.
tmp = Path(str(args.output) + ".tmp")
tmp.write_text(json.dumps(results, indent=2))
tmp.replace(args.output)
print(f" {incident['id']} | {condition:9s} | trial {trial} | score {score}/7 | investigate {agent_s:.0f}s", flush=True)
Path(args.output).write_text(json.dumps(results, indent=2))
print(f"\nRaw results: {args.output}\n")
print_summary(results)
return 0
def print_summary(results: list[dict]) -> None:
by_cell: dict[tuple[str, str], list[int]] = {}
dur_cell: dict[tuple[str, str], list[float]] = {}
for r in results:
by_cell.setdefault((r["incident"], r["condition"]), []).append(r["score"])
dur_cell.setdefault((r["incident"], r["condition"]), []).append(r.get("agent_s", r.get("elapsed_s", 0.0)))
# --- Quality table ---
print(f"{'Fixture':<48} {'Control mean':>14} {'Treatment mean':>16} {'Lift':>8} {'C-std':>7} {'T-std':>7}")
print("-" * 100)
lifts = []
for incident in INCIDENTS:
control_scores = by_cell.get((incident["id"], "control"), [])
treatment_scores = by_cell.get((incident["id"], "treatment"), [])
if not control_scores or not treatment_scores:
continue
c_mean = statistics.mean(control_scores)
t_mean = statistics.mean(treatment_scores)
lift = t_mean - c_mean
lifts.append(lift)
c_std = statistics.stdev(control_scores) if len(control_scores) > 1 else 0.0
t_std = statistics.stdev(treatment_scores) if len(treatment_scores) > 1 else 0.0
print(f"{incident['id']:<48} {c_mean:>14.2f} {t_mean:>16.2f} {lift:>+8.2f} {c_std:>7.2f} {t_std:>7.2f}")
print("-" * 100)
if lifts:
aggregate = statistics.mean(lifts)
positive = sum(1 for l in lifts if l > 0)
zero = sum(1 for l in lifts if l == 0)
negative = sum(1 for l in lifts if l < 0)
print(f"\nAggregate lift: {aggregate:+.2f}/7 across {len(lifts)} fixtures")
print(f" Positive lift: {positive}, Zero: {zero}, Negative: {negative}")
verdict = (
"Skill is clearly valuable" if aggregate >= 1.0 and positive >= 2 * negative
else "Skill provides marginal lift" if aggregate >= 0.3
else "Skill is not clearly adding value; investigate why before shipping"
)
print(f" Verdict: {verdict}")
# --- Duration table (investigation = agent call only, judge excluded) ---
print(f"\n{'Fixture':<48} {'Control s':>12} {'Treatment s':>14} {'Delta s':>9} {'Ratio':>7}")
print("-" * 100)
c_durs_all, t_durs_all = [], []
for incident in INCIDENTS:
c_durs = dur_cell.get((incident["id"], "control"), [])
t_durs = dur_cell.get((incident["id"], "treatment"), [])
if not c_durs or not t_durs:
continue
c_d = statistics.mean(c_durs)
t_d = statistics.mean(t_durs)
c_durs_all.extend(c_durs)
t_durs_all.extend(t_durs)
ratio = (t_d / c_d) if c_d else 0.0
print(f"{incident['id']:<48} {c_d:>12.1f} {t_d:>14.1f} {t_d - c_d:>+9.1f} {ratio:>6.2f}x")
print("-" * 100)
if c_durs_all and t_durs_all:
c_mean_d = statistics.mean(c_durs_all)
t_mean_d = statistics.mean(t_durs_all)
ratio = (t_mean_d / c_mean_d) if c_mean_d else 0.0
print(f"\nMean investigation duration: control {c_mean_d:.1f}s vs treatment {t_mean_d:.1f}s "
f"({t_mean_d - c_mean_d:+.1f}s, {ratio:.2f}x)")
print(" Note: agent call only (judge excluded). Treatment loads the full SKILL.md and emits")
print(" seven structured sections, so it is expected to run longer; that is the latency cost of the lift.")
if __name__ == "__main__":
sys.exit(main())
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |