Skip to content

anyshift-io/sre-skills / skills/kubectl-investigator

kubectl-investigator

Unverified

Investigate a live or recent incident in a Kubernetes cluster. Anchor the window, bisect the change surface (rollouts, ConfigMaps/Secrets, RBAC, HPA/cluster changes, CronJobs), classify against four reference failure paths (OOM, DNS, cascading-failure, deploy-correlator), confirm the hypothesis with three independent signals, quantify blast radius, and propose mitigation before root cause. Use whenever an agent is asked "what is breaking in the cluster right now", "why did this pod/Deployment just page", "did the rollout cause Z", or to triage an active Kubernetes incident. Vendor-neutral by default (works with kubectl, kube-state-metrics, and whatever telemetry you have); an opt-in Anyshift integration is documented separately.

FileKindSize
FAILURE_MODES.mdmarkdown6.8 KB
README.mdmarkdown4.0 KB
SKILL.mdskill md19.4 KB
examples/01-oom-cascade.mdmarkdown7.4 KB
examples/02-dns-resolution-failure.mdmarkdown7.1 KB
examples/03-cascading-failure-retry-storm.mdmarkdown8.2 KB
examples/04-deploy-correlator-serialization.mdmarkdown7.9 KB
examples/05-outside-reference-paths-third-party-rate-limit.mdmarkdown7.8 KB
examples/06-ambiguous-t0-slow-burn.mdmarkdown7.9 KB
examples/07-blast-radius-asymmetric-revert.mdmarkdown8.1 KB
examples/08-deploy-correlator-confirmation-bias.mdmarkdown9.1 KB
examples/09-zero-changes-external-cert-expiry.mdmarkdown6.3 KB
examples/10-multi-region-asymmetry.mdmarkdown7.1 KB
examples/11-capacity-bound-organic-growth.mdmarkdown6.6 KB
fixtures/01-oom-cascade/deploys.jsonconfig789 B
fixtures/01-oom-cascade/logs.jsonlother1.3 KB
fixtures/01-oom-cascade/metrics.jsonconfig1.3 KB
fixtures/01-oom-cascade/pod_events.jsonlother1.4 KB
fixtures/01-oom-cascade/traces.jsonlother1.5 KB
fixtures/02-dns-resolution-failure/deploys.jsonconfig706 B
fixtures/02-dns-resolution-failure/logs.jsonlother2.4 KB
fixtures/02-dns-resolution-failure/metrics.jsonconfig1.3 KB
fixtures/02-dns-resolution-failure/traces.jsonlother1.4 KB
fixtures/03-cascading-failure-retry-storm/deploys.jsonconfig416 B
fixtures/03-cascading-failure-retry-storm/logs.jsonlother1.4 KB
fixtures/03-cascading-failure-retry-storm/metrics.jsonconfig1.7 KB
fixtures/03-cascading-failure-retry-storm/traces.jsonlother1.6 KB
fixtures/04-deploy-correlator-serialization/deploys.jsonconfig803 B
fixtures/04-deploy-correlator-serialization/logs.jsonlother1.4 KB
fixtures/04-deploy-correlator-serialization/metrics.jsonconfig1.0 KB
fixtures/04-deploy-correlator-serialization/traces.jsonlother1.3 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/deploys.jsonconfig390 B
fixtures/05-outside-reference-paths-third-party-rate-limit/logs.jsonlother1.6 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/metrics.jsonconfig1.3 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/traces.jsonlother1.3 KB
fixtures/06-ambiguous-t0-slow-burn/deploys.jsonconfig517 B
fixtures/06-ambiguous-t0-slow-burn/logs.jsonlother862 B
fixtures/06-ambiguous-t0-slow-burn/metrics.jsonconfig1.2 KB
fixtures/06-ambiguous-t0-slow-burn/traces.jsonlother933 B
fixtures/07-blast-radius-asymmetric-revert/deploys.jsonconfig905 B
fixtures/07-blast-radius-asymmetric-revert/logs.jsonlother1.7 KB
fixtures/07-blast-radius-asymmetric-revert/metrics.jsonconfig1.4 KB
fixtures/07-blast-radius-asymmetric-revert/traces.jsonlother1.5 KB
fixtures/08-deploy-correlator-confirmation-bias/deploys.jsonconfig812 B
fixtures/08-deploy-correlator-confirmation-bias/logs.jsonlother2.0 KB
fixtures/08-deploy-correlator-confirmation-bias/metrics.jsonconfig1.3 KB
fixtures/08-deploy-correlator-confirmation-bias/traces.jsonlother1.3 KB
fixtures/09-zero-changes-external-cert-expiry/deploys.jsonconfig145 B
fixtures/09-zero-changes-external-cert-expiry/logs.jsonlother1.2 KB
fixtures/09-zero-changes-external-cert-expiry/metrics.jsonconfig1.3 KB
fixtures/09-zero-changes-external-cert-expiry/traces.jsonlother1.4 KB
fixtures/10-multi-region-asymmetry/deploys.jsonconfig138 B
fixtures/10-multi-region-asymmetry/logs.jsonlother1.7 KB
fixtures/10-multi-region-asymmetry/metrics.jsonconfig2.0 KB
fixtures/10-multi-region-asymmetry/traces.jsonlother1.6 KB
fixtures/11-capacity-bound-organic-growth/deploys.jsonconfig139 B
fixtures/11-capacity-bound-organic-growth/logs.jsonlother948 B
fixtures/11-capacity-bound-organic-growth/metrics.jsonconfig1.2 KB
fixtures/11-capacity-bound-organic-growth/traces.jsonlother1.2 KB
tests/README.mdmarkdown4.3 KB
tests/_methodology.pyscript22.6 KB
tests/eval/README.mdmarkdown12.7 KB
tests/eval/eval_results.jsonconfig852.7 KB
tests/eval/incidents.pyscript8.6 KB
tests/eval/judge_prompt.mdmarkdown2.4 KB
tests/eval/rubric.mdmarkdown3.1 KB
tests/eval/run_eval.pyscript15.0 KB
tests/eval/run_until_pass.shscript2.1 KB
tests/replay_01_oom_cascade.pyscript3.6 KB
tests/replay_02_dns.pyscript3.3 KB
tests/replay_03_cascade.pyscript3.2 KB
tests/replay_04_deploy_correlator.pyscript3.3 KB
tests/replay_05_outside_paths.pyscript2.4 KB
tests/replay_06_ambiguous_t0.pyscript2.3 KB
tests/replay_07_blast_radius.pyscript2.4 KB
tests/replay_08_confirmation_bias.pyscript2.7 KB
tests/replay_09_cert_expiry.pyscript2.0 KB
tests/replay_10_multi_region.pyscript2.4 KB
tests/replay_11_capacity_bound.pyscript2.3 KB

tests/eval/incidents.py

View on GitHub
"""
Per-fixture incident contexts and expected answers, used by run_eval.py.

The "expected_*" fields are the deterministic answers from _methodology.py
run against each fixture (see tests/replay_*.py). They are the source of
truth the judge model compares the agent's output against.

Stdlib only. No external dependencies.
"""

from __future__ import annotations

from pathlib import Path

FIXTURES_DIR = Path(__file__).resolve().parent.parent.parent / "fixtures"

# Each entry: a fixture-specific incident the eval will run agents against.
# Keep this list aligned with the replay_*.py files under tests/.

INCIDENTS = [
    {
        "id": "01-oom-cascade",
        "fixture_dir": FIXTURES_DIR / "01-oom-cascade",
        "service": "payments-api",
        "alert_time": "2026-03-12T14:32:00Z",
        "alert_message": "payments_api_error_rate > 5%",
        "tnow": "2026-03-12T14:36:00Z",
        "failing_surface_hints": ["webhook", "buffer"],
        "expected_t0": "2026-03-12T14:32:00Z",
        "expected_change_count": 1,
        "expected_change_summary": "deploy payments-api v4.18.0 (commit 9f3a2c1) in window",
        "expected_path": "OOM",
        "expected_top_mitigation": "revert payments-api to v4.17.4",
        "expected_escalate": False,
    },
    {
        "id": "02-dns-resolution-failure",
        "fixture_dir": FIXTURES_DIR / "02-dns-resolution-failure",
        "service": "inventory-svc",
        "alert_time": "2026-04-08T09:47:00Z",
        "alert_message": "inventory_svc_error_rate > 3%",
        "tnow": "2026-04-08T09:53:00Z",
        "failing_surface_hints": ["coredns", ".internal"],
        "expected_t0": "2026-04-08T09:47:00Z",
        "expected_change_count": 1,
        "expected_change_summary": "kube-system/coredns ConfigMap edit (typo in .internal forward)",
        "expected_path": "DNS",
        "expected_top_mitigation": "revert the kube-system/coredns ConfigMap",
        "expected_escalate": False,
    },
    {
        "id": "03-cascading-failure-retry-storm",
        "fixture_dir": FIXTURES_DIR / "03-cascading-failure-retry-storm",
        "service": "payments-api",
        "alert_time": "2026-03-20T11:08:00Z",
        "alert_message": "payments_api_latency_p99 > 1000ms",
        "tnow": "2026-03-20T11:15:00Z",
        "failing_surface_hints": [],
        "expected_t0": "2026-03-20T11:08:00Z",
        "expected_change_count": 0,
        "expected_change_summary": "zero changes in window",
        "expected_path": "cascading-failure",
        "expected_top_mitigation": "open circuit breaker on the upstream ledger-svc call path",
        "expected_escalate": False,
    },
    {
        "id": "04-deploy-correlator-serialization",
        "fixture_dir": FIXTURES_DIR / "04-deploy-correlator-serialization",
        "service": "checkout-api",
        "alert_time": "2026-02-15T13:25:00Z",
        "alert_message": "checkout_api_error_rate > 2%",
        "tnow": "2026-02-15T13:30:00Z",
        "failing_surface_hints": ["cart", "serializer"],
        "expected_t0": "2026-02-15T13:25:00Z",
        "expected_change_count": 1,
        "expected_change_summary": "deploy checkout-api v6.4.0 (commit d1c4e22)",
        "expected_path": "deploy-correlator",
        "expected_top_mitigation": "revert checkout-api to v6.3.7",
        "expected_escalate": False,
    },
    {
        "id": "05-outside-reference-paths-third-party-rate-limit",
        "fixture_dir": FIXTURES_DIR / "05-outside-reference-paths-third-party-rate-limit",
        "service": "payments-api",
        "alert_time": "2026-05-04T16:44:00Z",
        "alert_message": "payments_api_error_rate > 2%",
        "tnow": "2026-05-04T16:50:00Z",
        "failing_surface_hints": [],
        "expected_t0": "2026-05-04T16:44:00Z",
        "expected_change_count": 0,
        "expected_change_summary": "zero changes in window",
        "expected_path": "outside-reference-paths",
        "expected_top_mitigation": "escalate to a human (third-party rate limit hypothesis)",
        "expected_escalate": True,
    },
    {
        "id": "06-ambiguous-t0-slow-burn",
        "fixture_dir": FIXTURES_DIR / "06-ambiguous-t0-slow-burn",
        "service": "recommendations-api",
        "alert_time": "2026-04-19T12:15:00Z",
        "alert_message": "operator-triggered investigation: 'slow all morning'",
        "tnow": "2026-04-19T12:15:00Z",
        "failing_surface_hints": [],
        "expected_t0": "2026-04-19T09:32:14Z",
        "expected_change_count": 0,
        "expected_change_summary": "zero changes in window (causal deploy is outside window)",
        "expected_path": "OOM",
        "expected_top_mitigation": "re-run investigation with widened window before acting",
        "expected_escalate": True,
    },
    {
        "id": "07-blast-radius-asymmetric-revert",
        "fixture_dir": FIXTURES_DIR / "07-blast-radius-asymmetric-revert",
        "service": "notifications-svc",
        "alert_time": "2026-06-02T10:03:00Z",
        "alert_message": "notifications_sms_error_rate > 50%",
        "tnow": "2026-06-02T10:10:00Z",
        "failing_surface_hints": ["twilio", "sms"],
        "expected_t0": "2026-06-02T10:03:00Z",
        "expected_change_count": 1,
        "expected_change_summary": "deploy notifications-svc v8.12.0 bundle of 6 PRs",
        "expected_path": "deploy-correlator",
        "expected_top_mitigation": "revert v8.12.0 (escalation: M3 bundle of 6 changes, broader blast radius)",
        "expected_escalate": True,
    },
    {
        "id": "08-deploy-correlator-confirmation-bias",
        "fixture_dir": FIXTURES_DIR / "08-deploy-correlator-confirmation-bias",
        "service": "users-api",
        "alert_time": "2026-07-11T14:33:00Z",
        "alert_message": "users_api_error_rate > 2%",
        "tnow": "2026-07-11T14:38:00Z",
        "failing_surface_hints": ["secret", "auth", "rbac"],
        "expected_t0": "2026-07-11T14:33:00Z",
        "expected_change_count": 2,
        "expected_change_summary": "deploy (innocent) + RBAC RoleBinding deletion (the actual cause)",
        "expected_path": "outside-reference-paths",
        "expected_top_mitigation": "escalate; the deploy is NOT the cause; investigate RBAC RoleBinding revert",
        "expected_escalate": True,
    },
    {
        "id": "09-zero-changes-external-cert-expiry",
        "fixture_dir": FIXTURES_DIR / "09-zero-changes-external-cert-expiry",
        "service": "webhook-receiver",
        "alert_time": "2026-08-20T03:18:00Z",
        "alert_message": "webhook_receiver_error_rate > 4%",
        "tnow": "2026-08-20T03:25:00Z",
        "failing_surface_hints": [],
        "expected_t0": "2026-08-20T03:18:00Z",
        "expected_change_count": 0,
        "expected_change_summary": "zero changes in window",
        "expected_path": "outside-reference-paths",
        "expected_top_mitigation": "escalate; external partner TLS cert expired",
        "expected_escalate": True,
    },
    {
        "id": "10-multi-region-asymmetry",
        "fixture_dir": FIXTURES_DIR / "10-multi-region-asymmetry",
        "service": "image-svc",
        "alert_time": "2026-09-14T11:42:00Z",
        "alert_message": "image_svc_error_rate > 5% (aggregate)",
        "tnow": "2026-09-14T11:50:00Z",
        "failing_surface_hints": [],
        "expected_t0": "2026-09-14T11:42:00Z",
        "expected_change_count": 0,
        "expected_change_summary": "zero changes in window; regional asymmetry us-east-1 vs us-west-2",
        "expected_path": "outside-reference-paths",
        "expected_top_mitigation": "traffic-shift to us-west-2; escalate config-drift investigation",
        "expected_escalate": True,
    },
    {
        "id": "11-capacity-bound-organic-growth",
        "fixture_dir": FIXTURES_DIR / "11-capacity-bound-organic-growth",
        "service": "search-svc",
        "alert_time": "2026-10-08T10:55:00Z",
        "alert_message": "search_svc_error_rate > 3%",
        "tnow": "2026-10-08T11:05:00Z",
        "failing_surface_hints": [],
        "expected_t0": "2026-10-08T10:55:00Z",
        "expected_change_count": 0,
        "expected_change_summary": "zero changes in window; sustained request-rate growth",
        "expected_path": "outside-reference-paths",
        "expected_top_mitigation": "scale search-svc horizontally",
        "expected_escalate": True,
    },
]


def load_fixture_text(fixture_dir: Path) -> str:
    """Concatenate the fixture files into a single text blob for prompt embedding."""
    parts = []
    for name in ("deploys.json", "pod_events.jsonl", "metrics.json", "logs.jsonl", "traces.jsonl"):
        path = fixture_dir / name
        if not path.exists():
            continue
        parts.append(f"=== {name} ===\n{path.read_text()}")
    return "\n\n".join(parts)
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 196kUpdated Claude CodeCodexVerifiedInstall