Skip to content

anyshift-io/sre-skills / skills/kubectl-investigator

kubectl-investigator

Unverified

Investigate a live or recent incident in a Kubernetes cluster. Anchor the window, bisect the change surface (rollouts, ConfigMaps/Secrets, RBAC, HPA/cluster changes, CronJobs), classify against four reference failure paths (OOM, DNS, cascading-failure, deploy-correlator), confirm the hypothesis with three independent signals, quantify blast radius, and propose mitigation before root cause. Use whenever an agent is asked "what is breaking in the cluster right now", "why did this pod/Deployment just page", "did the rollout cause Z", or to triage an active Kubernetes incident. Vendor-neutral by default (works with kubectl, kube-state-metrics, and whatever telemetry you have); an opt-in Anyshift integration is documented separately.

FileKindSize
FAILURE_MODES.mdmarkdown6.8 KB
README.mdmarkdown4.0 KB
SKILL.mdskill md19.4 KB
examples/01-oom-cascade.mdmarkdown7.4 KB
examples/02-dns-resolution-failure.mdmarkdown7.1 KB
examples/03-cascading-failure-retry-storm.mdmarkdown8.2 KB
examples/04-deploy-correlator-serialization.mdmarkdown7.9 KB
examples/05-outside-reference-paths-third-party-rate-limit.mdmarkdown7.8 KB
examples/06-ambiguous-t0-slow-burn.mdmarkdown7.9 KB
examples/07-blast-radius-asymmetric-revert.mdmarkdown8.1 KB
examples/08-deploy-correlator-confirmation-bias.mdmarkdown9.1 KB
examples/09-zero-changes-external-cert-expiry.mdmarkdown6.3 KB
examples/10-multi-region-asymmetry.mdmarkdown7.1 KB
examples/11-capacity-bound-organic-growth.mdmarkdown6.6 KB
fixtures/01-oom-cascade/deploys.jsonconfig789 B
fixtures/01-oom-cascade/logs.jsonlother1.3 KB
fixtures/01-oom-cascade/metrics.jsonconfig1.3 KB
fixtures/01-oom-cascade/pod_events.jsonlother1.4 KB
fixtures/01-oom-cascade/traces.jsonlother1.5 KB
fixtures/02-dns-resolution-failure/deploys.jsonconfig706 B
fixtures/02-dns-resolution-failure/logs.jsonlother2.4 KB
fixtures/02-dns-resolution-failure/metrics.jsonconfig1.3 KB
fixtures/02-dns-resolution-failure/traces.jsonlother1.4 KB
fixtures/03-cascading-failure-retry-storm/deploys.jsonconfig416 B
fixtures/03-cascading-failure-retry-storm/logs.jsonlother1.4 KB
fixtures/03-cascading-failure-retry-storm/metrics.jsonconfig1.7 KB
fixtures/03-cascading-failure-retry-storm/traces.jsonlother1.6 KB
fixtures/04-deploy-correlator-serialization/deploys.jsonconfig803 B
fixtures/04-deploy-correlator-serialization/logs.jsonlother1.4 KB
fixtures/04-deploy-correlator-serialization/metrics.jsonconfig1.0 KB
fixtures/04-deploy-correlator-serialization/traces.jsonlother1.3 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/deploys.jsonconfig390 B
fixtures/05-outside-reference-paths-third-party-rate-limit/logs.jsonlother1.6 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/metrics.jsonconfig1.3 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/traces.jsonlother1.3 KB
fixtures/06-ambiguous-t0-slow-burn/deploys.jsonconfig517 B
fixtures/06-ambiguous-t0-slow-burn/logs.jsonlother862 B
fixtures/06-ambiguous-t0-slow-burn/metrics.jsonconfig1.2 KB
fixtures/06-ambiguous-t0-slow-burn/traces.jsonlother933 B
fixtures/07-blast-radius-asymmetric-revert/deploys.jsonconfig905 B
fixtures/07-blast-radius-asymmetric-revert/logs.jsonlother1.7 KB
fixtures/07-blast-radius-asymmetric-revert/metrics.jsonconfig1.4 KB
fixtures/07-blast-radius-asymmetric-revert/traces.jsonlother1.5 KB
fixtures/08-deploy-correlator-confirmation-bias/deploys.jsonconfig812 B
fixtures/08-deploy-correlator-confirmation-bias/logs.jsonlother2.0 KB
fixtures/08-deploy-correlator-confirmation-bias/metrics.jsonconfig1.3 KB
fixtures/08-deploy-correlator-confirmation-bias/traces.jsonlother1.3 KB
fixtures/09-zero-changes-external-cert-expiry/deploys.jsonconfig145 B
fixtures/09-zero-changes-external-cert-expiry/logs.jsonlother1.2 KB
fixtures/09-zero-changes-external-cert-expiry/metrics.jsonconfig1.3 KB
fixtures/09-zero-changes-external-cert-expiry/traces.jsonlother1.4 KB
fixtures/10-multi-region-asymmetry/deploys.jsonconfig138 B
fixtures/10-multi-region-asymmetry/logs.jsonlother1.7 KB
fixtures/10-multi-region-asymmetry/metrics.jsonconfig2.0 KB
fixtures/10-multi-region-asymmetry/traces.jsonlother1.6 KB
fixtures/11-capacity-bound-organic-growth/deploys.jsonconfig139 B
fixtures/11-capacity-bound-organic-growth/logs.jsonlother948 B
fixtures/11-capacity-bound-organic-growth/metrics.jsonconfig1.2 KB
fixtures/11-capacity-bound-organic-growth/traces.jsonlother1.2 KB
tests/README.mdmarkdown4.3 KB
tests/_methodology.pyscript22.6 KB
tests/eval/README.mdmarkdown12.7 KB
tests/eval/eval_results.jsonconfig852.7 KB
tests/eval/incidents.pyscript8.6 KB
tests/eval/judge_prompt.mdmarkdown2.4 KB
tests/eval/rubric.mdmarkdown3.1 KB
tests/eval/run_eval.pyscript15.0 KB
tests/eval/run_until_pass.shscript2.1 KB
tests/replay_01_oom_cascade.pyscript3.6 KB
tests/replay_02_dns.pyscript3.3 KB
tests/replay_03_cascade.pyscript3.2 KB
tests/replay_04_deploy_correlator.pyscript3.3 KB
tests/replay_05_outside_paths.pyscript2.4 KB
tests/replay_06_ambiguous_t0.pyscript2.3 KB
tests/replay_07_blast_radius.pyscript2.4 KB
tests/replay_08_confirmation_bias.pyscript2.7 KB
tests/replay_09_cert_expiry.pyscript2.0 KB
tests/replay_10_multi_region.pyscript2.4 KB
tests/replay_11_capacity_bound.pyscript2.3 KB

examples/01-oom-cascade.md

View on GitHub
# Worked example 1: OOM cascade in `payments-api`

A realistic OOM incident in a payment processing service, triggered by a deploy that increased per-request memory footprint. Mirrors the seven methodology steps in [`../SKILL.md`](../SKILL.md). The fixtures and replay test under `../fixtures/01-oom-cascade/` and `../tests/replay_01_oom_cascade.py` exercise this example end-to-end.

## Scenario

- **Service**: `payments-api` (HTTP service, 8 pods, behind `api-gateway`).
- **Deploy at 2026-03-12 14:18 UTC**: introduced webhook payload buffering. Per-request memory footprint went from ~80 MB to ~210 MB. Pod memory limit unchanged at 512 MB.
- **Alert fires at 14:32 UTC**: `payments_api_error_rate > 5%`.
- **Cascade**: at 14:30 the pods hit the memory limit, the kubelet starts `OOMKill`ing them, the Deployment's ReplicaSet restarts them, in-flight requests fail with `502`, the `api-gateway` retries failed requests (3 retries, 500 ms backoff), the retry storm doubles the incoming request rate, the still-recovering pods OOM faster, error rate climbs from 0.4% to 47% in 90 seconds.

## Step 1: anchor the window

Earliest unambiguous symptom is the alert at `2026-03-12T14:32:00Z`. No customer reports filed yet. T0 set there.

- **T0**: `2026-03-12T14:32:00Z`
- **Tnow**: `2026-03-12T14:36:00Z` (investigation triggered four minutes after the alert)
- **Window**: `[14:17:00Z, 14:36:00Z]`

The 15-minute lead-in is what catches the 14:18 deploy in step 2. Without it, the bisection in step 2 would return empty.

## Step 2: bisect the change surface

Changes overlapping the window:

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | Deploy `payments-api@v4.18.0` | `14:18:14Z` | Merge commit `9f3a2c1`. Diff touches `internal/webhook/buffer.go` (new payload-buffering path). |
| Cluster / HPA | (none) | | No node-pool / HPA / resource-limit changes in window. |
| RBAC | (none) | | No Role / RoleBinding changes in window. |
| ConfigMap / flags | (none) | | No flips in window. |
| CronJob | (none) | | No batch jobs in window. |

One change in window: the `payments-api` deploy 14 minutes before T0. Strong candidate for the deploy-correlator path. Need to check the OOM path in parallel (both paths can be true; OOM is often *triggered* by a deploy).

## Step 3: classify against the four reference paths

Match against the four reference paths:

- **OOM**: pod restart count climbed from 0 to 12 in the window. Pod / kubelet events show `OOMKilled` on 6 of 8 pods. Memory metric shows RSS at 510 MB (limit 512 MB) immediately before each OOMKill. Strong match.
- **DNS**: no `NXDOMAIN` / `SERVFAIL` in logs, no `getaddrinfo` errors, outbound latency unchanged. No match.
- **Cascading-failure**: retry count at `api-gateway` spiked 3.2x in the window. Latency on downstream `ledger-svc` calls unchanged. Cascade signature is present but is *downstream of* the OOM, not the root.
- **Deploy-correlator**: 14:18 deploy is the only change in window, diff touches the failing surface (`internal/webhook/buffer.go` is in the request path), and the new code allocates ~130 MB more per concurrent request. Strong match.

Classification: **OOM**, triggered by the deploy-correlator path. The cascading-failure signature is a second-order effect (retry storm amplifying OOM pressure), not an independent path.

## Step 4: confirm with three independent signals

Three independent signals supporting the OOM-via-deploy hypothesis:

1. **Pod / kubelet events** (`fixtures/01-oom-cascade/pod_events.jsonl`): `OOMKilled` on 6 of 8 pods between 14:30 and 14:35.
2. **Metrics** (`fixtures/01-oom-cascade/metrics.json`): RSS at 510 MB at 14:30:12Z, immediately before the first OOMKill.
3. **Deploy diff** (`fixtures/01-oom-cascade/deploys.json`): commit `9f3a2c1` adds a `WebhookBuffer` struct that holds full payload bodies in memory; the per-request memory increase aligns with the observed RSS jump.

A fourth signal corroborates the cascade as second-order: **traces** (`fixtures/01-oom-cascade/traces.jsonl`) show retry-storm spans from `api-gateway` only after the first OOMKill, not before.

Hypothesis confidence is high. Three signals, three independent sources.

## Step 5: quantify blast radius

- **Users affected**: 100% of payment requests fail or hit retry latency. Read-only endpoints (`GET /payments/:id`) are also affected because they share the same pods.
- **Surfaces affected**: all payment endpoints (`POST /charge`, `POST /refund`, `GET /payments/:id`, webhook callbacks).
- **Business impact**: payment processing offline. SLO burn at 12x normal rate. In a real incident, the agent should also pull the revenue-per-minute figure from the finance dashboard the user has on hand.

## Step 6: propose mitigation before root cause

Ordered recommendation:

1. **`kubectl rollout undo`** `payments-api` to `v4.17.4` (the previous successful ReplicaSet). The deploy is the clearly implicated change, the rollback is one `kubectl rollout undo deployment/payments-api`, and the OOM behavior should clear within one pod-restart cycle (~30 s).
2. **ConfigMap-flag off** the webhook buffering path. Not applicable: the change shipped without a flag, which is itself a process gap to flag in the postmortem.
3. **Raise `resources.limits.memory`** from 512 MB to 1 GB as a stopgap if the rollback is delayed. Acknowledged that this does not address cause and will roughly double per-pod cost.
4. **Traffic-shift** to a previous region: not applicable, single-region service.
5. **Manual intervention** (`kubectl delete pod` to force fresh restarts): only if rollback and the limit bump are both blocked. Does not address cause; pods will OOM again within minutes.

Recommended action: **`kubectl rollout undo` to `v4.17.4`**.

## Step 7: hand off

Handoff payload for `postmortem-author`:

```json
{
  "timeline": [
    {"t": "2026-03-12T14:18:14Z", "event": "Deploy payments-api@v4.18.0 (commit 9f3a2c1)"},
    {"t": "2026-03-12T14:30:12Z", "event": "First OOMKill on pod payments-api-7d9c-x4k2"},
    {"t": "2026-03-12T14:32:00Z", "event": "T0: alert payments_api_error_rate > 5% fires"},
    {"t": "2026-03-12T14:33:45Z", "event": "api-gateway retry count 3.2x baseline"},
    {"t": "2026-03-12T14:36:00Z", "event": "Tnow: investigation triggered"}
  ],
  "ranked_hypotheses": [
    {
      "path": "OOM (deploy-triggered)",
      "confidence": "high",
      "evidence": ["OOMKilled events on 6/8 pods", "RSS at memory limit", "deploy diff adds ~130MB/req via WebhookBuffer"]
    }
  ],
  "mitigation_taken": null,
  "mitigation_recommended": "Revert payments-api to v4.17.4",
  "open_questions": [
    "Why did the new WebhookBuffer path ship without a memory-footprint review?",
    "Why is there no per-request memory guardrail in the payment service load tests?"
  ]
}
```

## Why this is the OOM reference path

This example is the reference because it shows all four of these in one incident:

- A clear OOM signature at the pod level.
- A clear deploy-correlator signature (one change in window, on the failing surface).
- A clear cascading-failure signature as a second-order effect (api-gateway retry storm amplifying the primary failure).
- A safe mitigation (revert) that comes before root cause.

The methodology handles the layering: classify the primary path, note the cascade as second-order, recommend the revert. Skipping any of steps 1 through 4 produces a wrong answer (most commonly: classifying as cascading-failure and recommending circuit-breaker tuning, which does nothing to fix the OOM).
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall