Skip to content

anyshift-io/sre-skills / skills/kubectl-investigator

kubectl-investigator

Unverified

Investigate a live or recent incident in a Kubernetes cluster. Anchor the window, bisect the change surface (rollouts, ConfigMaps/Secrets, RBAC, HPA/cluster changes, CronJobs), classify against four reference failure paths (OOM, DNS, cascading-failure, deploy-correlator), confirm the hypothesis with three independent signals, quantify blast radius, and propose mitigation before root cause. Use whenever an agent is asked "what is breaking in the cluster right now", "why did this pod/Deployment just page", "did the rollout cause Z", or to triage an active Kubernetes incident. Vendor-neutral by default (works with kubectl, kube-state-metrics, and whatever telemetry you have); an opt-in Anyshift integration is documented separately.

FileKindSize
FAILURE_MODES.mdmarkdown6.8 KB
README.mdmarkdown4.0 KB
SKILL.mdskill md19.4 KB
examples/01-oom-cascade.mdmarkdown7.4 KB
examples/02-dns-resolution-failure.mdmarkdown7.1 KB
examples/03-cascading-failure-retry-storm.mdmarkdown8.2 KB
examples/04-deploy-correlator-serialization.mdmarkdown7.9 KB
examples/05-outside-reference-paths-third-party-rate-limit.mdmarkdown7.8 KB
examples/06-ambiguous-t0-slow-burn.mdmarkdown7.9 KB
examples/07-blast-radius-asymmetric-revert.mdmarkdown8.1 KB
examples/08-deploy-correlator-confirmation-bias.mdmarkdown9.1 KB
examples/09-zero-changes-external-cert-expiry.mdmarkdown6.3 KB
examples/10-multi-region-asymmetry.mdmarkdown7.1 KB
examples/11-capacity-bound-organic-growth.mdmarkdown6.6 KB
fixtures/01-oom-cascade/deploys.jsonconfig789 B
fixtures/01-oom-cascade/logs.jsonlother1.3 KB
fixtures/01-oom-cascade/metrics.jsonconfig1.3 KB
fixtures/01-oom-cascade/pod_events.jsonlother1.4 KB
fixtures/01-oom-cascade/traces.jsonlother1.5 KB
fixtures/02-dns-resolution-failure/deploys.jsonconfig706 B
fixtures/02-dns-resolution-failure/logs.jsonlother2.4 KB
fixtures/02-dns-resolution-failure/metrics.jsonconfig1.3 KB
fixtures/02-dns-resolution-failure/traces.jsonlother1.4 KB
fixtures/03-cascading-failure-retry-storm/deploys.jsonconfig416 B
fixtures/03-cascading-failure-retry-storm/logs.jsonlother1.4 KB
fixtures/03-cascading-failure-retry-storm/metrics.jsonconfig1.7 KB
fixtures/03-cascading-failure-retry-storm/traces.jsonlother1.6 KB
fixtures/04-deploy-correlator-serialization/deploys.jsonconfig803 B
fixtures/04-deploy-correlator-serialization/logs.jsonlother1.4 KB
fixtures/04-deploy-correlator-serialization/metrics.jsonconfig1.0 KB
fixtures/04-deploy-correlator-serialization/traces.jsonlother1.3 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/deploys.jsonconfig390 B
fixtures/05-outside-reference-paths-third-party-rate-limit/logs.jsonlother1.6 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/metrics.jsonconfig1.3 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/traces.jsonlother1.3 KB
fixtures/06-ambiguous-t0-slow-burn/deploys.jsonconfig517 B
fixtures/06-ambiguous-t0-slow-burn/logs.jsonlother862 B
fixtures/06-ambiguous-t0-slow-burn/metrics.jsonconfig1.2 KB
fixtures/06-ambiguous-t0-slow-burn/traces.jsonlother933 B
fixtures/07-blast-radius-asymmetric-revert/deploys.jsonconfig905 B
fixtures/07-blast-radius-asymmetric-revert/logs.jsonlother1.7 KB
fixtures/07-blast-radius-asymmetric-revert/metrics.jsonconfig1.4 KB
fixtures/07-blast-radius-asymmetric-revert/traces.jsonlother1.5 KB
fixtures/08-deploy-correlator-confirmation-bias/deploys.jsonconfig812 B
fixtures/08-deploy-correlator-confirmation-bias/logs.jsonlother2.0 KB
fixtures/08-deploy-correlator-confirmation-bias/metrics.jsonconfig1.3 KB
fixtures/08-deploy-correlator-confirmation-bias/traces.jsonlother1.3 KB
fixtures/09-zero-changes-external-cert-expiry/deploys.jsonconfig145 B
fixtures/09-zero-changes-external-cert-expiry/logs.jsonlother1.2 KB
fixtures/09-zero-changes-external-cert-expiry/metrics.jsonconfig1.3 KB
fixtures/09-zero-changes-external-cert-expiry/traces.jsonlother1.4 KB
fixtures/10-multi-region-asymmetry/deploys.jsonconfig138 B
fixtures/10-multi-region-asymmetry/logs.jsonlother1.7 KB
fixtures/10-multi-region-asymmetry/metrics.jsonconfig2.0 KB
fixtures/10-multi-region-asymmetry/traces.jsonlother1.6 KB
fixtures/11-capacity-bound-organic-growth/deploys.jsonconfig139 B
fixtures/11-capacity-bound-organic-growth/logs.jsonlother948 B
fixtures/11-capacity-bound-organic-growth/metrics.jsonconfig1.2 KB
fixtures/11-capacity-bound-organic-growth/traces.jsonlother1.2 KB
tests/README.mdmarkdown4.3 KB
tests/_methodology.pyscript22.6 KB
tests/eval/README.mdmarkdown12.7 KB
tests/eval/eval_results.jsonconfig852.7 KB
tests/eval/incidents.pyscript8.6 KB
tests/eval/judge_prompt.mdmarkdown2.4 KB
tests/eval/rubric.mdmarkdown3.1 KB
tests/eval/run_eval.pyscript15.0 KB
tests/eval/run_until_pass.shscript2.1 KB
tests/replay_01_oom_cascade.pyscript3.6 KB
tests/replay_02_dns.pyscript3.3 KB
tests/replay_03_cascade.pyscript3.2 KB
tests/replay_04_deploy_correlator.pyscript3.3 KB
tests/replay_05_outside_paths.pyscript2.4 KB
tests/replay_06_ambiguous_t0.pyscript2.3 KB
tests/replay_07_blast_radius.pyscript2.4 KB
tests/replay_08_confirmation_bias.pyscript2.7 KB
tests/replay_09_cert_expiry.pyscript2.0 KB
tests/replay_10_multi_region.pyscript2.4 KB
tests/replay_11_capacity_bound.pyscript2.3 KB

examples/06-ambiguous-t0-slow-burn.md

View on GitHub
# Worked example 6: ambiguous T0 (slow-burn memory leak)

A failure where T0 is genuinely unclear: the operator says "the service has been slow all morning", error rate has been creeping up for hours, no single alert fire-time pinpoints the start. The methodology must **flag T0 as ambiguous, classify with reduced confidence, and escalate with M2** so a human re-runs the investigation against a widened window. Exercises FAILURE_MODES.md rule M2. Fixtures and replay test under `../fixtures/06-ambiguous-t0-slow-burn/` and `../tests/replay_06_ambiguous_t0.py`.

## Scenario

- **Service**: `recommendations-api`. Operator reports at 12:15 that "recs have been slow all morning, finally got around to looking".
- **Underlying issue**: a memory leak introduced by a Deployment rollout three days ago. Pod RSS has been climbing steadily for ~72 hours, finally crossing the threshold where GC pauses started showing up around 09:30 today. Error rate ticked up from 0.3% to 1.4% over the morning, well below the 2% alert threshold.
- **No alert fired**. The investigation was triggered manually by the operator.
- **Methodology must produce**: classification (likely OOM, given the signature), but with the **t0_ambiguous flag set**, escalation reasons including **M2**, and the recommended mitigation must include a "re-check after widening window" step before any irreversible action.

## Step 1: anchor the window

The operator-reported symptom ("slow all morning") does not pinpoint a single T0. Per SKILL.md step 1:

> If T0 is ambiguous (e.g. "slow all morning"), pick the earliest unambiguous signal and note the ambiguity in the timeline. Do not silently round.

Earliest unambiguous signal: the first GC pause warning at `2026-04-19T09:32:14Z`. T0 set there with `t0_ambiguous = true`.

- **T0**: `2026-04-19T09:32:14Z` (earliest unambiguous signal, ambiguity flagged).
- **Tnow**: `2026-04-19T12:15:00Z`.
- **Window**: `[09:17:14Z, 12:15:00Z]`.

The window is nearly 3 hours wide instead of the typical 20 to 30 minutes. The 15-minute lead-in is structurally insufficient: the actual triggering deploy is 3 days outside the window. The methodology must flag this.

## Step 2: bisect the change surface

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | (none in window) | | Most recent `recommendations-api` rollout: `v3.8.0` at `2026-04-16T15:22:00Z`, ~70 hours before T0. **Outside the window.** |
| Cluster / HPA | (none) | | |
| RBAC | (none) | | |
| ConfigMap / flags | (none) | | |
| CronJob | (none in window) | | |

Zero changes in the window. **This is a misleading "zero changes" signal**: it suggests an environmental cause when the actual cause is a 3-day-old rollout whose effect built up slowly. The methodology's standard "no changes → external cause" pivot would be wrong here, which is exactly why M2 escalation matters.

## Step 3: classify against the four reference paths

- **OOM**: pod RSS p95 climbs from baseline ~120 MB to ~485 MB over the window (`resources.limits.memory` 512 MB). GC pause durations grow from ~20 ms to ~180 ms. No `OOMKilled` events yet, but the trajectory points there within an hour. **Strong match for OOM signature** (RSS >= 90% of limit).
- **DNS**: no `SERVFAIL` / `getaddrinfo` errors. No match.
- **Cascading-failure**: no cascade signature. Upstream latency flat. No match.
- **Deploy-correlator**: no Deployment rollout in window. No match against the windowed surface (but the actual cause is a rollout 3 days outside the window).

Classification: **OOM**. The signature is strong even with the ambiguous T0.

## Step 4: confirm with three independent signals

1. **Metrics**: `rss_bytes_p95` 119 MB at 09:17 climbs to 485 MB at 12:15 (95% of the 512 MB `resources.limits.memory`). GC pause p99 from 22 ms to 184 ms across the same window.
2. **Logs**: `GC pause exceeded soft threshold` warnings starting 09:32:14Z, increasing in frequency through the window.
3. **Traces**: latency tail on `recommendations-api` spans grows from p99 ~80 ms at 09:30 to p99 ~310 ms at 12:00, consistent with GC pauses stealing serving time.

Three signals, three sources. But all of them are consistent with a *slow* OOM trajectory, not a sudden one. The M2 escalation matters because the change that *caused* the leak is invisible to this investigation.

## Step 5: quantify blast radius

- **Users affected**: error rate 1.4% at Tnow, slowly climbing. Latency degradation visible to all users of `recommendations-api`.
- **Surfaces affected**: every endpoint on `recommendations-api`. Recommendation cards on home page, related items on product pages, email recommendations pipeline.
- **Business impact**: degraded recommendations quality, lower click-through rate. No hard outage yet but trajectory points to OOMKills within the hour.

## Step 6: propose mitigation before root cause

Because T0 is ambiguous and the actual triggering change is outside the window, the methodology's mitigation list has a critical addition: **re-check with a widened window before executing**.

1. **Re-check with a widened window first.** Re-run the investigation with `T0 = T0 - 4h` or wider, until the change-surface bisection returns a meaningful candidate. The 3-day-old deploy will appear. *This must precede any irreversible mitigation.*
2. **Revert the implicated change** with `kubectl rollout undo` once the widened-window investigation identifies it. The current investigation has no change to revert.
3. **Scale up** by raising `resources.limits.memory` from 512 MB to 1 GB as a short-term stopgap to delay the OOMKill trajectory while step 1 runs. Acknowledged this does not address cause.
4. **Manual intervention** (rolling restart of `recommendations-api` pods to reset RSS): only if step 3 is blocked. The leak will resume; this is bridge time, not mitigation.

Recommended action: **re-run with widened window, then `kubectl rollout undo` the identified change**.

## Step 7: hand off

```json
{
  "timeline": [
    {"t": "2026-04-19T09:17:14Z", "event": "Window start (15 min before earliest unambiguous signal)"},
    {"t": "2026-04-19T09:32:14Z", "event": "T0 (ambiguous): first GC pause exceeded soft threshold"},
    {"t": "2026-04-19T10:45:00Z", "event": "Error rate crosses 1%"},
    {"t": "2026-04-19T12:15:00Z", "event": "Tnow: operator-triggered investigation"}
  ],
  "ranked_hypotheses": [
    {
      "path": "OOM (slow trajectory, cause outside window)",
      "confidence": "medium (T0 ambiguous; widened-window investigation required to identify causal change)",
      "evidence": ["RSS p95 reached 95% of limit", "GC pause p99 8x baseline", "latency tail growing"]
    }
  ],
  "t0_ambiguous": true,
  "escalate_to_human": true,
  "escalation_reasons": ["M2: T0 is ambiguous; re-run with a widened window before acting on the recommended mitigation"],
  "mitigation_taken": null,
  "mitigation_recommended": "Re-run investigation with widened window (T0 - 4h or more) to identify the causal change, then revert that change.",
  "open_questions": [
    "When did the leak actually start? Likely deploy v3.8.0 at 2026-04-16T15:22:00Z, but confirm with widened-window run.",
    "Is there a per-request memory allocation chart that would distinguish a leak from organic growth?",
    "Why did no alert fire in the 3-day climb? Is the alert threshold (2% error rate) the wrong shape for a slow leak?"
  ]
}
```

## Why this is the ambiguous-T0 reference example

- It's the case where the methodology's *default* T0-picking heuristic ("first user-visible symptom") produces a window that misses the actual cause. Without M2 escalation, the methodology would confidently recommend "no change to revert, must be environmental" and be wrong.
- It exercises the slow-trajectory vs. sudden-failure distinction. The OOM signature is present but evolving over hours, not minutes; the standard 15-minute lead-in is insufficient.
- It models the gap between *what the operator can see* and *what the agent can act on*. Escalation is the safety net: the agent surfaces the partial picture, flags the gap, recommends a re-run.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 196kUpdated Claude CodeCodexVerifiedInstall