Skip to content

anyshift-io/sre-skills / skills/kubectl-investigator

kubectl-investigator

Unverified

Investigate a live or recent incident in a Kubernetes cluster. Anchor the window, bisect the change surface (rollouts, ConfigMaps/Secrets, RBAC, HPA/cluster changes, CronJobs), classify against four reference failure paths (OOM, DNS, cascading-failure, deploy-correlator), confirm the hypothesis with three independent signals, quantify blast radius, and propose mitigation before root cause. Use whenever an agent is asked "what is breaking in the cluster right now", "why did this pod/Deployment just page", "did the rollout cause Z", or to triage an active Kubernetes incident. Vendor-neutral by default (works with kubectl, kube-state-metrics, and whatever telemetry you have); an opt-in Anyshift integration is documented separately.

FileKindSize
FAILURE_MODES.mdmarkdown6.8 KB
README.mdmarkdown4.0 KB
SKILL.mdskill md19.4 KB
examples/01-oom-cascade.mdmarkdown7.4 KB
examples/02-dns-resolution-failure.mdmarkdown7.1 KB
examples/03-cascading-failure-retry-storm.mdmarkdown8.2 KB
examples/04-deploy-correlator-serialization.mdmarkdown7.9 KB
examples/05-outside-reference-paths-third-party-rate-limit.mdmarkdown7.8 KB
examples/06-ambiguous-t0-slow-burn.mdmarkdown7.9 KB
examples/07-blast-radius-asymmetric-revert.mdmarkdown8.1 KB
examples/08-deploy-correlator-confirmation-bias.mdmarkdown9.1 KB
examples/09-zero-changes-external-cert-expiry.mdmarkdown6.3 KB
examples/10-multi-region-asymmetry.mdmarkdown7.1 KB
examples/11-capacity-bound-organic-growth.mdmarkdown6.6 KB
fixtures/01-oom-cascade/deploys.jsonconfig789 B
fixtures/01-oom-cascade/logs.jsonlother1.3 KB
fixtures/01-oom-cascade/metrics.jsonconfig1.3 KB
fixtures/01-oom-cascade/pod_events.jsonlother1.4 KB
fixtures/01-oom-cascade/traces.jsonlother1.5 KB
fixtures/02-dns-resolution-failure/deploys.jsonconfig706 B
fixtures/02-dns-resolution-failure/logs.jsonlother2.4 KB
fixtures/02-dns-resolution-failure/metrics.jsonconfig1.3 KB
fixtures/02-dns-resolution-failure/traces.jsonlother1.4 KB
fixtures/03-cascading-failure-retry-storm/deploys.jsonconfig416 B
fixtures/03-cascading-failure-retry-storm/logs.jsonlother1.4 KB
fixtures/03-cascading-failure-retry-storm/metrics.jsonconfig1.7 KB
fixtures/03-cascading-failure-retry-storm/traces.jsonlother1.6 KB
fixtures/04-deploy-correlator-serialization/deploys.jsonconfig803 B
fixtures/04-deploy-correlator-serialization/logs.jsonlother1.4 KB
fixtures/04-deploy-correlator-serialization/metrics.jsonconfig1.0 KB
fixtures/04-deploy-correlator-serialization/traces.jsonlother1.3 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/deploys.jsonconfig390 B
fixtures/05-outside-reference-paths-third-party-rate-limit/logs.jsonlother1.6 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/metrics.jsonconfig1.3 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/traces.jsonlother1.3 KB
fixtures/06-ambiguous-t0-slow-burn/deploys.jsonconfig517 B
fixtures/06-ambiguous-t0-slow-burn/logs.jsonlother862 B
fixtures/06-ambiguous-t0-slow-burn/metrics.jsonconfig1.2 KB
fixtures/06-ambiguous-t0-slow-burn/traces.jsonlother933 B
fixtures/07-blast-radius-asymmetric-revert/deploys.jsonconfig905 B
fixtures/07-blast-radius-asymmetric-revert/logs.jsonlother1.7 KB
fixtures/07-blast-radius-asymmetric-revert/metrics.jsonconfig1.4 KB
fixtures/07-blast-radius-asymmetric-revert/traces.jsonlother1.5 KB
fixtures/08-deploy-correlator-confirmation-bias/deploys.jsonconfig812 B
fixtures/08-deploy-correlator-confirmation-bias/logs.jsonlother2.0 KB
fixtures/08-deploy-correlator-confirmation-bias/metrics.jsonconfig1.3 KB
fixtures/08-deploy-correlator-confirmation-bias/traces.jsonlother1.3 KB
fixtures/09-zero-changes-external-cert-expiry/deploys.jsonconfig145 B
fixtures/09-zero-changes-external-cert-expiry/logs.jsonlother1.2 KB
fixtures/09-zero-changes-external-cert-expiry/metrics.jsonconfig1.3 KB
fixtures/09-zero-changes-external-cert-expiry/traces.jsonlother1.4 KB
fixtures/10-multi-region-asymmetry/deploys.jsonconfig138 B
fixtures/10-multi-region-asymmetry/logs.jsonlother1.7 KB
fixtures/10-multi-region-asymmetry/metrics.jsonconfig2.0 KB
fixtures/10-multi-region-asymmetry/traces.jsonlother1.6 KB
fixtures/11-capacity-bound-organic-growth/deploys.jsonconfig139 B
fixtures/11-capacity-bound-organic-growth/logs.jsonlother948 B
fixtures/11-capacity-bound-organic-growth/metrics.jsonconfig1.2 KB
fixtures/11-capacity-bound-organic-growth/traces.jsonlother1.2 KB
tests/README.mdmarkdown4.3 KB
tests/_methodology.pyscript22.6 KB
tests/eval/README.mdmarkdown12.7 KB
tests/eval/eval_results.jsonconfig852.7 KB
tests/eval/incidents.pyscript8.6 KB
tests/eval/judge_prompt.mdmarkdown2.4 KB
tests/eval/rubric.mdmarkdown3.1 KB
tests/eval/run_eval.pyscript15.0 KB
tests/eval/run_until_pass.shscript2.1 KB
tests/replay_01_oom_cascade.pyscript3.6 KB
tests/replay_02_dns.pyscript3.3 KB
tests/replay_03_cascade.pyscript3.2 KB
tests/replay_04_deploy_correlator.pyscript3.3 KB
tests/replay_05_outside_paths.pyscript2.4 KB
tests/replay_06_ambiguous_t0.pyscript2.3 KB
tests/replay_07_blast_radius.pyscript2.4 KB
tests/replay_08_confirmation_bias.pyscript2.7 KB
tests/replay_09_cert_expiry.pyscript2.0 KB
tests/replay_10_multi_region.pyscript2.4 KB
tests/replay_11_capacity_bound.pyscript2.3 KB

examples/07-blast-radius-asymmetric-revert.md

View on GitHub
# Worked example 7: blast-radius asymmetric revert (deploy bundle)

A deploy bundle that shipped six unrelated changes in one release train. Only one of them is causing the incident, but reverting the bundle reverts all six. The methodology must **classify the path correctly and rank revert as the top mitigation**, but then **escalate with M3** because the revert's blast radius exceeds the incident's blast radius. Exercises FAILURE_MODES.md rule M3. Fixtures and replay test under `../fixtures/07-blast-radius-asymmetric-revert/` and `../tests/replay_07_blast_radius.py`.

## Scenario

- **Service**: `notifications-svc` (a Kubernetes Deployment). Sends transactional emails, SMS, and push notifications; SMS goes through the external Twilio provider.
- **Rollout at 2026-06-02 09:48 UTC**: a weekly release train rolled out a new ReplicaSet bundling six unrelated PRs (commit `8b3f4d1`, version `v8.12.0`):
  1. SMS provider integration update (Twilio API v2 migration).
  2. Email template refactor.
  3. Push notification batching optimization.
  4. Removal of deprecated SES region fallback.
  5. New /healthz endpoint.
  6. Dependency upgrades (express 4.x → 5.x).
- **The actual failure**: the SMS integration update (PR 1 of 6) shipped with an invalid sender ID format that the new Twilio API rejects. SMS delivery is now failing 100% of the time. The other five changes are fine.
- **Alert at 10:03 UTC**: `notifications_sms_error_rate > 50%`.
- **Methodology must produce**: classification deploy-correlator (or close), `kubectl rollout undo` as the top mitigation, **escalation with M3** because rolling back `v8.12.0` reverts the other five working changes too.

## Step 1: anchor the window

- **T0**: `2026-06-02T10:03:00Z`.
- **Tnow**: `2026-06-02T10:10:00Z`.
- **Window**: `[09:48:00Z, 10:10:00Z]`.

## Step 2: bisect the change surface

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | Rollout `notifications-svc@v8.12.0` | `09:48:23Z` | Commit `8b3f4d1`. **Bundle of 6 PRs.** Diff summary lists Twilio v2 migration, email template refactor, push batching, SES fallback removal, /healthz, express 5.x upgrade. |
| Cluster / HPA | (none) | | |
| RBAC | (none) | | |
| ConfigMap / flags | (none) | | |
| CronJob | (none) | | |

One change in window, but the change is a bundle of six. This is the M3 trigger.

## Step 3: classify against the four reference paths

- **OOM**: pod RSS p95 flat at ~210 MB against the 512 MB `resources.limits.memory`. No `OOMKilled` events. No match.
- **DNS**: no `SERVFAIL` / `getaddrinfo` errors. Outbound to Twilio resolves fine. No match.
- **Cascading-failure**: no retry storm, no upstream-latency growth. The failure is at the external SMS provider hop with HTTP 400, not slowness. No match.
- **Deploy-correlator**: rollout in window; diff explicitly touches the SMS path (Twilio API v2 migration line in the bundle); SMS error rate jumped from 0 to 100% within 90 seconds of the rollout. **Match.**

Classification: **deploy-correlator**, specifically the Twilio API v2 migration PR within the bundle. The other five PRs are not implicated by any signal.

## Step 4: confirm with three independent signals

1. **Logs** (`fixtures/07-blast-radius-asymmetric-revert/logs.jsonl`): `Twilio API rejected request: 400 invalid 'from' field, expected E.164 format` errors starting 09:49:51Z.
2. **Metrics** (`fixtures/07-blast-radius-asymmetric-revert/metrics.json`): `sms_error_rate_pct` jumps from 0.0% baseline to 100% at 09:50; `email_error_rate_pct` and `push_error_rate_pct` flat (proving the failure is SMS-scoped, not service-wide).
3. **Deploy diff** (`fixtures/07-blast-radius-asymmetric-revert/deploys.json`): bundle includes Twilio v2 migration line that matches the failing surface.
4. **Traces** (`fixtures/07-blast-radius-asymmetric-revert/traces.jsonl`): SMS-sending spans terminate at the `twilio` hop with `http.status_code=400`; email-sending and push-sending spans complete successfully.

Four independent signal sources. Confidence high.

## Step 5: quantify blast radius

- **Users affected**: every user who would receive an SMS notification. Email and push channels are unaffected.
- **Surfaces affected**: SMS sending only. Other notification channels healthy.
- **Business impact**: SMS-dependent flows broken (2FA SMS, OTP, delivery alerts). Email and push fallbacks compensate partially for some flows but not for 2FA.

**Revert blast radius**: `kubectl rollout undo` to `v8.11.4` would also roll back five working changes (email template refactor, push batching, SES fallback removal, /healthz, express 5.x). Two of those (the SES removal and express 5.x upgrade) have downstream consumers already depending on the new state. The rollback is asymmetric: it removes a 100%-broken feature (SMS) at the cost of regressing five healthy features.

## Step 6: propose mitigation before root cause

1. **Revert** `notifications-svc` to `v8.11.4` with `kubectl rollout undo`. This is the methodology's top recommendation, BUT it carries the M3 caveat: blast radius exceeds the incident's blast radius. **Requires a human approver.**
2. **Feature-flag off**: not applicable; the Twilio v2 migration shipped without a ConfigMap flag.
3. **Scale**: not applicable.
4. **Traffic-shift**: not applicable.
5. **Manual intervention**: not applicable.

A human-approved alternative the methodology surfaces but does not unilaterally recommend: **forward-fix** by shipping a targeted patch to the SMS sender ID format (a one-line fix), rather than rolling back the bundle. Forward-fix avoids the regression of the five healthy changes but introduces deployment-pipeline risk (the bundle's other changes have been live for ~15 min; rolling back and re-shipping the five working changes is operationally noisier).

Recommended action: **`kubectl rollout undo` + escalate to human for blast-radius approval, with forward-fix surfaced as the cleaner alternative**.

## Step 7: hand off

```json
{
  "timeline": [
    {"t": "2026-06-02T09:48:23Z", "event": "Deploy notifications-svc@v8.12.0 (bundle of 6 PRs, commit 8b3f4d1)"},
    {"t": "2026-06-02T09:49:51Z", "event": "First Twilio 400 invalid 'from' field error"},
    {"t": "2026-06-02T10:03:00Z", "event": "T0: alert notifications_sms_error_rate > 50% fires"},
    {"t": "2026-06-02T10:10:00Z", "event": "Tnow: investigation triggered"}
  ],
  "ranked_hypotheses": [
    {
      "path": "Deploy-correlator (Twilio v2 migration within bundle)",
      "confidence": "high",
      "evidence": ["bundle diff includes Twilio v2 migration line", "sms-only error pattern, email/push unaffected", "Twilio 400 invalid 'from' field errors in logs", "traces fail at twilio hop"]
    }
  ],
  "mitigation_taken": null,
  "mitigation_recommended": "Revert notifications-svc to v8.11.4 (M3 escalation flagged: bundle of 6 changes, revert reverts all six)",
  "alternative_mitigation": "Forward-fix: ship a targeted patch to the SMS sender ID format. Preserves the five healthy changes in the bundle but takes longer than revert.",
  "escalate_to_human": true,
  "escalation_reasons": ["M3: recommended revert affects 6 bundled changes, broader blast radius than the incident; requires a human approver before executing"],
  "open_questions": [
    "Why did the SMS sender ID format change land without a contract test against the Twilio v2 sandbox?",
    "Should the release train be split into one-PR-per-deploy for high-risk integration changes like SMS provider migrations?",
    "Is there a way to revert a single commit out of a bundle, or does the deploy pipeline require atomic version rollback?"
  ]
}
```

## Why this is the asymmetric-revert reference example

- It models a real-world operational tension: revert is often correct *at the methodology level* but unsafe *at the operational level* when the change being reverted is bundled. The methodology must surface this conflict, not gloss it.
- It exercises M3 explicitly, with `bundle_size = 6` in the deploy fixture and a methodology that escalates when bundle size > 1 on a revert recommendation.
- It models the *alternative-mitigation* slot in the handoff: when the top recommendation is conflicted, the methodology should surface a cleaner alternative (forward-fix) rather than only flagging the conflict and stopping.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 196kUpdated Claude CodeCodexVerifiedInstall