Skip to content

anyshift-io/sre-skills / skills/kubectl-investigator

kubectl-investigator

Unverified

Investigate a live or recent incident in a Kubernetes cluster. Anchor the window, bisect the change surface (rollouts, ConfigMaps/Secrets, RBAC, HPA/cluster changes, CronJobs), classify against four reference failure paths (OOM, DNS, cascading-failure, deploy-correlator), confirm the hypothesis with three independent signals, quantify blast radius, and propose mitigation before root cause. Use whenever an agent is asked "what is breaking in the cluster right now", "why did this pod/Deployment just page", "did the rollout cause Z", or to triage an active Kubernetes incident. Vendor-neutral by default (works with kubectl, kube-state-metrics, and whatever telemetry you have); an opt-in Anyshift integration is documented separately.

FileKindSize
FAILURE_MODES.mdmarkdown6.8 KB
README.mdmarkdown4.0 KB
SKILL.mdskill md19.4 KB
examples/01-oom-cascade.mdmarkdown7.4 KB
examples/02-dns-resolution-failure.mdmarkdown7.1 KB
examples/03-cascading-failure-retry-storm.mdmarkdown8.2 KB
examples/04-deploy-correlator-serialization.mdmarkdown7.9 KB
examples/05-outside-reference-paths-third-party-rate-limit.mdmarkdown7.8 KB
examples/06-ambiguous-t0-slow-burn.mdmarkdown7.9 KB
examples/07-blast-radius-asymmetric-revert.mdmarkdown8.1 KB
examples/08-deploy-correlator-confirmation-bias.mdmarkdown9.1 KB
examples/09-zero-changes-external-cert-expiry.mdmarkdown6.3 KB
examples/10-multi-region-asymmetry.mdmarkdown7.1 KB
examples/11-capacity-bound-organic-growth.mdmarkdown6.6 KB
fixtures/01-oom-cascade/deploys.jsonconfig789 B
fixtures/01-oom-cascade/logs.jsonlother1.3 KB
fixtures/01-oom-cascade/metrics.jsonconfig1.3 KB
fixtures/01-oom-cascade/pod_events.jsonlother1.4 KB
fixtures/01-oom-cascade/traces.jsonlother1.5 KB
fixtures/02-dns-resolution-failure/deploys.jsonconfig706 B
fixtures/02-dns-resolution-failure/logs.jsonlother2.4 KB
fixtures/02-dns-resolution-failure/metrics.jsonconfig1.3 KB
fixtures/02-dns-resolution-failure/traces.jsonlother1.4 KB
fixtures/03-cascading-failure-retry-storm/deploys.jsonconfig416 B
fixtures/03-cascading-failure-retry-storm/logs.jsonlother1.4 KB
fixtures/03-cascading-failure-retry-storm/metrics.jsonconfig1.7 KB
fixtures/03-cascading-failure-retry-storm/traces.jsonlother1.6 KB
fixtures/04-deploy-correlator-serialization/deploys.jsonconfig803 B
fixtures/04-deploy-correlator-serialization/logs.jsonlother1.4 KB
fixtures/04-deploy-correlator-serialization/metrics.jsonconfig1.0 KB
fixtures/04-deploy-correlator-serialization/traces.jsonlother1.3 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/deploys.jsonconfig390 B
fixtures/05-outside-reference-paths-third-party-rate-limit/logs.jsonlother1.6 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/metrics.jsonconfig1.3 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/traces.jsonlother1.3 KB
fixtures/06-ambiguous-t0-slow-burn/deploys.jsonconfig517 B
fixtures/06-ambiguous-t0-slow-burn/logs.jsonlother862 B
fixtures/06-ambiguous-t0-slow-burn/metrics.jsonconfig1.2 KB
fixtures/06-ambiguous-t0-slow-burn/traces.jsonlother933 B
fixtures/07-blast-radius-asymmetric-revert/deploys.jsonconfig905 B
fixtures/07-blast-radius-asymmetric-revert/logs.jsonlother1.7 KB
fixtures/07-blast-radius-asymmetric-revert/metrics.jsonconfig1.4 KB
fixtures/07-blast-radius-asymmetric-revert/traces.jsonlother1.5 KB
fixtures/08-deploy-correlator-confirmation-bias/deploys.jsonconfig812 B
fixtures/08-deploy-correlator-confirmation-bias/logs.jsonlother2.0 KB
fixtures/08-deploy-correlator-confirmation-bias/metrics.jsonconfig1.3 KB
fixtures/08-deploy-correlator-confirmation-bias/traces.jsonlother1.3 KB
fixtures/09-zero-changes-external-cert-expiry/deploys.jsonconfig145 B
fixtures/09-zero-changes-external-cert-expiry/logs.jsonlother1.2 KB
fixtures/09-zero-changes-external-cert-expiry/metrics.jsonconfig1.3 KB
fixtures/09-zero-changes-external-cert-expiry/traces.jsonlother1.4 KB
fixtures/10-multi-region-asymmetry/deploys.jsonconfig138 B
fixtures/10-multi-region-asymmetry/logs.jsonlother1.7 KB
fixtures/10-multi-region-asymmetry/metrics.jsonconfig2.0 KB
fixtures/10-multi-region-asymmetry/traces.jsonlother1.6 KB
fixtures/11-capacity-bound-organic-growth/deploys.jsonconfig139 B
fixtures/11-capacity-bound-organic-growth/logs.jsonlother948 B
fixtures/11-capacity-bound-organic-growth/metrics.jsonconfig1.2 KB
fixtures/11-capacity-bound-organic-growth/traces.jsonlother1.2 KB
tests/README.mdmarkdown4.3 KB
tests/_methodology.pyscript22.6 KB
tests/eval/README.mdmarkdown12.7 KB
tests/eval/eval_results.jsonconfig852.7 KB
tests/eval/incidents.pyscript8.6 KB
tests/eval/judge_prompt.mdmarkdown2.4 KB
tests/eval/rubric.mdmarkdown3.1 KB
tests/eval/run_eval.pyscript15.0 KB
tests/eval/run_until_pass.shscript2.1 KB
tests/replay_01_oom_cascade.pyscript3.6 KB
tests/replay_02_dns.pyscript3.3 KB
tests/replay_03_cascade.pyscript3.2 KB
tests/replay_04_deploy_correlator.pyscript3.3 KB
tests/replay_05_outside_paths.pyscript2.4 KB
tests/replay_06_ambiguous_t0.pyscript2.3 KB
tests/replay_07_blast_radius.pyscript2.4 KB
tests/replay_08_confirmation_bias.pyscript2.7 KB
tests/replay_09_cert_expiry.pyscript2.0 KB
tests/replay_10_multi_region.pyscript2.4 KB
tests/replay_11_capacity_bound.pyscript2.3 KB

examples/02-dns-resolution-failure.md

View on GitHub
# Worked example 2: DNS resolution failure in `inventory-svc`

A realistic DNS incident in an internal HTTP service caused by a typo in a CoreDNS `Corefile` ConfigMap update. Mirrors the seven methodology steps in [`../SKILL.md`](../SKILL.md). Fixtures and replay test under `../fixtures/02-dns-resolution-failure/` and `../tests/replay_02_dns.py`.

## Scenario

- **Service**: `inventory-svc` (HTTP service, calls upstream `catalog-svc.internal:8080`).
- **Infra change at 2026-04-08 09:32 UTC**: `Corefile` ConfigMap updated to add a new `.internal` forward. The change introduced a typo in the upstream resolver IP, breaking resolution for `*.internal` zones intermittently (about 30% of queries returned `SERVFAIL`).
- **Alert fires at 09:47 UTC**: `inventory_svc_error_rate > 3%`.
- **No code deploy in window**. The change surface in step 2 contains only the ConfigMap update.

## Step 1: anchor the window

- **T0**: `2026-04-08T09:47:00Z` (alert fire).
- **Tnow**: `2026-04-08T09:53:00Z` (investigation triggered).
- **Window**: `[09:32:00Z, 09:53:00Z]`.

The 15-minute lead-in catches the 09:32 ConfigMap update. Without it, the change surface would return empty and the agent would risk misclassifying as a flaky-network external failure.

## Step 2: bisect the change surface

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | (none) | | No `inventory-svc` Deployment rollouts in window. |
| Cluster / HPA | `kube-system/coredns` Corefile ConfigMap updated | `09:32:18Z` | Added `forward .internal 10.100.0.53` line; typo: should be `10.100.0.5`. |
| RBAC | (none) | | |
| ConfigMap / flags | (none) | | |
| CronJob | (none) | | |

One change in window: a cluster-level change to CoreDNS configuration. The fact that *zero* Deployment rollouts touched `inventory-svc` is itself a signal pushing away from the deploy-correlator path and toward an environmental cause.

## Step 3: classify against the four reference paths

- **OOM**: no `OOMKilled` events, memory metrics flat at baseline. No match.
- **DNS**: 47 `SERVFAIL` log lines for `catalog-svc.internal` in window, `getaddrinfo: Name or service not known` errors in client pod logs, recent change to CoreDNS configuration. Strong match.
- **Cascading-failure**: retry counts elevated but pattern is intermittent (correlates with the ~30% SERVFAIL rate), not the saturating-thread-pool shape of a true cascade. Match is weak; signature is downstream of DNS.
- **Deploy-correlator**: no `inventory-svc` Deployment rollouts in window. No match against `inventory-svc`. The CoreDNS configuration *change* is structurally a "change in window touching the failing surface", flagged as a secondary explanation.

Classification: **DNS**, triggered by a cluster-level change to CoreDNS. The agent does not need to invent a separate "config-correlator" path because the DNS path already encodes "recent change to the CoreDNS `Corefile` ConfigMap, kube-dns, `ndots`/search-domain, or a NetworkPolicy" as a confirming signal.

## Step 4: confirm with three independent signals

1. **Logs** (`fixtures/02-dns-resolution-failure/logs.jsonl`): 47 `SERVFAIL` log lines from `inventory-svc` calling `catalog-svc.internal`, plus 8 `getaddrinfo` errors in `payments-api` (a second consumer of the same zone).
2. **Metrics** (`fixtures/02-dns-resolution-failure/metrics.json`): DNS resolver error counter elevated 12x baseline. Application error rate at `inventory-svc` climbed from 0.2% to 3.4%. Memory and CPU flat.
3. **Cluster change** (`fixtures/02-dns-resolution-failure/deploys.json`): `kube-system/coredns` ConfigMap updated 15 minutes before T0, diff touches the `.internal` zone forward configuration.

A fourth signal: **traces** (`fixtures/02-dns-resolution-failure/traces.jsonl`) show failing spans with `dns.error=SERVFAIL` attribute, isolated to outbound calls to `*.internal` hostnames. Other outbound calls (e.g. to `api.stripe.com`) are unaffected. Asymmetric pattern is consistent with DNS scoped to one zone.

## Step 5: quantify blast radius

- **Users affected**: ~30% of requests that depend on `catalog-svc` fail or hit retry latency. Read-only catalog browsing degraded; checkout still works because the order-creation path does not call `catalog-svc` synchronously.
- **Surfaces affected**: all `inventory-svc` endpoints that hit `catalog-svc`, plus any other consumer of `*.internal` zones (at least `payments-api`).
- **Business impact**: catalog browsing degraded across the platform. SLO burn at 8x normal rate for read endpoints.

## Step 6: propose mitigation before root cause

1. **Revert** the `kube-system/coredns` ConfigMap to the version from `09:30:00Z`. The change is the clearly implicated single change in window, and reverting it takes one `kubectl apply` of the previous ConfigMap.
2. **Feature-flag off**: not applicable. No ConfigMap flag controls DNS resolution.
3. **Scale up**: not applicable. The failure is not capacity-bound (no `kubectl scale` / HPA ceiling / `resources.limits` change would help).
4. **Traffic-shift**: not applicable in this single-cluster topology. If the cluster had a sibling cluster on a different DNS configuration, shifting traffic there would be a viable stopgap.
5. **Manual intervention**: restarting CoreDNS pods does not help because they reload from the same broken ConfigMap. Restarting client pods does not help because they will hit the same resolver. Skip.

Recommended action: **revert the `kube-system/coredns` ConfigMap**.

## Step 7: hand off

```json
{
  "timeline": [
    {"t": "2026-04-08T09:32:18Z", "event": "kube-system/coredns ConfigMap updated (typo in .internal forward IP)"},
    {"t": "2026-04-08T09:34:00Z", "event": "First SERVFAIL log line from inventory-svc"},
    {"t": "2026-04-08T09:47:00Z", "event": "T0: alert inventory_svc_error_rate > 3% fires"},
    {"t": "2026-04-08T09:53:00Z", "event": "Tnow: investigation triggered"}
  ],
  "ranked_hypotheses": [
    {
      "path": "DNS (infra-change-triggered)",
      "confidence": "high",
      "evidence": ["47 SERVFAIL log lines in window", "DNS resolver error counter 12x baseline", "CoreDNS ConfigMap change 15min before T0 touches .internal zone", "outbound calls to non-.internal hosts unaffected"]
    }
  ],
  "mitigation_taken": null,
  "mitigation_recommended": "Revert kube-system/coredns ConfigMap to the 09:30:00Z version",
  "open_questions": [
    "Why did the ConfigMap change ship without a syntax-check or a staged rollout?",
    "Why is there no end-to-end DNS resolution health check on the cluster's internal zones?"
  ]
}
```

## Why this is the DNS reference path

- The failure shape is intermittent (not 100%), which is the signature DNS issues often present and which the OOM and deploy-correlator paths cannot explain.
- The change surface contains an infrastructure change rather than a code deploy, which forces the agent to widen its mental model beyond "what code shipped".
- The signal asymmetry (outbound calls to `*.internal` fail, calls to `*.com` work) is diagnostic for DNS scoped to a zone, and would be invisible without the trace-level attribute.

Skipping step 4 (three independent signals) is the most likely failure mode here: an agent that classifies on logs alone may over-attribute the failure to the consumer service rather than the resolver.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall