Skip to content

anyshift-io/sre-skills / skills/kubectl-investigator

kubectl-investigator

Unverified

Investigate a live or recent incident in a Kubernetes cluster. Anchor the window, bisect the change surface (rollouts, ConfigMaps/Secrets, RBAC, HPA/cluster changes, CronJobs), classify against four reference failure paths (OOM, DNS, cascading-failure, deploy-correlator), confirm the hypothesis with three independent signals, quantify blast radius, and propose mitigation before root cause. Use whenever an agent is asked "what is breaking in the cluster right now", "why did this pod/Deployment just page", "did the rollout cause Z", or to triage an active Kubernetes incident. Vendor-neutral by default (works with kubectl, kube-state-metrics, and whatever telemetry you have); an opt-in Anyshift integration is documented separately.

FileKindSize
FAILURE_MODES.mdmarkdown6.8 KB
README.mdmarkdown4.0 KB
SKILL.mdskill md19.4 KB
examples/01-oom-cascade.mdmarkdown7.4 KB
examples/02-dns-resolution-failure.mdmarkdown7.1 KB
examples/03-cascading-failure-retry-storm.mdmarkdown8.2 KB
examples/04-deploy-correlator-serialization.mdmarkdown7.9 KB
examples/05-outside-reference-paths-third-party-rate-limit.mdmarkdown7.8 KB
examples/06-ambiguous-t0-slow-burn.mdmarkdown7.9 KB
examples/07-blast-radius-asymmetric-revert.mdmarkdown8.1 KB
examples/08-deploy-correlator-confirmation-bias.mdmarkdown9.1 KB
examples/09-zero-changes-external-cert-expiry.mdmarkdown6.3 KB
examples/10-multi-region-asymmetry.mdmarkdown7.1 KB
examples/11-capacity-bound-organic-growth.mdmarkdown6.6 KB
fixtures/01-oom-cascade/deploys.jsonconfig789 B
fixtures/01-oom-cascade/logs.jsonlother1.3 KB
fixtures/01-oom-cascade/metrics.jsonconfig1.3 KB
fixtures/01-oom-cascade/pod_events.jsonlother1.4 KB
fixtures/01-oom-cascade/traces.jsonlother1.5 KB
fixtures/02-dns-resolution-failure/deploys.jsonconfig706 B
fixtures/02-dns-resolution-failure/logs.jsonlother2.4 KB
fixtures/02-dns-resolution-failure/metrics.jsonconfig1.3 KB
fixtures/02-dns-resolution-failure/traces.jsonlother1.4 KB
fixtures/03-cascading-failure-retry-storm/deploys.jsonconfig416 B
fixtures/03-cascading-failure-retry-storm/logs.jsonlother1.4 KB
fixtures/03-cascading-failure-retry-storm/metrics.jsonconfig1.7 KB
fixtures/03-cascading-failure-retry-storm/traces.jsonlother1.6 KB
fixtures/04-deploy-correlator-serialization/deploys.jsonconfig803 B
fixtures/04-deploy-correlator-serialization/logs.jsonlother1.4 KB
fixtures/04-deploy-correlator-serialization/metrics.jsonconfig1.0 KB
fixtures/04-deploy-correlator-serialization/traces.jsonlother1.3 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/deploys.jsonconfig390 B
fixtures/05-outside-reference-paths-third-party-rate-limit/logs.jsonlother1.6 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/metrics.jsonconfig1.3 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/traces.jsonlother1.3 KB
fixtures/06-ambiguous-t0-slow-burn/deploys.jsonconfig517 B
fixtures/06-ambiguous-t0-slow-burn/logs.jsonlother862 B
fixtures/06-ambiguous-t0-slow-burn/metrics.jsonconfig1.2 KB
fixtures/06-ambiguous-t0-slow-burn/traces.jsonlother933 B
fixtures/07-blast-radius-asymmetric-revert/deploys.jsonconfig905 B
fixtures/07-blast-radius-asymmetric-revert/logs.jsonlother1.7 KB
fixtures/07-blast-radius-asymmetric-revert/metrics.jsonconfig1.4 KB
fixtures/07-blast-radius-asymmetric-revert/traces.jsonlother1.5 KB
fixtures/08-deploy-correlator-confirmation-bias/deploys.jsonconfig812 B
fixtures/08-deploy-correlator-confirmation-bias/logs.jsonlother2.0 KB
fixtures/08-deploy-correlator-confirmation-bias/metrics.jsonconfig1.3 KB
fixtures/08-deploy-correlator-confirmation-bias/traces.jsonlother1.3 KB
fixtures/09-zero-changes-external-cert-expiry/deploys.jsonconfig145 B
fixtures/09-zero-changes-external-cert-expiry/logs.jsonlother1.2 KB
fixtures/09-zero-changes-external-cert-expiry/metrics.jsonconfig1.3 KB
fixtures/09-zero-changes-external-cert-expiry/traces.jsonlother1.4 KB
fixtures/10-multi-region-asymmetry/deploys.jsonconfig138 B
fixtures/10-multi-region-asymmetry/logs.jsonlother1.7 KB
fixtures/10-multi-region-asymmetry/metrics.jsonconfig2.0 KB
fixtures/10-multi-region-asymmetry/traces.jsonlother1.6 KB
fixtures/11-capacity-bound-organic-growth/deploys.jsonconfig139 B
fixtures/11-capacity-bound-organic-growth/logs.jsonlother948 B
fixtures/11-capacity-bound-organic-growth/metrics.jsonconfig1.2 KB
fixtures/11-capacity-bound-organic-growth/traces.jsonlother1.2 KB
tests/README.mdmarkdown4.3 KB
tests/_methodology.pyscript22.6 KB
tests/eval/README.mdmarkdown12.7 KB
tests/eval/eval_results.jsonconfig852.7 KB
tests/eval/incidents.pyscript8.6 KB
tests/eval/judge_prompt.mdmarkdown2.4 KB
tests/eval/rubric.mdmarkdown3.1 KB
tests/eval/run_eval.pyscript15.0 KB
tests/eval/run_until_pass.shscript2.1 KB
tests/replay_01_oom_cascade.pyscript3.6 KB
tests/replay_02_dns.pyscript3.3 KB
tests/replay_03_cascade.pyscript3.2 KB
tests/replay_04_deploy_correlator.pyscript3.3 KB
tests/replay_05_outside_paths.pyscript2.4 KB
tests/replay_06_ambiguous_t0.pyscript2.3 KB
tests/replay_07_blast_radius.pyscript2.4 KB
tests/replay_08_confirmation_bias.pyscript2.7 KB
tests/replay_09_cert_expiry.pyscript2.0 KB
tests/replay_10_multi_region.pyscript2.4 KB
tests/replay_11_capacity_bound.pyscript2.3 KB

examples/11-capacity-bound-organic-growth.md

View on GitHub
# Worked example 11: capacity-bound organic growth

A failure with no change, no provider issue, no internal cascade. The service has been growing organically for weeks and finally crossed its capacity threshold. The methodology should **recommend scaling, not revert**, because there is no change to revert and the dominant signal is request-rate growth. Exercises the "scale_resource as primary mitigation" branch for outside-paths classifications with growing traffic. Fixtures and replay test under `../fixtures/11-capacity-bound-organic-growth/` and `../tests/replay_11_capacity_bound.py`.

## Scenario

- **Service**: `search-svc` (search index queries).
- **Pattern**: request rate has been climbing organically for ~3 weeks as a marketing campaign drove user growth. Capacity was sized for ~400 rps; today's morning peak hit 720 rps. The service's query queue starts rejecting requests when it overflows. No deploy, no DNS, no internal cascade.
- **Alert at 10:55 UTC**: `search_svc_error_rate > 3%`.
- **Methodology must produce**: classification `outside-reference-paths`, blast-radius detector flags `request_rate_growing=true`, and the mitigation list leads with **scale_resource** instead of revert (no change to revert).

## Step 1: anchor the window

- **T0**: `2026-10-08T10:55:00Z`.
- **Tnow**: `2026-10-08T11:05:00Z`.
- **Window**: `[10:40:00Z, 11:05:00Z]`.

## Step 2: bisect the change surface

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | (none) | | |
| Cluster / HPA | (none) | | |
| RBAC | (none) | | |
| ConfigMap / flags | (none) | | |
| CronJob | (none) | | |

Zero changes in window. Same pattern as examples 09 and 10: hunt for an external or environmental cause.

## Step 3: classify against the four reference paths

- **OOM**: RSS p95 flat at ~290 MB against 512 MB limit (~57%). No `OOMKilled`. No match.
- **DNS**: no resolution errors. No match.
- **Cascading-failure**: no upstream-latency growth. No retry-storm pattern (the rejections are from the service itself, not from cascading retries). No match.
- **Deploy-correlator**: no deploy. No match.

Classification: **outside-reference-paths**. The signature is *request-rate growth* + *internal queue saturation*, which isn't one of the four reference paths.

## Step 4: confirm with three independent signals

1. **Metrics**: `request_rate_rps` climbed from 412 at 10:40 to 718 at T0 (1.7x in 15 minutes); `error_rate_pct` from 0.3% to 4.6%; `queue_depth_p99` from 12 to 184 (15x); `rss_bytes_p95` flat at ~290 MB.
2. **Logs**: `query queue full, rejecting request` warnings from `search-svc`, climbing in frequency through the window.
3. **Traces**: failing spans show `error=queue_full` at the `search-svc` hop. The failing requests never reach the search index; they are rejected at admission.

Three independent sources. The diagnostic is clean: the service is dropping load it cannot serve.

## Step 5: quantify blast radius

- **Users affected**: ~4.6% of searches at T0, climbing as traffic climbs. Successful searches still complete normally (no latency degradation on accepted requests).
- **Surfaces affected**: search query endpoint. All other endpoints healthy.
- **Business impact**: degraded search quality (some users see "search temporarily unavailable"). No silent corruption or data loss.

The `request_rate_growing` flag is `true` in the blast radius output (rate grew >= 1.5x across the window), which triggers the scaling mitigation branch.

## Step 6: propose mitigation before root cause

Because the classification is outside-paths AND `request_rate_growing` is true, the methodology's mitigation list leads differently from a change-induced incident:

1. **`kubectl scale` / raise the HPA ceiling** for `search-svc`. Request rate has climbed past current capacity headroom; the immediate fix is more replicas (`kubectl scale deployment/search-svc --replicas=N`, or raise the HPA `maxReplicas`). Acknowledged that capacity planning (forecasting, HPA tuning) is the root cause; scaling addresses the symptom.
2. **`kubectl rollout undo`**: not applicable; no change to revert.
3. **ConfigMap-flag off**: only as a temporary degradation play if scaling is delayed. Pausing low-value search features can free queue slots.
4. **Traffic-shift**: only if a secondary search cluster exists. Same caveat as 09 and 10.
5. **Manual intervention**: not applicable.

Recommended action: **raise the HPA ceiling / `kubectl scale` `search-svc` now; queue capacity planning review as a follow-up**.

## Step 7: hand off

```json
{
  "timeline": [
    {"t": "2026-10-08T10:40:00Z", "event": "request rate 412 rps (sustainable)"},
    {"t": "2026-10-08T10:48:00Z", "event": "request rate crosses 600 rps; first 'queue full' warning"},
    {"t": "2026-10-08T10:55:00Z", "event": "T0: alert search_svc_error_rate > 3% fires; request rate 692 rps"},
    {"t": "2026-10-08T11:05:00Z", "event": "Tnow: investigation triggered; request rate 718 rps, error rate 4.6%"}
  ],
  "ranked_hypotheses": [
    {
      "path": "Capacity-bound organic growth (outside reference paths)",
      "confidence": "high (on hypothesis), classified as outside-reference-paths",
      "evidence": ["request rate grew 412 -> 718 rps (1.7x) across window", "queue_depth_p99 grew 12 -> 184 (15x)", "query queue full warnings in logs", "no changes in window", "RSS / cpu / DNS / cascade signatures absent"]
    }
  ],
  "mitigation_taken": null,
  "mitigation_recommended": "Scale search-svc horizontally to absorb the new traffic level. Capacity planning review as a follow-up.",
  "escalate_to_human": true,
  "escalation_reasons": ["M1: failure classified outside the four reference paths"],
  "open_questions": [
    "Why did autoscaling not kick in? Are the autoscale thresholds tuned to absolute capacity instead of queue saturation?",
    "Is the 3-week request-rate climb visible in a forecast dashboard? If not, where would it have been caught?",
    "What is the upper bound on this service's horizontal scaling? Are downstream dependencies (search index shards) also at risk?"
  ]
}
```

## Why this is the capacity-bound reference example

- It models the case where the methodology's *default mitigation* (revert) is structurally wrong. There is no change to revert; recommending one would be a confident-looking error.
- It exercises the `request_rate_growing` blast-radius signal, which is the methodology's mechanism for detecting saturation in the absence of a discrete event.
- It surfaces a class of incidents where the underlying cause is operational (capacity planning) rather than software-defect. The methodology surfaces this honestly via outside-paths classification and scaling mitigation, instead of inventing a software hypothesis to explain the error rate.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 196kUpdated Claude CodeCodexVerifiedInstall