Skip to content

anyshift-io/sre-skills / skills/kubectl-investigator

kubectl-investigator

Unverified

Investigate a live or recent incident in a Kubernetes cluster. Anchor the window, bisect the change surface (rollouts, ConfigMaps/Secrets, RBAC, HPA/cluster changes, CronJobs), classify against four reference failure paths (OOM, DNS, cascading-failure, deploy-correlator), confirm the hypothesis with three independent signals, quantify blast radius, and propose mitigation before root cause. Use whenever an agent is asked "what is breaking in the cluster right now", "why did this pod/Deployment just page", "did the rollout cause Z", or to triage an active Kubernetes incident. Vendor-neutral by default (works with kubectl, kube-state-metrics, and whatever telemetry you have); an opt-in Anyshift integration is documented separately.

FileKindSize
FAILURE_MODES.mdmarkdown6.8 KB
README.mdmarkdown4.0 KB
SKILL.mdskill md19.4 KB
examples/01-oom-cascade.mdmarkdown7.4 KB
examples/02-dns-resolution-failure.mdmarkdown7.1 KB
examples/03-cascading-failure-retry-storm.mdmarkdown8.2 KB
examples/04-deploy-correlator-serialization.mdmarkdown7.9 KB
examples/05-outside-reference-paths-third-party-rate-limit.mdmarkdown7.8 KB
examples/06-ambiguous-t0-slow-burn.mdmarkdown7.9 KB
examples/07-blast-radius-asymmetric-revert.mdmarkdown8.1 KB
examples/08-deploy-correlator-confirmation-bias.mdmarkdown9.1 KB
examples/09-zero-changes-external-cert-expiry.mdmarkdown6.3 KB
examples/10-multi-region-asymmetry.mdmarkdown7.1 KB
examples/11-capacity-bound-organic-growth.mdmarkdown6.6 KB
fixtures/01-oom-cascade/deploys.jsonconfig789 B
fixtures/01-oom-cascade/logs.jsonlother1.3 KB
fixtures/01-oom-cascade/metrics.jsonconfig1.3 KB
fixtures/01-oom-cascade/pod_events.jsonlother1.4 KB
fixtures/01-oom-cascade/traces.jsonlother1.5 KB
fixtures/02-dns-resolution-failure/deploys.jsonconfig706 B
fixtures/02-dns-resolution-failure/logs.jsonlother2.4 KB
fixtures/02-dns-resolution-failure/metrics.jsonconfig1.3 KB
fixtures/02-dns-resolution-failure/traces.jsonlother1.4 KB
fixtures/03-cascading-failure-retry-storm/deploys.jsonconfig416 B
fixtures/03-cascading-failure-retry-storm/logs.jsonlother1.4 KB
fixtures/03-cascading-failure-retry-storm/metrics.jsonconfig1.7 KB
fixtures/03-cascading-failure-retry-storm/traces.jsonlother1.6 KB
fixtures/04-deploy-correlator-serialization/deploys.jsonconfig803 B
fixtures/04-deploy-correlator-serialization/logs.jsonlother1.4 KB
fixtures/04-deploy-correlator-serialization/metrics.jsonconfig1.0 KB
fixtures/04-deploy-correlator-serialization/traces.jsonlother1.3 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/deploys.jsonconfig390 B
fixtures/05-outside-reference-paths-third-party-rate-limit/logs.jsonlother1.6 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/metrics.jsonconfig1.3 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/traces.jsonlother1.3 KB
fixtures/06-ambiguous-t0-slow-burn/deploys.jsonconfig517 B
fixtures/06-ambiguous-t0-slow-burn/logs.jsonlother862 B
fixtures/06-ambiguous-t0-slow-burn/metrics.jsonconfig1.2 KB
fixtures/06-ambiguous-t0-slow-burn/traces.jsonlother933 B
fixtures/07-blast-radius-asymmetric-revert/deploys.jsonconfig905 B
fixtures/07-blast-radius-asymmetric-revert/logs.jsonlother1.7 KB
fixtures/07-blast-radius-asymmetric-revert/metrics.jsonconfig1.4 KB
fixtures/07-blast-radius-asymmetric-revert/traces.jsonlother1.5 KB
fixtures/08-deploy-correlator-confirmation-bias/deploys.jsonconfig812 B
fixtures/08-deploy-correlator-confirmation-bias/logs.jsonlother2.0 KB
fixtures/08-deploy-correlator-confirmation-bias/metrics.jsonconfig1.3 KB
fixtures/08-deploy-correlator-confirmation-bias/traces.jsonlother1.3 KB
fixtures/09-zero-changes-external-cert-expiry/deploys.jsonconfig145 B
fixtures/09-zero-changes-external-cert-expiry/logs.jsonlother1.2 KB
fixtures/09-zero-changes-external-cert-expiry/metrics.jsonconfig1.3 KB
fixtures/09-zero-changes-external-cert-expiry/traces.jsonlother1.4 KB
fixtures/10-multi-region-asymmetry/deploys.jsonconfig138 B
fixtures/10-multi-region-asymmetry/logs.jsonlother1.7 KB
fixtures/10-multi-region-asymmetry/metrics.jsonconfig2.0 KB
fixtures/10-multi-region-asymmetry/traces.jsonlother1.6 KB
fixtures/11-capacity-bound-organic-growth/deploys.jsonconfig139 B
fixtures/11-capacity-bound-organic-growth/logs.jsonlother948 B
fixtures/11-capacity-bound-organic-growth/metrics.jsonconfig1.2 KB
fixtures/11-capacity-bound-organic-growth/traces.jsonlother1.2 KB
tests/README.mdmarkdown4.3 KB
tests/_methodology.pyscript22.6 KB
tests/eval/README.mdmarkdown12.7 KB
tests/eval/eval_results.jsonconfig852.7 KB
tests/eval/incidents.pyscript8.6 KB
tests/eval/judge_prompt.mdmarkdown2.4 KB
tests/eval/rubric.mdmarkdown3.1 KB
tests/eval/run_eval.pyscript15.0 KB
tests/eval/run_until_pass.shscript2.1 KB
tests/replay_01_oom_cascade.pyscript3.6 KB
tests/replay_02_dns.pyscript3.3 KB
tests/replay_03_cascade.pyscript3.2 KB
tests/replay_04_deploy_correlator.pyscript3.3 KB
tests/replay_05_outside_paths.pyscript2.4 KB
tests/replay_06_ambiguous_t0.pyscript2.3 KB
tests/replay_07_blast_radius.pyscript2.4 KB
tests/replay_08_confirmation_bias.pyscript2.7 KB
tests/replay_09_cert_expiry.pyscript2.0 KB
tests/replay_10_multi_region.pyscript2.4 KB
tests/replay_11_capacity_bound.pyscript2.3 KB

examples/09-zero-changes-external-cert-expiry.md

View on GitHub
# Worked example 9: zero changes in window (external TLS certificate expiry)

A failure where the change-surface bisection returns empty and the actual cause is external: a partner's TLS certificate expired. Tests the methodology's pivot from "find the change" to "consider external causes" when step 2 returns zero. Fixtures and replay test under `../fixtures/09-zero-changes-external-cert-expiry/` and `../tests/replay_09_cert_expiry.py`.

## Scenario

- **Service**: `webhook-receiver`, a Kubernetes Deployment whose pods call `partner.example.com` to register webhook subscriptions.
- **Failure**: at 2026-08-20 03:00 UTC, the partner's TLS certificate expired. Outbound calls now fail with `tls: certificate has expired`. The partner has not yet rotated the cert.
- **Alert at 03:18 UTC**: `webhook_receiver_error_rate > 4%`.
- **No changes anywhere on our side**. The cert expiry happens at midnight UTC for whatever timezone the partner is in, with no announcement.
- **Methodology must produce**: classification `outside-reference-paths`, the change-surface bisection should return zero, and the methodology should surface the "zero changes → external cause" pivot in its reasoning. Escalation with M1.

## Step 1: anchor the window

- **T0**: `2026-08-20T03:18:00Z` (alert fire).
- **Tnow**: `2026-08-20T03:25:00Z`.
- **Window**: `[03:03:00Z, 03:25:00Z]`.

## Step 2: bisect the change surface

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | (none) | | |
| Cluster / HPA | (none) | | |
| RBAC | (none) | | |
| ConfigMap / flags | (none) | | |
| CronJob | (none) | | |

**Zero changes in window.** Per SKILL.md step 2: "treat as a strong signal in itself: the failure is likely external (upstream provider, certificate expiry, DNS, capacity drift) rather than a self-inflicted regression."

The methodology must explicitly enumerate this and pivot to external-cause hypotheses for step 3.

## Step 3: classify against the four reference paths

- **OOM**: RSS p95 flat. No `OOMKilled`. No match.
- **DNS**: `partner.example.com` resolves successfully (`dns_target` attribute on traces shows valid IPs). The handshake fails *after* DNS, not at DNS. No DNS-classification match.
- **Cascading-failure**: no internal upstream-latency growth, no retry-storm signature beyond a modest retry-budget increase. No match.
- **Deploy-correlator**: no deploy. No match.

Classification: **outside-reference-paths**. The failure shape (TLS handshake error, `certificate has expired` in error message) points to a third-party cert issue.

## Step 4: confirm with three independent signals

1. **Logs**: `tls: certificate has expired` errors from `webhook-receiver` calling `partner.example.com`, starting 03:01:14Z (just after midnight UTC).
2. **Traces**: failing spans terminate at the `webhook-receiver` → `partner` hop with `tls.error=certificate_expired`. The `dns_target` attribute resolves successfully; the failure is post-DNS.
3. **Metrics**: error rate climbs from 0.2% baseline to 6.8% as queued webhook deliveries fail; internal RSS flat; DNS resolver error counter flat.

Three independent signal sources. Hypothesis (external partner cert expiry) is high-confidence on the evidence even though the classification is outside-paths.

## Step 5: quantify blast radius

- **Users affected**: every webhook subscription update to this partner. New subscriptions queue; existing connections to other partners work fine.
- **Surfaces affected**: only the partner-specific webhook path.
- **Business impact**: integration with this partner degraded. Other partners unaffected.

## Step 6: propose mitigation before root cause

Outside-paths classification means the methodology constrains the mitigation set:

1. **`kubectl rollout undo`**: not applicable; no rollout or change to revert.
2. **Feature-flag off** the partner integration if a flag exists, surfacing a "partner integration paused" message to users. Allowed without escalation.
3. **Traffic-shift** to a redundant partner endpoint if one is configured. Verify the endpoint actually bypasses the expired cert.
4. **Scale**: not applicable.
5. **Manual intervention**: not applicable.

The actual remediation requires the *partner* to rotate the cert. We can only escalate and pause the integration on our side.

Recommended next action: **escalate to a human with the cert-expiry hypothesis and the partner's incident contact details**.

## Step 7: hand off

```json
{
  "timeline": [
    {"t": "2026-08-20T03:01:14Z", "event": "First tls: certificate has expired error from partner.example.com"},
    {"t": "2026-08-20T03:18:00Z", "event": "T0: alert webhook_receiver_error_rate > 4% fires"},
    {"t": "2026-08-20T03:25:00Z", "event": "Tnow: investigation triggered"}
  ],
  "ranked_hypotheses": [
    {
      "path": "External: partner TLS certificate expired (outside reference paths)",
      "confidence": "high (on hypothesis), classified as outside-reference-paths",
      "evidence": ["tls: certificate has expired in logs", "TLS error post-DNS in traces", "zero changes on our side in window", "partner.example.com is the consistent failing target"]
    }
  ],
  "mitigation_taken": null,
  "mitigation_recommended": "Escalate. Pause partner integration via feature flag if available. Contact partner's incident channel for cert rotation ETA.",
  "escalate_to_human": true,
  "escalation_reasons": ["M1: failure classified outside the four reference paths"],
  "open_questions": [
    "Does the partner have a cert-monitoring SLA they violated?",
    "Should we monitor partner cert expiry proactively (out-of-band check that warns 14 days before expiry)?",
    "Do we have a redundant partner endpoint with an independent cert chain?"
  ]
}
```

## Why this is the zero-changes reference example

- It validates SKILL.md step 2's claim that "zero changes is itself a strong signal." Without that pivot, an agent that always hunts for an internal cause would waste investigation time and possibly invent a phantom hypothesis.
- It separates *DNS path* from *TLS issue*: both involve failing outbound calls, but the diagnostic shape is different (DNS fails at resolution; TLS fails after resolution). The example exercises the methodology's discipline on this distinction.
- It models the case where remediation is fundamentally *out of our control*. The methodology's job in that case is to surface a clear hypothesis and stop, not to invent action.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall