Skip to content

anyshift-io/sre-skills / skills/kubectl-investigator

kubectl-investigator

Unverified

Investigate a live or recent incident in a Kubernetes cluster. Anchor the window, bisect the change surface (rollouts, ConfigMaps/Secrets, RBAC, HPA/cluster changes, CronJobs), classify against four reference failure paths (OOM, DNS, cascading-failure, deploy-correlator), confirm the hypothesis with three independent signals, quantify blast radius, and propose mitigation before root cause. Use whenever an agent is asked "what is breaking in the cluster right now", "why did this pod/Deployment just page", "did the rollout cause Z", or to triage an active Kubernetes incident. Vendor-neutral by default (works with kubectl, kube-state-metrics, and whatever telemetry you have); an opt-in Anyshift integration is documented separately.

FileKindSize
FAILURE_MODES.mdmarkdown6.8 KB
README.mdmarkdown4.0 KB
SKILL.mdskill md19.4 KB
examples/01-oom-cascade.mdmarkdown7.4 KB
examples/02-dns-resolution-failure.mdmarkdown7.1 KB
examples/03-cascading-failure-retry-storm.mdmarkdown8.2 KB
examples/04-deploy-correlator-serialization.mdmarkdown7.9 KB
examples/05-outside-reference-paths-third-party-rate-limit.mdmarkdown7.8 KB
examples/06-ambiguous-t0-slow-burn.mdmarkdown7.9 KB
examples/07-blast-radius-asymmetric-revert.mdmarkdown8.1 KB
examples/08-deploy-correlator-confirmation-bias.mdmarkdown9.1 KB
examples/09-zero-changes-external-cert-expiry.mdmarkdown6.3 KB
examples/10-multi-region-asymmetry.mdmarkdown7.1 KB
examples/11-capacity-bound-organic-growth.mdmarkdown6.6 KB
fixtures/01-oom-cascade/deploys.jsonconfig789 B
fixtures/01-oom-cascade/logs.jsonlother1.3 KB
fixtures/01-oom-cascade/metrics.jsonconfig1.3 KB
fixtures/01-oom-cascade/pod_events.jsonlother1.4 KB
fixtures/01-oom-cascade/traces.jsonlother1.5 KB
fixtures/02-dns-resolution-failure/deploys.jsonconfig706 B
fixtures/02-dns-resolution-failure/logs.jsonlother2.4 KB
fixtures/02-dns-resolution-failure/metrics.jsonconfig1.3 KB
fixtures/02-dns-resolution-failure/traces.jsonlother1.4 KB
fixtures/03-cascading-failure-retry-storm/deploys.jsonconfig416 B
fixtures/03-cascading-failure-retry-storm/logs.jsonlother1.4 KB
fixtures/03-cascading-failure-retry-storm/metrics.jsonconfig1.7 KB
fixtures/03-cascading-failure-retry-storm/traces.jsonlother1.6 KB
fixtures/04-deploy-correlator-serialization/deploys.jsonconfig803 B
fixtures/04-deploy-correlator-serialization/logs.jsonlother1.4 KB
fixtures/04-deploy-correlator-serialization/metrics.jsonconfig1.0 KB
fixtures/04-deploy-correlator-serialization/traces.jsonlother1.3 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/deploys.jsonconfig390 B
fixtures/05-outside-reference-paths-third-party-rate-limit/logs.jsonlother1.6 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/metrics.jsonconfig1.3 KB
fixtures/05-outside-reference-paths-third-party-rate-limit/traces.jsonlother1.3 KB
fixtures/06-ambiguous-t0-slow-burn/deploys.jsonconfig517 B
fixtures/06-ambiguous-t0-slow-burn/logs.jsonlother862 B
fixtures/06-ambiguous-t0-slow-burn/metrics.jsonconfig1.2 KB
fixtures/06-ambiguous-t0-slow-burn/traces.jsonlother933 B
fixtures/07-blast-radius-asymmetric-revert/deploys.jsonconfig905 B
fixtures/07-blast-radius-asymmetric-revert/logs.jsonlother1.7 KB
fixtures/07-blast-radius-asymmetric-revert/metrics.jsonconfig1.4 KB
fixtures/07-blast-radius-asymmetric-revert/traces.jsonlother1.5 KB
fixtures/08-deploy-correlator-confirmation-bias/deploys.jsonconfig812 B
fixtures/08-deploy-correlator-confirmation-bias/logs.jsonlother2.0 KB
fixtures/08-deploy-correlator-confirmation-bias/metrics.jsonconfig1.3 KB
fixtures/08-deploy-correlator-confirmation-bias/traces.jsonlother1.3 KB
fixtures/09-zero-changes-external-cert-expiry/deploys.jsonconfig145 B
fixtures/09-zero-changes-external-cert-expiry/logs.jsonlother1.2 KB
fixtures/09-zero-changes-external-cert-expiry/metrics.jsonconfig1.3 KB
fixtures/09-zero-changes-external-cert-expiry/traces.jsonlother1.4 KB
fixtures/10-multi-region-asymmetry/deploys.jsonconfig138 B
fixtures/10-multi-region-asymmetry/logs.jsonlother1.7 KB
fixtures/10-multi-region-asymmetry/metrics.jsonconfig2.0 KB
fixtures/10-multi-region-asymmetry/traces.jsonlother1.6 KB
fixtures/11-capacity-bound-organic-growth/deploys.jsonconfig139 B
fixtures/11-capacity-bound-organic-growth/logs.jsonlother948 B
fixtures/11-capacity-bound-organic-growth/metrics.jsonconfig1.2 KB
fixtures/11-capacity-bound-organic-growth/traces.jsonlother1.2 KB
tests/README.mdmarkdown4.3 KB
tests/_methodology.pyscript22.6 KB
tests/eval/README.mdmarkdown12.7 KB
tests/eval/eval_results.jsonconfig852.7 KB
tests/eval/incidents.pyscript8.6 KB
tests/eval/judge_prompt.mdmarkdown2.4 KB
tests/eval/rubric.mdmarkdown3.1 KB
tests/eval/run_eval.pyscript15.0 KB
tests/eval/run_until_pass.shscript2.1 KB
tests/replay_01_oom_cascade.pyscript3.6 KB
tests/replay_02_dns.pyscript3.3 KB
tests/replay_03_cascade.pyscript3.2 KB
tests/replay_04_deploy_correlator.pyscript3.3 KB
tests/replay_05_outside_paths.pyscript2.4 KB
tests/replay_06_ambiguous_t0.pyscript2.3 KB
tests/replay_07_blast_radius.pyscript2.4 KB
tests/replay_08_confirmation_bias.pyscript2.7 KB
tests/replay_09_cert_expiry.pyscript2.0 KB
tests/replay_10_multi_region.pyscript2.4 KB
tests/replay_11_capacity_bound.pyscript2.3 KB

tests/README.md

View on GitHub
# Replay tests for `kubectl-investigator`

Stdlib-only Python tests that exercise the methodology in [`../SKILL.md`](../SKILL.md) against committed fixtures. No external credentials required.

## Running the tests

From the skill directory (`skills/kubectl-investigator/`):

```bash
for t in tests/replay_*.py; do python "$t" || exit 1; done
```

Each test prints `PASS` or `FAIL` and exits with the appropriate code. The current suite has 11 tests covering the four reference paths, the FAILURE_MODES escalation rules (M1, M2, M3, M4), and edge cases (zero changes in window, multi-region asymmetry, capacity saturation), totalling 99 assertions. Wire them into CI as plain `python` invocations.

## What the tests assert

Each replay test loads the fixtures for one worked example, runs the reference methodology (`_methodology.py`) end-to-end against them, and asserts:

- The investigation window is anchored correctly (step 1).
- The change surface contains the expected change(s) (step 2).
- The failure is classified into the right reference path (step 3).
- At least three independent signal sources are confirmed (step 4).
- The blast-radius numbers fall in the expected range (step 5).
- The recommended mitigation is correct, with the right action ordering (step 6).
- The handoff payload is well-formed and the escalation flag matches expectations (step 7).

A test fails when the methodology regresses on any of these. Treat a failed replay test as a regression in `SKILL.md` or in the reference implementation, not a test bug.

## Fixture schema

Each example has its own fixture directory under `../fixtures/<example-slug>/`. Files are committed JSON / JSONL with no external dependencies.

| File | Format | Purpose |
|---|---|---|
| `deploys.json` | JSON | Single object with `deploys`, `infra_changes` (cluster / node-pool / HPA / manifest edits), `rbac_changes` (Role / RoleBinding / ServiceAccount edits), `feature_flags`, `scheduled_jobs` (CronJobs) arrays. Each event has a timestamp field (`deployed_at`, `changed_at`, `flipped_at`, `ran_at`) and a `diff_summary` string. |
| `pod_events.jsonl` | JSONL (optional) | Kubernetes pod / kubelet events. One JSON object per line with `t`, `pod`, `reason` (e.g. `OOMKilled`, `BackOff`, `Unhealthy`), `message`. Skip if the example is not pod-level. |
| `metrics.json` | JSON | Time-series snapshot. Fields: `service`, `memory_limit_bytes`, `samples[]`. Each sample has `t`, `rss_bytes_p95`, `error_rate_pct`, `request_rate_rps`, `gateway_retry_rate_rps`, and any path-specific counter (e.g. `dns_resolver_errors_rps`). |
| `logs.jsonl` | JSONL (optional) | Application or system logs. One JSON object per line with `t`, `level`, `service`, `msg`, and any path-specific fields. |
| `traces.jsonl` | JSONL (optional) | Distributed traces. One JSON object per line with `t`, `trace_id`, `request_id`, `spans[]`, `outcome`, `retries`. |

The reference implementation (`_methodology.py`) gracefully handles missing optional files (returns empty lists).

## Adding a new replay test

When you contribute a new worked example to the skill:

1. Drop fixtures under `../fixtures/<example-slug>/` following the schema above.
2. Add `replay_NN_<example-slug>.py` in this directory, modeled on the existing two.
3. Pick assertions that cover the seven methodology steps. Existing tests range from 6 to 13 assertions. New tests typically want 7 to 15.
4. Run locally, commit, and reference the test in the example's markdown narrative.

A new test that does not exercise at least one independent signal source the existing tests do not exercise will fail review. The point of the replay corpus is breadth.

## Why stdlib only

Skills get adopted when they run anywhere with zero setup. A `pip install` is an adoption tax. The reference implementation uses only `datetime`, `json`, `pathlib`, `dataclasses`, and `typing`. If a future test requires a third-party dependency (e.g. `pytest`), that's a signal the methodology is leaking implementation detail.

## Future: agent-shaped tests

The reference implementation in `_methodology.py` is a deterministic stand-in. A natural follow-up is to run the same fixtures through an actual LLM agent loaded with `SKILL.md` and assert the agent produces the same classification + mitigation. That work is out of scope for the first reference example; contributions welcome.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 196kUpdated Claude CodeCodexVerifiedInstall