anyshift-io/sre-skills / skills/kubectl-investigator
kubectl-investigator
UnverifiedInvestigate a live or recent incident in a Kubernetes cluster. Anchor the window, bisect the change surface (rollouts, ConfigMaps/Secrets, RBAC, HPA/cluster changes, CronJobs), classify against four reference failure paths (OOM, DNS, cascading-failure, deploy-correlator), confirm the hypothesis with three independent signals, quantify blast radius, and propose mitigation before root cause. Use whenever an agent is asked "what is breaking in the cluster right now", "why did this pod/Deployment just page", "did the rollout cause Z", or to triage an active Kubernetes incident. Vendor-neutral by default (works with kubectl, kube-state-metrics, and whatever telemetry you have); an opt-in Anyshift integration is documented separately.
tests/eval/rubric.md
View on GitHub# Rubric: kubectl-investigator skill ablation
The rubric grades an agent's incident investigation output against the seven methodology steps. Each item is binary (0 or 1). A perfect investigation scores 7. The same rubric is applied to both the control condition (agent invoked with no methodology guidance) and the treatment condition (agent invoked with `SKILL.md` loaded), so the delta between conditions measures the value the skill adds.
The expected answers for each fixture are generated by `tests/_methodology.py`, the deterministic reference implementation. The agent under test is graded against those known-good outputs.
## The 7 items
| # | Item | Pass criterion | Source of truth |
|---|---|---|---|
| 1 | **T0 anchor** | Agent picks T0 within ±2 minutes of the alert fire time stated in the prompt. | The alert time embedded in the incident prompt. |
| 2 | **Change surface** | Agent correctly enumerates the changes in the window (count and kind). "Zero changes" is a valid answer when correct. | `bisect_change_surface()` output on the fixture. |
| 3 | **Classification** | Agent classifies the failure into the correct reference path (or correctly escalates to outside-reference-paths). | `classify()` output on the fixture. |
| 4 | **Independent signals** | Agent cites at least 3 signals drawn from independent source categories (logs, metrics, traces, change events, etc.). Two signals from the same category count as one. | `confirm_signals()` output (count of distinct sources >=3). |
| 5 | **Blast radius** | Agent quantifies blast radius across users, surfaces, and business impact. Partial credit available but the binary score requires all three. | Agent narrative must address users + surfaces + business impact. |
| 6 | **Mitigation ranking** | Agent's TOP recommended action matches the expected action: revert for code-deploy / infra-change / DNS / OOM-with-deploy; circuit_breaker for pure cascade; scale_resource for capacity-bound; escalate for outside-reference-paths. | `propose_mitigation()` first action (or escalation) on the fixture. |
| 7 | **Handoff** | Agent produces a structured handoff including timeline, ranked hypothesis(es), recommended mitigation, and open questions. | Structural check: all four components present in the output. |
## Scoring
- Per (fixture, condition, trial): score = sum of 7 items, range [0, 7].
- Per (fixture, condition): mean and stdev across trials.
- **Lift** = mean(treatment) − mean(control). Range [-7, +7]. Positive = skill helps. Near zero = skill is decorative. Negative = skill is hurting.
The eval reports per-fixture lift and aggregate lift. A skill is "valuable" when the aggregate lift is consistently positive across most fixtures.
## What the rubric does not measure
- **Narrative quality.** A correct classification dressed up in flowery prose scores the same as a correct classification stated tersely. The rubric is methodology-driven, not stylistic.
- **Speed.** No timing measurement. Both conditions get the same wall-clock budget per trial.
- **Cost.** No token accounting. The runner can add it as a side measurement if desired.
These can be added as extension rubric items in a follow-up.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |