Skip to content

lidge-jun/codexclaw / plugins/codexclaw/skills/dev-devops

cxc-dev-devops

Verified

Use for infra, CI/CD, releases, repo policy, worktree cleanup and native desktop apps. Triggers: Dockerfile, K8s, IaC, SRE, Tauri, 스택 PR CI, 배포, 인프라, 쿠버네티스, 브랜치 정리, 브랜치 삭제, 워크트리 정리, 저장소 세팅, 브랜치 보호, 에이전트 PR, PR 정책, 데스크톱 앱, 메뉴 막대.

references/sre-foundations.md

View on GitHub
# SRE Foundations — SLO, Incident Response, Postmortem

Last reviewed: 2026-06-16
Applies to: Google SRE principles, 2026 observability practices
When to read: Production operations, incident response, SLO definition
Canonical owner: dev-devops §5

Cross-ref: read `platform-engineering.md` for DORA metrics and platform
engineering context before broad DevOps capability refreshes.

---

## §1 SLO/SLI Design Guide

### SLI Selection Principles

SLIs measure **user experience**, not infrastructure health.

| ❌ Wrong SLI | ✅ Right SLI |
|-------------|-------------|
| CPU utilization < 80% | Request success rate > 99.9% |
| Memory free > 2GB | p99 latency < 500ms |
| Disk IOPS < 5000 | Data freshness < 5 minutes |
| Pod restart count < 3 | Error rate < 0.1% |

### SLI by Service Type

| Service Type | Recommended SLIs |
|-------------|------------------|
| Request-driven (API) | Availability, latency (p50/p95/p99), error rate |
| Pipeline/batch | Freshness, correctness, throughput |
| Storage | Durability, availability, latency |
| Streaming | Throughput, end-to-end latency, message loss rate |

### SLO Definition

```yaml
# Example SLO definition (documentation format)
service: payments-api
slos:
  - name: availability
    sli: "successful requests / total requests"
    target: 99.9%
    window: 28 days (rolling)
    error_budget: 0.1% (≈ 40 minutes/28 days)

  - name: latency
    sli: "requests completing within 500ms / total requests"
    target: 99.0%
    window: 28 days (rolling)

  - name: error-rate
    sli: "5xx responses / total responses"
    target: "< 0.1%"
    window: 28 days (rolling)
```

### Population Definition Checklist

- [ ] Which endpoints are included? (all API routes? only customer-facing?)
- [ ] Which regions count? (global or per-region SLO?)
- [ ] How are retries handled? (count each attempt or only final result?)
- [ ] Are health checks excluded from SLI calculation?
- [ ] Are internal/admin endpoints excluded?

---

## §2 Error Budget Policy

### 3-Stage Policy Template

```
┌─────────────────────────────────────────────────┐
│ NORMAL (budget > 50%)                           │
│ → Standard release velocity                     │
│ → Routine monitoring                            │
│ → Normal on-call rotation                       │
├─────────────────────────────────────────────────┤
│ ACCELERATED BURN (budget 20-50%)                │
│ → Heightened alerting thresholds                │
│ → Reliability triage before new features        │
│ → Reduced release frequency                    │
│ → Team standup includes budget status           │
├─────────────────────────────────────────────────┤
│ EXHAUSTED (budget ≤ 0%)                         │
│ → FEATURE FREEZE (security/bugfix only)         │
│ → Next sprint: reliability work only            │
│ → Exception requires VP-level approval          │
│ → Postmortem mandatory for contributing events  │
└─────────────────────────────────────────────────┘
```

### Escalation Triggers

| Trigger | Action |
|---------|--------|
| Single incident consumes >20% budget | Mandatory postmortem + P1 action items |
| Two consecutive window misses | Structural problem; architecture review |
| Budget exhausted for 2+ windows | Executive escalation; team capacity review |

---

## §3 Burn-Rate Alerting

### Multi-Window Alert Design

Traditional threshold alerts fire too often (alert fatigue) or too late.
Burn-rate alerts use the SLO itself as the reference.

| Alert Type | Burn Rate | Time to Exhaust | Window | Action |
|-----------|-----------|-----------------|--------|--------|
| Fast burn | 14.4× | ~2 hours (of 28-day budget) | 1h long, 5m short | **Page** (wake someone up) |
| Slow burn | 6× | ~5 days | 6h long, 30m short | **Ticket** (next business day) |

### Formula

```
error_rate = errors / total_requests
burn_rate = error_rate / (1 - SLO)

Alert fires when:
  burn_rate(short_window) > threshold
  AND
  burn_rate(long_window) > threshold
```

### Prometheus Example

```yaml
# Fast burn alert (page)
- alert: PaymentsHighBurnRate
  expr: |
    (
      sum(rate(http_requests_total{service="payments",code=~"5.."}[5m]))
      / sum(rate(http_requests_total{service="payments"}[5m]))
    ) > 14.4 * (1 - 0.999)
    AND
    (
      sum(rate(http_requests_total{service="payments",code=~"5.."}[1h]))
      / sum(rate(http_requests_total{service="payments"}[1h]))
    ) > 14.4 * (1 - 0.999)
  for: 2m
  labels:
    severity: page
  annotations:
    summary: "Payments API burning error budget at 14.4x rate"
```

### Anti-Patterns

| Banned | Fix |
|--------|-----|
| Alert on every 5xx | Burn-rate multi-window |
| Single-window alert | Dual-window (short AND long) |
| No SLO reference in alert | Alert threshold derived from SLO |

---

## §4 Incident Response Playbook

### Role Definitions

| Role | Responsibilities |
|------|-----------------|
| IC (Incident Commander) | Coordinate response, delegate tasks, make decisions |
| Primary Responder | Diagnose and fix; technical execution |
| Comms Lead | Status updates to stakeholders, status page |
| Scribe | Timeline documentation, action item tracking |

### Incident Lifecycle

```
0min   Alert fires → acknowledge
5min   Declare incident → assign IC → open channel
10min  IC assigns roles → begin diagnosis
15min  First status update (S1: every 15min, S2: every 30min)
N min  Stabilize (mitigation > diagnosis during active incident)
N+M    Root cause identified → permanent fix planned
5 days Postmortem completed and shared
```

### Severity Definitions

| Severity | Impact | Response |
|----------|--------|----------|
| S1 | Full service outage, revenue impact | IC mandatory, 15min updates, all-hands |
| S2 | Major feature broken, significant user impact | IC recommended, 30min updates |
| S3 | Partial degradation, workaround available | Assignee handles, async updates |

### Key Principle

> **Mitigation first, diagnosis second** during active incidents.
> Stop the bleeding before understanding the wound.

---

## §5 Blameless Postmortem Template

```markdown
# Postmortem: [Title]

**Date**: YYYY-MM-DD
**Severity**: S1/S2/S3
**Duration**: HH:MM (detect → resolve)
**IC**: @name
**Author**: @name

## User Impact
- Affected: N% of users for M minutes
- Revenue impact: estimated $X (if applicable)
- Support tickets: N

## Timeline
| Time (UTC) | Event |
|-----------|-------|
| HH:MM | Alert fired: [alert name] |
| HH:MM | IC assigned, incident channel opened |
| HH:MM | Root cause identified: [brief] |
| HH:MM | Mitigation applied: [action] |
| HH:MM | Service fully restored |

## Detection Gap
- How could we have detected this sooner?
- Was monitoring adequate? What was missing?

## Contributing Factors (NOT "Root Cause")
1. [Factor 1]: [description]
2. [Factor 2]: [description]

## What Went Well
- [positive aspect of the response]

## What Could Be Improved
- [improvement area]

## Action Items
| Item | Owner | Due | Priority |
|------|-------|-----|----------|
| Add circuit breaker to payment gateway | @engineer | YYYY-MM-DD | P1 |
| Improve alert coverage for [gap] | @sre | YYYY-MM-DD | P2 |
| Update runbook with new failure mode | @oncall | YYYY-MM-DD | P2 |
```

### Postmortem Rules

| Rule | Detail |
|------|--------|
| Blameless | Focus on systems, not individuals |
| Timeline accuracy | Use logs/metrics, not memory |
| Action items | Specific, owned, dated, prioritized |
| Follow-through | Review action items in next team sync |
| Required for | All S1, all S2, any event consuming >20% error budget |

---

## §6 Anti-Patterns

| Banned | Symptom | Fix |
|--------|---------|-----|
| Infrastructure-only SLIs | "Server CPU is fine" while users see errors | User-experience-based SLIs |
| SLO without consequences | SLO exists on paper but nobody acts on breaches | Error budget policy with freeze gate |
| Too many SLIs (>5/service) | Dashboard overload, unclear priority | 2-4 meaningful SLIs per service |
| Page on every deviation | Alert fatigue, ignored alerts | Burn-rate multi-window alerting |
| Blame individuals | Engineers hide mistakes, no systemic improvement | Blameless postmortem culture |
| No error budget policy | SLO violations have no defined response | Define 3-stage policy |
| Postmortem without action items | Theater; same incidents repeat | Specific, owned, tracked items |
| Measuring a long-running process without proving what it is | You benchmarked the bug, not the fix | `DEVOPS-STALE-PROCESS-01` §7.1 |
| Flipping a true status bit to signal a missing one | The one honest signal now lies too | `DEVOPS-OBS-SIGNAL-01` §7.2 |

---

## §7 Evidence & Operator Signals

### §7.1 Process identity (`DEVOPS-STALE-PROCESS-01`, STRICT)

A live long-running process is not candidate evidence until its start time,
binary, and config are proven to match the build under test.

This exists because an OpenCodex train had a 100-call live canary as a GO
criterion, and the process on the expected port turned out to be PID 922, started
two days earlier: the reporter's own pre-fix proxy
(`260824_v2_32_1_hotfix_train/090_wp9_issue2472_mixed_sequence_regression.md`:21-23).
The canary was rejected as the wrong instrument BEFORE it was run, and the phase
substituted a deterministic mixed-sequence regression
(`080_wp8_freeze_verification_and_go_nogo.md`:24). The lesson is the identity
check, not a bad measurement — the measurement never happened, because the
identity check came first.

Check, in this order: process start time versus the artifact's mtime, the
resolved binary path versus the one you built, and the config the process
actually loaded versus the one on disk. A matching port and a healthy `/healthz`
prove neither.

Independently learned in this project's own history: a cli-jaw deployment proof
required comparing process start time against the installed `dist` mtime, because
a matching version string and a green health endpoint were both satisfied by a
process running the previous build.

### §7.2 Operator signals (`DEVOPS-OBS-SIGNAL-01`, DEFAULT)

When an operator surface is missing a signal, **add the missing signal**. Never
flip an already-true status bit to compensate.

The instructive case: a status command showed a green proxy line, and users in a
degraded routing state complained it was misleading. It was not — the proxy was
up and answering `/healthz`, and the reporter proved that himself by curling it.
Turning that line yellow would have made the one honest signal lie in order to
cover for one that was never emitted. The fix is a second line, not a corrupted
first one
(`260825_operator_visibility_train/020_wp3_issue2411_status_routing_visibility.md`:16-22).

The negative form of the same rule: a degraded, ineligible, or skipped verdict
that can reach an operator command must carry a message. A silent `ineligible`
is how `start`, `ensure`, and `repair` all reported success while doing nothing
(`260825_operator_visibility_train/001_current_state_inventory.md`:97-118).
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 319kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall