Skip to content

JPeetz/agent-skills / skills/devops-infrastructure/observability-engineering

observability-engineering

Verified

Use this skill when setting up monitoring, adding observability, or instrumenting with OpenTelemetry — also when creating SLOs and error budgets, designing alerts, aggregating logs, tracing distributed requests, or building Grafana/Prometheus dashboards. Implements vendor-neutral telemetry with structured incident response and alert-fatigue control.

FileKindSize
CHANGELOG.mdmarkdown2.8 KB
LICENSEother1.0 KB
SKILL.mdskill md37.1 KB
evals/evals.jsonconfig4.0 KB
references/incident-response.mdmarkdown22.8 KB
references/otel-instrumentation-guide.mdmarkdown19.3 KB
references/sli-slo-cookbook.mdmarkdown16.2 KB
scripts/generate-slo-dashboard.shexecutablescript14.6 KB
scripts/validate-observability-stack.shexecutablescript12.5 KB

references/sli-slo-cookbook.md

View on GitHub
# SLI / SLO Cookbook

SLI definition patterns, error budget formulas, and burn rate alert configurations for production observability.

---

## Table of Contents

1. [SLI Patterns](#1-sli-patterns)
2. [Error Budget Mechanics](#2-error-budget-mechanics)
3. [Burn Rate Alert Configurations](#3-burn-rate-alert-configurations)
4. [SLO Dashboard Design](#4-slo-dashboard-design)
5. [Multi-Service SLO Strategies](#5-multi-service-slo-strategies)
6. [SLO Review Cadence](#6-slo-review-cadence)

---

## 1. SLI Patterns

### 1.1 Common SLI Types

| SLI Type | What It Measures | Good Count | Bad Count | Query Pattern |
|----------|-----------------|------------|-----------|---------------|
| **Availability** | Success rate | HTTP 2xx/3xx/4xx | HTTP 5xx, timeouts | `http_requests{status!~"5.."} / http_requests_total` |
| **Latency** | Speed compliance | Duration ≤ threshold | Duration > threshold | `http_request_duration_seconds_bucket{le="0.3"} / ..._count` |
| **Freshness** | Data timeliness | Updated within window | Older than window | `data_timestamp_seconds - time() ≥ -300` |
| **Coverage** | Data completeness | Processed data | Dropped data | `events_processed / events_ingested` |
| **Throughput** | Processing rate | Rate ≥ minimum | Rate < minimum | `rate(events_processed[5m])` |
| **Durability** | Data persistence | Acknowledged writes | Lost data | `1 - (lost_events / total_events)` |
| **Correctness** | Data accuracy | Correct outputs | Incorrect outputs | `correct_events / total_events` |

### 1.2 Availability SLI Patterns

#### Standard Availability

```
SLI = sum(rate(http_requests_total{status!~"5.."}[$WINDOW]))
    / sum(rate(http_requests_total[$WINDOW]))

Good:   200, 201, 301, 302, 400, 401, 403, 404, 422, 429
Bad:    500, 502, 503, 504, timeouts, connection refused
```

**Grafana / PromQL:**

```promql
# Availability over 30 days
sum(rate(http_requests_total{status!~"5.."}[30d]))
/
sum(rate(http_requests_total[30d]))
```

#### Degraded Availability

Some services consider 4xx client errors as "user mistakes" and exclude them from the SLO:

```promql
sum(rate(http_requests_total{status=~"2..|3.."}[30d]))
/
sum(rate(http_requests_total{status!~"4.."}[30d]))
```

#### Batch Job Availability

```promql
# Job success ratio (batch/worker)
sum(rate(job_completed_total{status="success"}[7d]))
/
sum(rate(job_completed_total[7d]))
```

### 1.3 Latency SLI Patterns

#### Request Latency

```
SLI = count of requests ≤ threshold / total requests

Examples:
  Critical:  p99 < 100ms   → SLI threshold: 100ms
  Normal:    p99 < 500ms   → SLI threshold: 500ms
  Relaxed:   p99 < 2000ms  → SLI threshold: 2s
```

```promql
# Latency SLI for <300ms threshold (using histogram)
sum(rate(http_request_duration_seconds_bucket{le="0.3"}[30d]))
/
sum(rate(http_request_duration_seconds_count[30d]))
```

#### Multi-Tier Latency SLI

Different endpoints have different latency expectations:

```promql
# Per-route latency SLI
(
  sum(rate(http_request_duration_seconds_bucket{le="0.1", endpoint=~"/healthz|/readyz"}[30d]))
  +
  sum(rate(http_request_duration_seconds_bucket{le="0.5", endpoint="/api/orders"}[30d]))
  +
  sum(rate(http_request_duration_seconds_bucket{le="2", endpoint="/api/reports"}[30d]))
)
/
sum(rate(http_request_duration_seconds_count[30d]))
```

### 1.4 Freshness SLI (Data Pipelines)

```promql
# If data events carry a timestamp:
#   freshness_seconds = time() - event_timestamp_seconds
# SLI: what % of data is within freshness window?

# Using a metric that reports max staleness
(
  count by (dataset) (
    max_over_time(data_staleness_seconds[5m]) < 300
  )
)
/
count by (dataset) (data_staleness_seconds)
```

### 1.5 Defining SLIs for Different Service Types

| Service Type | Primary SLI | Secondary SLI | Example Threshold |
|-------------|------------|---------------|-------------------|
| **API Gateway** | Availability | Latency (p99) | 99.95%, <200ms |
| **Payment Service** | Availability | Latency (p95) | 99.99%, <500ms |
| **Batch Processor** | Success rate | Freshness | 99.9%, <5min |
| **Database** | Query success | Latency (p99) | 99.99%, <100ms |
| **CDN / Static** | Availability | Latency | 99.99%, <50ms |
| **Message Queue** | Publish success | Consumer lag | 99.99%, <10s lag |
| **Data Lake** | Ingestion rate | Freshness | 99%, <1hr |

---

## 2. Error Budget Mechanics

### 2.1 Error Budget Calculation

```
Error Budget = 1 - SLO Target
Budget Consumed = (1 - observed_SLI) / (1 - SLO_target)
Budget Remaining = 1 - Budget Consumed

Example — 99.9% SLO over 30 days:
  Total requests:    10,000,000
  Error budget:            10,000  (0.1%)
  If 5,000 errors in 30 days:
    Budget consumed: 5,000 / 10,000 = 50%
    Budget remaining: 50%
```

### 2.2 Allowed Downtime Table

| SLO | Per Day | Per Week | Per 30 Days | Per Quarter |
|-----|---------|----------|-------------|-------------|
| **99.9%** ("three nines") | 86.4s | 10m 5s | 43m 12s | 2h 10m |
| **99.95%** | 43.2s | 5m 2s | 21m 36s | 1h 5m |
| **99.99%** ("four nines") | 8.6s | 1m 0s | 4m 19s | 13m 0s |
| **99.999%** ("five nines") | 0.9s | 6s | 25.9s | 1m 18s |

### 2.3 Error Budget Burn Rate

```
Burn Rate = (1 - observed_SLI) / (1 - SLO_target)

Burn rate of 1:    Consuming budget exactly at the allowed rate
Burn rate of 10:   Consuming budget 10x faster than allowed
Burn rate of 14.4: Will exhaust 30-day budget in ~2 days (48h)
Burn rate of 36:   Will exhaust 30-day budget in ~1 day (20h)
Burn rate of 720:  Will exhaust 30-day budget in ~1 hour
```

### 2.4 Burn Rate to Exhaustion Table (30-day window)

| Burn Rate | Time to exhaust 30d budget | Alert Severity |
|-----------|---------------------------|----------------|
| 1x | 30 days | No alert |
| 2x | 15 days | Info |
| 3x | 10 days | Warning (slow burn) |
| 6x | 5 days | Warning |
| 10x | 3 days | High |
| 14.4x | ~2 days (48h) | Critical |
| 36x | ~20 hours | Critical |
| 144x | ~5 hours | Critical |
| 720x | ~1 hour | Critical |

---

## 3. Burn Rate Alert Configurations

### 3.1 Standard Multi-Window Burn Rate Alerts

Based on Google SRE Workbook recommendation: two windows per burn rate (short window + long window).

```yaml
# PrometheusRule: SLO Alerting
# Configures multi-window, multi-burn-rate alerts for a 99.9% SLO
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: slo-http-requests
  labels:
    prometheus: k8s
    role: alert-rules
spec:
  groups:
    - name: slo-rules
      interval: 30s
      rules:
        # ── Recording Rules ──
        - record: sli:http_requests:error_ratio_1m
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[1m]))
            /
            sum(rate(http_requests_total[1m]))

        - record: sli:http_requests:error_ratio_5m
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))

        - record: sli:http_requests:error_ratio_30m
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[30m]))
            /
            sum(rate(http_requests_total[30m]))

        - record: sli:http_requests:error_ratio_6h
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[6h]))
            /
            sum(rate(http_requests_total[6h]))

        - record: sli:http_requests:error_budget_remaining
          expr: 1 - (sli:http_requests:error_ratio_30m / 0.001)

        # ── Burn Rate Alerts ──
        # Critical: budget burning 14.4x (1h window, 5m short window)
        - alert: SLOBurnRateCritical
          expr: |
            sli:http_requests:error_ratio_1m > (0.001 * 14.4)
            and
            sli:http_requests:error_ratio_5m > (0.001 * 14.4)
          for: 2m
          labels:
            severity: critical
            slo: "99.9%"
          annotations:
            summary: "SLO error budget burning at critical rate (14.4x)"
            description: |
              Error ratio over 1m/5m exceeds 14.4x the 99.9% SLO budget rate.
              Budget will be exhausted in ~48 hours at this rate.
            runbook: "https://runbooks.example.com/slo-burn-critical.md"

        # Warning: budget burning 3x (6h window, 30m short window)
        - alert: SLOBurnRateWarning
          expr: |
            sli:http_requests:error_ratio_30m > (0.001 * 3)
            and
            sli:http_requests:error_ratio_6h > (0.001 * 3)
          for: 5m
          labels:
            severity: warning
            slo: "99.9%"
          annotations:
            summary: "SLO error budget burning at warning rate (3x)"
            description: |
              Error ratio over 30m/6h exceeds 3x the 99.9% SLO budget rate.
              Under 10% error budget remaining.
            runbook: "https://runbooks.example.com/slo-burn-warning.md"

        # Info: budget below 50%
        - alert: SLOBudgetLow
          expr: sli:http_requests:error_budget_remaining < 0.5
          for: 5m
          labels:
            severity: info
          annotations:
            summary: "Error budget below 50%"
            description: "Error budget remaining: {{ $value | humanizePercentage }}"
```

### 3.2 Burn Rate Thresholds by SLO Target

```yaml
# Burn rate thresholds for common SLO targets
# Format: burn_rate_threshold (windows: short, long)

# For 99.9% SLO (error budget = 0.001):
#   Fast burn threshold:  0.001 * 14.4 = 0.0144 (1.44% error rate)
#   Slow burn threshold:  0.001 * 3    = 0.003  (0.3% error rate)

# For 99.99% SLO (error budget = 0.0001):
#   Fast burn threshold:  0.0001 * 14.4 = 0.00144 (0.144% error rate)
#   Slow burn threshold:  0.0001 * 3    = 0.0003  (0.03% error rate)

# For 99.5% SLO (error budget = 0.005):
#   Fast burn threshold:  0.005 * 14.4 = 0.072 (7.2% error rate)
#   Slow burn threshold:  0.005 * 3    = 0.015 (1.5% error rate)
```

### 3.3 Burn Rate Alert Calculator (CLI)

```bash
# Calculate burn rate thresholds for a given SLO target
calculate_slo_thresholds() {
    local slo_target=$1
    local budget
    budget=$(echo "scale=10; 1 - ($slo_target / 100)" | bc)

    echo "SLO Target:             ${slo_target}%"
    echo "Error Budget:           $budget"
    echo ""
    echo "Fast Burn (14.4x):      $(echo "scale=6; $budget * 14.4 * 100" | bc)% error rate"
    echo "  Window: 1m + 5m, for: 2m"
    echo "Slow Burn (3x):         $(echo "scale=6; $budget * 3 * 100" | bc)% error rate"
    echo "  Window: 30m + 6h, for: 5m"
    echo ""
    echo "Time to exhaust budget at 14.4x: ~48 hours"
    echo "Time to exhaust budget at 3x:    ~10 days"
    echo "Time to exhaust budget at 1x:    30 days"
}

# Usage
calculate_slo_thresholds 99.95
# Output:
#   SLO Target:             99.95%
#   Error Budget:           .0005
#   Fast Burn (14.4x):      .0072% error rate
#   Slow Burn (3x):         .0015% error rate
```

### 3.4 Per-Endpoint SLO Alerts

```yaml
# Alerting per critical endpoint
groups:
  - name: per_endpoint_slo
    rules:
      - alert: EndpointSLOCritical
        expr: |
          (
            sum by (endpoint) (rate(http_requests_total{status=~"5.."}[1m]))
            /
            sum by (endpoint) (rate(http_requests_total[1m]))
          )
          > 0.01
          and
          (
            sum by (endpoint) (rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum by (endpoint) (rate(http_requests_total[5m]))
          )
          > 0.01
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Endpoint {{ $labels.endpoint }} error rate > 1%"
```

---

## 4. SLO Dashboard Design

### 4.1 Key Dashboard Panels

| Panel | Type | Purpose |
|-------|------|---------|
| Error Budget Remaining | Stat (gauge) | Percentage of remaining budget |
| SLI Compliance | Timeseries | % good requests over time |
| Burn Rate | Timeseries | Fast (1h) + Slow (6h) burn rates |
| Error Budget Burn-down | Timeseries | Budget consumed over SLO window |
| Request Volume | Timeseries | Rate + error rate overlay |
| Latency p50/p90/p99 | Timeseries | Latency distribution |
| Top-N Error Endpoints | Table | Highest error rate endpoints |
| SLO Target line | Threshold overlay | Reference line on all panels |

### 4.2 Prometheus Recording Rules for Dashboard Performance

```yaml
groups:
  - name: slo_dashboard
    interval: 30s
    rules:
      # Pre-aggregate for faster dashboard queries
      - record: dashboard:slo:availability_30d
        expr: |
          sum(rate(http_requests_total{status!~"5.."}[30d]))
          /
          sum(rate(http_requests_total[30d]))

      - record: dashboard:slo:error_budget_remaining
        expr: |
          1 - (
            sum(rate(http_requests_total{status=~"5.."}[30d]))
            /
            sum(rate(http_requests_total[30d]))
          ) / 0.001  # Replace with your SLO budget

      - record: dashboard:slo:burn_rate_1h
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          ) / 0.001  # Replace with your SLO budget

      - record: dashboard:slo:latency_p99
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
          )
```

---

## 5. Multi-Service SLO Strategies

### 5.1 Composite SLO

When a user request traverses multiple services, the overall SLO is the **product** of each service's SLO:

```
Composite SLO = SLO_service_A × SLO_service_B × SLO_service_C

Example:
  API Gateway:       99.99%
  Payment Service:   99.99%
  Database:          99.999%
  ─────────────────────────────
  Composite user SLO: 99.979%
```

### 5.2 Tiered SLO Model

```
Critical Path SLOs (99.99%):
  ├── User authentication
  ├── Payment processing
  └── Database writes

Important Path SLOs (99.9%):
  ├── Product search
  ├── Order history
  └── Notification delivery

Best Effort SLOs (99%):
  ├── Analytics pipeline
  ├── Recommendation engine
  └── Report generation
```

### 5.3 Dependency-Based SLO Attribution

```
When service B depends on service A:
  Service B's SLO = Service B's own SLO / Service A's SLO

Example:
  Service A (auth):  99.99%
  Service B (orders): 99.95%
  ──────────────────────────
  Service B's own performance SLO: 99.95% / 99.99% = 99.96%
```

---

## 6. SLO Review Cadence

### 6.1 Recommended Review Cycle

| Frequency | Activity | Who |
|-----------|----------|-----|
| **Weekly** | Check error budget burn rate in dashboard | On-call engineer |
| **Monthly** | Review SLO attainment report; adjust dashboards | Service team |
| **Quarterly** | Re-evaluate SLO targets; review budget consumption trends | Product + Engineering |
| **Annually** | Adjust SLO tiers with business stakeholders | Engineering + Product leadership |

### 6.2 SLO Target Adjustment Decision Matrix

| Budget Consumption | Trend | Action |
|--------------------|-------|--------|
| <50% used | Consistent | Consider tightening SLO (e.g., 99.9% → 99.95%) |
| >80% used | Growing | Review reliability investments; loosen SLO or improve |
| >100% used | Frequent | SLO is unrealistic for current investment; lower target or allocate resources |
| >100% used | Rare (postmortem follow-up) | Accept one-time miss; no change to target |

### 6.3 SLO Review Template

```markdown
## Monthly SLO Review — {team}

### Service: {service_name}
### Period: {YYYY-MM-DD} → {YYYY-MM-DD}

### Summary
- Current SLO target: {99.9%}
- Actual SLI: {99.92%}
- Error budget consumed: {20%}
- Burn rate events: {3 warnings, 0 criticals}

### Notable Events
- {date}: {deploy vX.Y caused 5min 500 spike — consumed 0.5% budget}
- {date}: {db performance regression, TTR: 15min}

### Actions
- [ ] Re-evaluate latency SLI threshold (currently 300ms, p99 actual is 180ms → tighten?)
- [ ] Add endpoint-specific SLO for /api/payments
- [ ] Review burn rate alert sensitivity (0 false positives this month)

### Decision
SLO target: ☐ Keep at {99.9%} / ☐ Tighten to {99.95%} / ☐ Loosen to {99.5%}
```

---

## References

- [Google SRE Book — SLO Chapter](https://sre.google/workbook/implementing-slos/)
- [Google SRE Workbook — Alerting on SLOs](https://sre.google/workbook/alerting-on-slos/)
- [Prometheus Alerting Rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
- [Grafana SLO Dashboard Tutorial](https://grafana.com/go/observabilitycon/2023/grafana-slo-dashboard/)
brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 322kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 198kUpdated Claude CodeCodexVerifiedInstall