borghei/Claude-Skills / engineering/observability-designer
observability-designer
VerifiedDesign observability strategies: SLI/SLO frameworks, alerting, and dashboards. Use when instrumenting a production service, tuning alert rules, designing Grafana dashboards, defining SLOs and error budgets, or reducing alert fatigue.
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 11.6 KB |
| SKILL.md | skill md | 7.5 KB |
| assets/sample_alerts.json | config | 8.0 KB |
| assets/sample_service_api.json | config | 1.7 KB |
| assets/sample_service_web.json | config | 2.4 KB |
| expected_outputs/sample_dashboard.json | config | 19.1 KB |
| expected_outputs/sample_slo_framework.json | config | 21.6 KB |
| references/alert_design_patterns.md | markdown | 11.7 KB |
| references/dashboard_best_practices.md | markdown | 13.8 KB |
| references/dashboards-logs-traces.md | markdown | 3.4 KB |
| references/slo-and-alerting.md | markdown | 4.1 KB |
| references/slo_cookbook.md | markdown | 10.4 KB |
| references/tools-integration-and-troubleshooting.md | markdown | 8.9 KB |
| scripts/alert_optimizer.py | script | 45.6 KB |
| scripts/dashboard_generator.py | script | 48.4 KB |
| scripts/slo_designer.py | script | 28.8 KB |
references/slo-and-alerting.md
View on GitHub# SLO Framework & Alerting Design
Read this when designing the observability strategy itself: the end-to-end workflow, SLI/SLO definitions, error-budget math, multi-window burn-rate alerts, alert classification, and alert-fatigue prevention.
## Workflow
1. **Catalogue services** -- List every service in scope with its type (request-driven, pipeline, storage), criticality tier (T1-T3), and owning team. Validate that at least one T1 service exists before proceeding.
2. **Define SLIs per service** -- For each service, select SLIs from the Golden Signals table. Map each SLI to a concrete Prometheus/InfluxDB metric expression.
3. **Set SLO targets** -- Assign SLO targets based on criticality tier and user expectations. Calculate the corresponding error budget (e.g., 99.9% = 43.8 min/month).
4. **Design burn-rate alerts** -- Create multi-window burn-rate alert rules for each SLO. Validate that every alert has a clear runbook link and response action.
5. **Build dashboards** -- Generate dashboard specs following the hierarchy: Overview > Service > Component > Instance. Cap each screen at 7 panels. Include SLO target reference lines.
6. **Configure log aggregation** -- Define structured log format, set log levels, assign correlation IDs, and configure retention policies per tier.
7. **Instrument traces** -- Set up distributed tracing with sampling strategy (head-based for dev, tail-based for production). Define span boundaries at service and database call points.
8. **Validate coverage** -- Confirm every T1 service has metrics, logs, and traces. Confirm every alert has a runbook. Confirm dashboard load time is under 2 seconds.
## SLI/SLO Quick Reference
| SLI Type | Metric Expression (Prometheus) | Typical SLO |
|----------|-------------------------------|-------------|
| Availability | `1 - (sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])))` | 99.9% |
| Latency (P99) | `histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))` | < 500ms |
| Error rate | `sum(rate(grpc_server_handled_total{grpc_code!="OK"}[5m])) / sum(rate(grpc_server_handled_total[5m]))` | < 0.1% |
| Throughput | `sum(rate(http_requests_total[5m]))` | > baseline |
## Error Budget Calculation
```
Error Budget = 1 - SLO target
Example (99.9% availability):
Monthly budget = 30d x 24h x 60m x 0.001 = 43.2 minutes
If 20 minutes consumed, remaining = 23.2 minutes (53.7% left)
```
## Burn-Rate Alert Design
| Window | Burn Rate | Severity | Budget Consumed |
|--------|-----------|----------|-----------------|
| 5 min / 1 hr | 14.4x | Critical (page) | 2% in 1 hour |
| 30 min / 6 hr | 6x | Warning (ticket) | 5% in 6 hours |
| 2 hr / 3 day | 1x | Info (dashboard) | 10% in 3 days |
**Rule**: Every critical alert must have an actionable runbook. If no clear action exists, downgrade to warning.
## Alert Classification
| Severity | Meaning | Response | Routing |
|----------|---------|----------|---------|
| Critical | Service down or SLO burn rate high | Page on-call immediately | PagerDuty escalation |
| Warning | Approaching threshold, non-user-facing | Create ticket, fix in business hours | Slack channel |
| Info | Deployment notification, capacity trend | Review in next standup | Dashboard only |
## Alert Fatigue Prevention
- **Hysteresis**: Set different thresholds for firing (e.g., > 90% CPU for 5 min) and resolving (e.g., < 80% CPU for 10 min).
- **Suppression**: Suppress dependent alerts during known outages (e.g., suppress pod alerts when node is down).
- **Grouping**: Group related alerts into a single notification (e.g., all pods in one deployment).
- **Precision over recall**: A missed alert that would self-resolve is better than 50 false pages per week.
## Golden Signals
| Signal | What to Monitor | Key Metrics |
|--------|----------------|-------------|
| Latency | Request duration | P50, P95, P99 response time; queue wait; DB query time |
| Traffic | Request volume | RPS with burst detection; active sessions; bandwidth |
| Errors | Failure rate | 4xx/5xx rates; error budget consumption; silent failures |
| Saturation | Resource pressure | CPU/memory/disk utilization; queue depth; connection pool usage |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation). | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK). | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |