Skip to content

fattain-naime/engineering-docs / skills/technical-runbook

technical-runbook

Flagged

Write a production operations runbook that provides on-call engineers with step-by-step procedures for operating, monitoring, and recovering a system. Covers system overview, alert response procedures, common failure modes, diagnostic commands, escalation paths, and maintenance procedures. Modeled on Google SRE runbook standards.

FileKindSize
SKILL.mdskill md17.1 KB
template.mdmarkdown23.9 KB

template.md

View on GitHub
---
title: Technical Runbook
skill: technical-runbook
status: draft
owner_reviewed: false
last_updated: 2026-07-17
depends_on: []
supersedes: ""
---

# Technical Runbook

**Service:** [Service Name]
**Document ID:** RUN-[SERVICE]-[VERSION]
**Version:** 1.0.0
**Last Updated:** YYYY-MM-DD
**Last Verified Against Production:** YYYY-MM-DD
**Owner Team:** [Team Name]
**On-call Rotation:** [Link to PagerDuty / OpsGenie rotation]

---

## ⚡ Quick Reference

> This section is for engineers who are mid-incident and need answers in 60 seconds.

| Symptom | Jump To |
| :--- | :--- |
| High 5xx error rate | [Section 4.1](#41-alert-high-5xx-error-rate) |
| Site is down / not responding | [Section 4.2](#42-alert-service-unreachable) |
| Database is slow or unresponsive | [Section 4.3](#43-alert-database-slow-query--connection-exhaustion) |
| Queue depth growing / jobs not processing | [Section 4.4](#44-alert-queue-depth-elevated) |
| Memory or CPU critically high | [Section 4.5](#45-alert-resource-exhaustion) |
| Need to restart a service | [Section 7.1](#71-restart-application-service) |
| Need to check logs | [Section 6.1](#61-log-locations-and-search) |
| Need to escalate | [Section 5](#5-escalation-path) |

---

## 1. Service Overview

### 1.1 What This Service Does

[2-3 sentences: what does this service do, who depends on it, and what is the impact if it goes down?]

### 1.2 Service SLA

| SLA Component | Target | Alert Threshold |
| :--- | :--- | :--- |
| Availability | `99.9%` (monthly) | Page if < `99.5%` over 5 min |
| p99 Latency | `< 200ms` | Page if > `500ms` sustained 2 min |
| Error Rate | `< 0.1%` | Page if > `1%` sustained 2 min |

### 1.3 Architecture Summary

```
[Client] → [Load Balancer] → [Web/App Server(s)] → [Database Primary]
                                                   → [Redis Cache/Queue]
                                                   → [Queue Worker(s)]
```

| Component | Technology | Count | Responsibility |
| :--- | :--- | :--- | :--- |
| Web server | Nginx + PHP-FPM | [N] | HTTP request handling |
| Database | MySQL 8.x | 1 primary + [N] replicas | Data persistence |
| Cache / Queue | Redis | [N] | Session cache, async job queue |
| Queue worker | PHP CLI process | [N] | Background job processing |

### 1.4 External Dependencies

| Dependency | Type | Impact if Down | Fallback |
| :--- | :--- | :--- | :--- |
| [External service A] | Critical | [Service is fully degraded] | [None / Graceful degradation] |
| [External service B] | Non-critical | [Feature X unavailable] | [Cached response for N hours] |

---

## 2. Monitoring and Dashboards

| Dashboard | URL | What to Check |
| :--- | :--- | :--- |
| Primary monitoring | [URL - Grafana / Datadog] | Error rate, latency, throughput |
| Error tracking | [URL - Sentry / Rollbar] | Exception rate, new errors |
| Database | [URL] | Query time, connections, replication lag |
| Queue | [URL] | Queue depth, job failure rate, worker count |
| Infrastructure | [URL] | CPU, memory, disk, network |

### Key Metrics to Watch

| Metric | Normal Range | Warning | Critical |
| :--- | :--- | :--- | :--- |
| HTTP 5xx rate | `< 0.1%` | `0.1% - 1%` | `> 1%` |
| p99 API latency | `< 200ms` | `200-500ms` | `> 500ms` |
| DB connections active | `< [N]` | `> [N*0.7]` | `> [N*0.9]` |
| DB replication lag | `< 1s` | `1-5s` | `> 5s` |
| Queue depth | `< 100` | `100-1000` | `> 1000` |
| Server CPU | `< 60%` | `60-85%` | `> 85%` |
| Server memory | `< 70%` | `70-90%` | `> 90%` |
| Disk usage | `< 70%` | `70-85%` | `> 85%` |

---

## 3. Alert Index

| Alert Name | Severity | Trigger Condition | Runbook Section |
| :--- | :--- | :--- | :--- |
| `[SERVICE]_high_error_rate` | `Critical` | 5xx rate > 1% for 2 min | [4.1] |
| `[SERVICE]_unreachable` | `Critical` | Health check fails for 1 min | [4.2] |
| `[SERVICE]_db_slow` | `Critical` | Query p99 > 1s for 5 min | [4.3] |
| `[SERVICE]_queue_depth` | `Warning` | Queue depth > 1000 for 5 min | [4.4] |
| `[SERVICE]_resource_high` | `Warning` | CPU > 85% or Memory > 90% for 5 min | [4.5] |
| `[SERVICE]_disk_full` | `Critical` | Disk usage > 90% | [4.6] |

---

## 4. Alert Response Procedures

---

### 4.1 Alert: High 5xx Error Rate

**Severity:** Critical | **Page:** Yes | **SLA Impact:** Yes

#### Diagnosis (< 5 minutes)

1. **Check error tracking** for new exceptions:
   - Open [Sentry/Rollbar URL]
   - Filter to last 15 minutes
   - Look for new or spiking error classes

2. **Check recent deployments:**
   - Was a deployment made in the last 30 minutes?
   ```bash
   # Check deployment log
   cat /var/www/current/RELEASE
   ```
   - If yes, prepare for rollback (Section 7.4)

3. **Check application logs:**
   ```bash
   # Last 100 error lines
   tail -n 200 /var/log/[app]/error.log | grep -E "ERROR|CRITICAL|Exception"

   # PHP-FPM errors
   tail -n 100 /var/log/php8.3-fpm.log
   ```

4. **Check if specific endpoints are failing** (not all):
   - Look at error tracking for which URL patterns are throwing 5xx
   - Isolate: is this all traffic or specific functionality?

#### Mitigation Options

| Root Cause | Action |
| :--- | :--- |
| Recent bad deployment | Rollback → Section 7.4 |
| Database down / unreachable | → Section 4.3 |
| PHP-FPM pool exhausted | Restart PHP-FPM → Section 7.2 |
| Memory exhaustion causing OOM kills | Check memory → Scale or clear cache |
| External dependency down | Enable fallback if available; notify [Dependency team] |
| Bad data input causing exception | Identify specific input; consider feature flag to disable if safe |

#### Escalation Trigger

Escalate to [Lead Engineer] if:
- Root cause not identified within 15 minutes
- Mitigation does not reduce error rate within 10 minutes of applying
- Error rate reaches 5% sustained

---

### 4.2 Alert: Service Unreachable

**Severity:** Critical | **Page:** Yes | **SLA Impact:** Yes

#### Diagnosis

1. **Check from outside** (is it a network issue?):
   ```bash
   curl -v https://[domain]/health
   # Expected: HTTP 200 {"status":"ok"}
   # If timeout: check load balancer / networking
   ```

2. **Check web server status:**
   ```bash
   sudo systemctl status nginx
   sudo systemctl status php8.3-fpm
   ```

3. **Check process resource limits:**
   ```bash
   # PHP-FPM pool status
   sudo php-fpm8.3 -t  # Config test
   ps aux | grep php-fpm | wc -l  # Count running workers
   ```

4. **Check system resources:**
   ```bash
   free -h          # Memory
   df -h            # Disk
   top -b -n1       # CPU snapshot
   ```

#### Mitigation

```bash
# Restart web server (graceful)
sudo systemctl reload nginx

# Restart PHP-FPM (graceful reload preserves in-flight requests)
sudo systemctl reload php8.3-fpm

# If still unresponsive, full restart (brief 5xx spike expected):
sudo systemctl restart php8.3-fpm
sudo systemctl restart nginx
```

---

### 4.3 Alert: Database Slow Query / Connection Exhaustion

**Severity:** Critical | **Page:** Yes | **SLA Impact:** Yes

#### Diagnosis

1. **Check active queries:**
   ```sql
   -- Connect to MySQL
   mysql -u [user] -p

   -- Show currently running queries (anything > 5s is suspicious)
   SHOW PROCESSLIST;
   -- Or:
   SELECT id, user, host, db, time, state, info
   FROM information_schema.processlist
   WHERE time > 5
   ORDER BY time DESC;
   ```

2. **Check replication lag** (if using replicas):
   ```sql
   -- On replica:
   SHOW REPLICA STATUS\G
   -- Check: Seconds_Behind_Source
   ```

3. **Check connection count:**
   ```sql
   SHOW STATUS LIKE 'Threads_connected';
   SHOW VARIABLES LIKE 'max_connections';
   ```

4. **Check slow query log:**
   ```bash
   tail -n 50 /var/log/mysql/slow.log
   ```

#### Mitigation

| Issue | Action |
| :--- | :--- |
| Specific slow query blocking everything | `KILL [processlist_id];` |
| Connection pool exhausted | Reduce app workers temporarily; restart worker processes |
| Replication lag > 30s | Route reads to primary temporarily; investigate replica load |
| Index missing | Identify query pattern; add index during low-traffic window |
| Table lock | Identify locking query; kill if appropriate; investigate why lock occurred |

---

### 4.4 Alert: Queue Depth Elevated

**Severity:** Warning | **Page:** Yes (if Critical threshold) | **SLA Impact:** Delayed jobs

#### Diagnosis

1. **Check queue worker status:**
   ```bash
   sudo systemctl status [queue-worker-service]
   ps aux | grep [worker-process-name]
   ```

2. **Check for failed jobs:**
   ```bash
   # Application-specific command
   php artisan queue:failed  # Or equivalent
   ```

3. **Check worker logs:**
   ```bash
   tail -n 100 /var/log/[app]/worker.log | grep -E "ERROR|FAILED|Exception"
   ```

#### Mitigation

```bash
# Restart workers (if they are stuck or crashed)
sudo systemctl restart [queue-worker-service]

# Scale workers temporarily (if depth is due to volume, not errors)
# Start N additional worker processes
for i in {1..3}; do
  nohup php /var/www/current/artisan queue:work --max-time=3600 &
done

# Clear failed jobs (only after investigating root cause)
php artisan queue:flush
```

---

### 4.5 Alert: Resource Exhaustion (CPU/Memory)

**Severity:** Warning → Critical | **Page:** Yes at Critical | **SLA Impact:** Degraded performance

#### Diagnosis

```bash
# CPU
top -b -n1 | head -20
# Identify high-CPU processes

# Memory
free -h
ps aux --sort=-%mem | head -20

# Check for memory leaks in PHP-FPM (workers using > N MB may have leak)
ps aux | grep php-fpm | awk '{print $6}' | sort -n | tail -5
```

#### Mitigation

```bash
# Graceful PHP-FPM restart (recycles workers, frees leaked memory)
sudo systemctl reload php8.3-fpm

# If disk full (different from memory):
# Find large files
du -sh /var/log/[app]/* | sort -h
# Rotate/clear old logs
sudo logrotate -f /etc/logrotate.d/[app]
# Clear PHP opcache
# (if applicable to your setup)
```

---

### 4.6 Alert: Disk Full

**Severity:** Critical | **Page:** Yes | **SLA Impact:** Writes fail, application crashes**

```bash
# Find top disk consumers
du -sh /* 2>/dev/null | sort -rh | head -20
du -sh /var/log/* 2>/dev/null | sort -rh | head -10

# Rotate logs immediately
sudo journalctl --vacuum-size=500M
sudo logrotate -f /etc/logrotate.conf

# Clear old application exports or temporary files
find /var/www/current/storage/exports -mtime +7 -delete
find /tmp -mtime +1 -delete

# Check for unrotated application logs
ls -lah /var/log/[app]/
```

---

## 5. Escalation Path

> Contact in order. Do not skip levels during business hours. Skip directly to L2 for Severity 1 (full outage) at any hour.

| Level | Who | When to Contact | How to Reach |
| :--- | :--- | :--- | :--- |
| L0 | This runbook | Always try first | You are reading it |
| L1 | [On-call Engineer] | Runbook step fails or unknown issue | PagerDuty / Phone: [number] |
| L2 | [Senior/Lead Engineer] | L1 cannot resolve in 15 min, or Sev-1 | Phone: [number] / Slack: @[handle] |
| L3 | [Architect / Principal] | L2 cannot resolve; data loss risk; security incident | Phone: [number] |
| Vendor | [Cloud provider / DB vendor support] | Infrastructure-level issue confirmed | Support portal: [URL] |

**Incident channel:** `#incidents` in Slack

---

## 6. Diagnostic Reference

### 6.1 Log Locations and Search

| Log | Path | Format |
| :--- | :--- | :--- |
| Application error log | `/var/log/[app]/error.log` | JSON |
| PHP-FPM log | `/var/log/php8.3-fpm.log` | Text |
| Nginx access log | `/var/log/nginx/access.log` | Combined format |
| Nginx error log | `/var/log/nginx/error.log` | Text |
| MySQL slow query log | `/var/log/mysql/slow.log` | Text |
| Queue worker log | `/var/log/[app]/worker.log` | JSON |

```bash
# Find all ERROR-level entries in the last 1 hour
grep '"level":"error"' /var/log/[app]/error.log | \
  awk -v ts="$(date -d '1 hour ago' +%s)" \
  '{ if ($0 ~ /"timestamp":"[0-9]+"/) { ... } }' | tail -50
# Or simpler:
tail -f /var/log/[app]/error.log | grep -E "ERROR|CRITICAL"
```

### 6.2 Health Check Commands

```bash
# Full health check
curl -s https://[domain]/health | python3 -m json.tool

# Database connectivity check
php /var/www/current/artisan db:show  # Or equivalent
mysql -u [user] -p -e "SELECT 1;" [database]

# Redis connectivity check
redis-cli ping  # Expected: PONG

# Queue worker alive check
ps aux | grep queue:work | grep -v grep
```

### 6.3 Useful SQL Snippets

```sql
-- Check table sizes
SELECT table_name, round(data_length/1024/1024, 2) AS data_mb,
       round(index_length/1024/1024, 2) AS index_mb
FROM information_schema.tables
WHERE table_schema = '[database_name]'
ORDER BY data_length DESC
LIMIT 20;

-- Kill a slow query by ID (get ID from SHOW PROCESSLIST)
KILL QUERY [process_id];

-- Check replication status
SHOW REPLICA STATUS\G
```

---

## 7. Maintenance Procedures

### 7.1 Restart Application Service

```bash
# Graceful restart (preferred - no request drops)
sudo systemctl reload php8.3-fpm
sudo systemctl reload nginx

# Full restart (use only if graceful fails)
sudo systemctl restart php8.3-fpm
sudo systemctl restart nginx

# Verify:
curl -f https://[domain]/health
```

**Impact:** Graceful reload: < 1 second latency spike. Full restart: 5-10 second 502 spike.

---

### 7.2 Restart Queue Workers

```bash
# Graceful stop (workers finish current job then stop)
sudo systemctl stop [queue-worker-service]

# Start workers
sudo systemctl start [queue-worker-service]

# Verify workers running:
sudo systemctl status [queue-worker-service]
ps aux | grep queue:work | grep -v grep
```

---

### 7.3 Clear Application Cache

```bash
# Clear all caches
php /var/www/current/artisan cache:clear

# Rebuild config/route/view cache
php /var/www/current/artisan config:cache
php /var/www/current/artisan route:cache
php /var/www/current/artisan view:cache
```

---

### 7.4 Emergency Rollback

See [Deployment Plan template] for the full rollback procedure.

**Quick version:**
```bash
# List available releases
ls -lt /var/www/releases/

# Switch to previous release
ln -sfn /var/www/releases/[PREVIOUS_VERSION] /var/www/current
sudo systemctl reload php8.3-fpm

# Verify:
curl -f https://[domain]/health
```

---

### 7.5 Database Backup (Emergency)

```bash
# Take an emergency backup before a risky operation
mysqldump \
  -u [user] -p \
  --single-transaction \
  --quick \
  --lock-tables=false \
  [database_name] \
  > /var/backups/mysql/emergency_$(date +%Y%m%d_%H%M%S).sql

# Verify backup:
ls -lh /var/backups/mysql/emergency_*.sql
```

---

## 8. Certificate Management

### 8.1 Certificate Inventory

| Domain | Issuer | Type | Expiry Date | Auto-Renew? | Service(s) |
| :--- | :--- | :--- | :--- | :--- | :--- |
| [api.example.com] | [Let's Encrypt / DigiCert] | [DV / OV / EV] | [YYYY-MM-DD] | `Yes` / `No` | [Nginx, Load Balancer] |
| [admin.example.com] | [Issuer] | [Type] | [YYYY-MM-DD] | `Yes` / `No` | [Service] |

### 8.2 Certificate Rotation Procedure

**Last Verified:** YYYY-MM-DD

**Automated renewal (Let's Encrypt / certbot):**
```bash
# Check certbot renewal status
sudo certbot certificates

# Force renewal (if needed)
sudo certbot renew --force-renewal

# Reload Nginx to pick up new cert
sudo systemctl reload nginx

# Verify new cert
echo | openssl s_client -connect [domain]:443 -servername [domain] 2>/dev/null | openssl x509 -noout -dates
```

**Manual renewal (purchased certificate):**
```bash
# Generate new CSR
openssl req -new -newkey rsa:2048 -nodes \
  -keyout /etc/ssl/private/[domain].key \
  -out /etc/ssl/certs/[domain].csr

# After receiving cert from CA, install:
sudo cp [new-cert].crt /etc/ssl/certs/[domain].crt
sudo systemctl reload nginx

# Verify
curl -vI https://[domain] 2>&1 | grep -i "expire\|issuer"
```

### 8.3 Emergency Certificate Rotation (Compromised or Expired)

```bash
# If certificate is compromised or expired:
# 1. Generate new key immediately
openssl req -new -newkey rsa:2048 -nodes \
  -keyout /etc/ssl/private/[domain]-new.key \
  -out /etc/ssl/certs/[domain]-new.csr

# 2. Submit to CA for emergency issuance (or use certbot)
sudo certbot certonly --standalone -d [domain] --force-renewal

# 3. Deploy new cert
sudo systemctl reload nginx

# 4. Revoke old cert (if compromised)
sudo certbot revoke --cert-name [domain] --reason keycompromise
```

**Escalation:** If certificate cannot be renewed within [2 hours], escalate to [L2 - Lead Engineer].

---

## 9. Secret Rotation

### 9.1 Secret Inventory

| Secret | Storage Location | Consumed By | Last Rotated | Rotation Schedule | Zero-Downtime? |
| :--- | :--- | :--- | :--- | :--- | :--- |
| [Database password] | [Vault / AWS Secrets Manager / env file] | [App servers, queue workers] | [YYYY-MM-DD] | [Monthly] | `Yes` - dual-user rotation |
| [API key - payment provider] | [Vault] | [App servers] | [YYYY-MM-DD] | [Quarterly] | `No` - restart required |
| [Redis password] | [Vault] | [App servers, workers] | [YYYY-MM-DD] | [Quarterly] | `Yes` - hot reload |
| [Encryption key] | [Vault / KMS] | [App servers] | [YYYY-MM-DD] | [Annually] | `Yes` - dual-key support |

### 9.2 Database Password Rotation

**Last Verified:** YYYY-MM-DD

```bash
# Step 1: Create new user with new password
mysql -u root -p -e "
  CREATE USER '[app_user_new]'@'%' IDENTIFIED BY '[new_password]';
  GRANT ALL PRIVILEGES ON [database].* TO '[app_user_new]'@'%';
  FLUSH PRIVILEGES;
"

# Step 2: Update secret store with new credentials
# [Vault/Secrets Manager update command]

# Step 3: Restart application to pick up new credentials
sudo systemctl reload php8.3-fpm
sudo systemctl restart [queue-worker-service]

# Step 4: Verify no connections using old user
mysql -u root -p -e "
  SELECT user, host, db FROM information_schema.processlist WHERE user = '[app_user_old]';
"
# Expected: 0 rows

# Step 5: Drop old user (after grace period of [24 hours])
mysql -u root -p -e "DROP USER '[app_user_old]'@'%';"
```

### 9.3 API Key Rotation

```bash
# Step 1: Generate new key in provider dashboard
# Step 2: Update secret store
# [Vault/Secrets Manager update command]
# Step 3: Restart consumers
sudo systemctl reload php8.3-fpm
# Step 4: Verify new key works (call provider test endpoint)
curl -H "Authorization: Bearer [new_key]" https://[provider]/test
# Step 5: Revoke old key (after grace period)
```

---

## 10. Configuration Change

### 10.1 Configuration Inventory

| Configuration | Location | Type | Consumers | Hot-Reload? |
| :--- | :--- | :--- | :--- | :--- |
| [APP_DEBUG] | [.env / Vault] | [Environment variable] | [App servers] | `No` - restart required |
| [Feature flag: new_checkout] | [LaunchDarkly / config file] | [Feature flag] | [App servers] | `Yes` - evaluated per request |
| [DB_CONNECTION_TIMEOUT] | [config/database.php] | [Config file] | [App servers] | `No` - restart required |

### 10.2 Configuration Change Procedure

**Last Verified:** YYYY-MM-DD

```bash
# Step 1: Document current value
cat /var/www/current/.env | grep [CONFIG_KEY]

# Step 2: Apply change
# For .env files:
sudo sed -i 's/[CONFIG_KEY]=[old_value]/[CONFIG_KEY]=[new_value]/' /var/www/current/.env

# Step 3: Pick up change
# If hot-reload: no restart needed
# If restart required:
sudo systemctl reload php8.3-fpm

# Step 4: Verify change took effect
curl -s https://[domain]/health | python3 -m json.tool
# Check logs for config reload confirmation:
tail -f /var/log/[app]/error.log | grep -i "config"

# Step 5: Rollback if needed
sudo sed -i 's/[CONFIG_KEY]=[new_value]/[CONFIG_KEY]=[old_value]/' /var/www/current/.env
sudo systemctl reload php8.3-fpm
```

### 10.3 Feature Flag Management

| Flag | State | Authority to Toggle | Blast Radius | Rollback |
| :--- | :--- | :--- | :--- | :--- |
| `[flag_name]` | `ON` / `OFF` | [Name, Role] | [Which features/users affected] | [Set to OFF] |

---

## 11. Performance Profiling

### 11.1 When to Profile

Profile when:
- p99 latency exceeds warning threshold (> 200ms) but is below critical (< 500ms)
- A specific endpoint is slow while others are normal
- After a deployment that introduced new database queries or external API calls

### 11.2 Profiling Commands

**Last Verified:** YYYY-MM-DD

**Xdebug profiler (PHP):**
```bash
# Enable for one request via query parameter
curl "https://[domain]/[slow-endpoint]?XDEBUG_PROFILE=1"

# Cachegrind output location
ls -lt /tmp/cachegrind.out.*

# Analyze with KCachegrind (GUI) or qcachegrind
qcachegrind /tmp/cachegrind.out.[pid]
```

**Blackfire profiler:**
```bash
# Profile a request
blackfire curl https://[domain]/[slow-endpoint]

# Review results at blackfire.io dashboard
```

**New Relic / Datadog APM:**
1. Open APM dashboard: [URL]
2. Navigate to slow transaction traces
3. Identify slow span (DB query, external HTTP, template rendering)
4. Check SQL query details for missing indexes

**MySQL slow query log:**
```bash
# Check for slow queries
tail -n 50 /var/log/mysql/slow.log

# Look for high Rows_examined / Rows_sent ratio (missing index)
grep "Rows_examined" /var/log/mysql/slow.log | awk -F'Rows_examined:' '{print $2}' | sort -n | tail -20
```

**PHP-FPM slow log:**
```bash
# Check for slow requests at the PHP-FPM level
tail -n 50 /var/log/php-fpm-slow.log
```

### 11.3 Safe Production Profiling

- **Sampling:** Enable profiler for 1-5% of requests only (Xdebug: use `xdebug.profiler_enable=0` + trigger; Blackfire: use sampling mode)
- **Duration:** Disable profiling after capturing sufficient samples (10-20 slow requests)
- **Overhead:** Xdebug adds 20-50% overhead; Blackfire adds 2-5%. Never leave Xdebug profiler enabled in production permanently.

---

## 12. Multi-Region / Multi-Server Procedures

### 12.1 Server Inventory

| Server / Instance | Region | Role | IP / Hostname | Status |
| :--- | :--- | :--- | :--- | :--- |
| [web-01] | [us-east-1] | [Web server] | [10.0.1.10] | `Active` |
| [web-02] | [us-east-1] | [Web server] | [10.0.1.11] | `Active` |
| [web-03] | [eu-west-1] | [Web server] | [10.0.2.10] | `Active` |
| [db-primary] | [us-east-1] | [MySQL primary] | [10.0.1.20] | `Active` |
| [db-replica] | [us-east-1] | [MySQL replica] | [10.0.1.21] | `Active` |

### 12.2 Region-Specific Procedures

| Procedure | Region-Specific? | Details |
| :--- | :--- | :--- |
| [DNS failover] | `Yes` | [Different DNS records per region; see DR plan] |
| [Backup location] | `Yes` | [us-east-1: s3://backups-us; eu-west-1: s3://backups-eu] |
| [Scaling group] | `Yes` | [Each region has its own ASG with different min/max] |

### 12.3 Distributed Debugging

**Request tracing:**
```bash
# Search for a specific request across all servers
# [Trace ID / Request ID] is set in the X-Request-ID header

# On each server:
grep "[request-id]" /var/log/[app]/error.log

# Or via centralized logging:
# [Datadog/ELK query: @request_id:"[request-id]"]
```

### 12.4 Cross-Region Failover Quick Reference

> For full failover procedure, see the disaster-recovery-plan.

```bash
# Quick check: is the other region healthy?
curl -f https://[domain-failover-region]/health

# If primary region is down, update DNS:
# [Route53 / DNS provider command to failover]
```

---

## 13. Runbook Validation Drills

### 13.1 Drill Schedule

| Drill Date | Scenario | Engineer | Result | Gaps Found | Action Items |
| :--- | :--- | :--- | :--- | :--- | :--- |
| YYYY-MM-DD | [Simulated high 5xx alert] | [Name] | `Pass` / `Fail` | [Description] | [Owner, deadline] |
| YYYY-MM-DD | [Simulated DB connection exhaustion] | [Name] | `Pass` / `Fail` | [Description] | [Owner, deadline] |

### 13.2 Quarterly Review Checklist

> Complete this checklist every quarter. Each unchecked item is a gap that needs resolution.

- [ ] Every diagnostic command has been executed against current production; output matches documentation
- [ ] Every alert in the Alert Index (Section 3) still exists in the monitoring system with the documented trigger conditions
- [ ] Escalation contacts (Section 5) are current - no one has left the team or changed roles
- [ ] Log paths and formats have not changed since last verification
- [ ] All referenced tools and CLIs are installed and accessible on the on-call engineer's workstation
- [ ] Certificate expiry dates (Section 8) are all > 30 days out
- [ ] Secret rotation dates (Section 9) are all within schedule
- [ ] "Last Verified" date updated on each procedure section

**Last quarterly review:** YYYY-MM-DD | **Reviewer:** [Name] | **Next review due:** YYYY-MM-DD

---

## 14. Change Log

| Date | Version | Author | Change |
| :--- | :--- | :--- | :--- |
| YYYY-MM-DD | 1.0.0 | [Name] | Initial runbook created |
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 197kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall