Skip to content

SpillwaveSolutions/document-specialist-skill / skills/documentation-specialist

documentation-specialist

Verified

This skill should be used when creating professional software documentation (SRS, PRD, OpenAPI, SDD, architecture docs, user manuals, tutorials, runbooks) from templates (greenfield) or reverse-engineering documentation from existing code like Spring Boot or FastAPI (brownfield). Also handles documentation audits/reviews, format conversion (Markdown, DOCX, PDF), and diagram generation (Mermaid first: C4, sequence, class, ER, state, flowchart; PlantUML Salt for UI wireframes and leftover UML). Use when asked to "create documentation", "document my code", "write SRS", "generate PRD", "write DESIGN_DOC.md", "architecture document", "wireframe", or "documentation specialist".

FileKindSize
SKILL.mdskill md10.0 KB
references/examples/TOC.mdmarkdown6.5 KB
references/examples/brownfield/database-failover-runbook.mdmarkdown8.1 KB
references/examples/brownfield/spring-boot-sdd.mdmarkdown5.6 KB
references/examples/greenfield/adr-microservices.mdmarkdown6.3 KB
references/examples/greenfield/architecture-doc.mdmarkdown3.7 KB
references/examples/greenfield/billing-srs.mdmarkdown12.4 KB
references/examples/greenfield/cli-getting-started.mdmarkdown9.3 KB
references/examples/greenfield/collaboration-prd.mdmarkdown13.8 KB
references/examples/greenfield/document-specialist-sdd.mdmarkdown15.4 KB
references/examples/greenfield/ecommerce-sdd.mdmarkdown7.8 KB
references/examples/greenfield/rest-api-tutorial.mdmarkdown13.8 KB
references/examples/greenfield/task-api-openapi.yamlconfig19.0 KB
references/examples/greenfield/taskmanager-user-manual.mdmarkdown8.7 KB
references/examples/greenfield/ui-wireframes.mdmarkdown5.8 KB
references/guides/requirements/README.mdmarkdown3.7 KB
references/mappings/backend/fastapi-mapping.yamlconfig9.7 KB
references/mappings/backend/spring-boot-mapping.yamlconfig24.1 KB
references/reference/01-philosophy.mdmarkdown2.0 KB
references/reference/02-requirements-srs-vs-prd.mdmarkdown1.2 KB
references/reference/02-requirements-traceability.mdmarkdown1.7 KB
references/reference/02-requirements-writing.mdmarkdown1.4 KB
references/reference/03-design-adrs.mdmarkdown2.2 KB
references/reference/03-design-arc42.mdmarkdown2.6 KB
references/reference/03-design-requirements-matrix.mdmarkdown1.3 KB
references/reference/04-diagrams-c4-hierarchy.mdmarkdown2.0 KB
references/reference/04-diagrams-selection.mdmarkdown5.1 KB
references/reference/04-diagrams-state-vs-activity.mdmarkdown2.3 KB
references/reference/05-api-checklist.mdmarkdown2.0 KB
references/reference/05-api-openapi.mdmarkdown3.4 KB
references/reference/05-api-stripe-gold-standard.mdmarkdown1.8 KB
references/reference/06-deployment-diagrams.mdmarkdown1.8 KB
references/reference/06-deployment-documentation.mdmarkdown2.9 KB
references/reference/06-deployment-runbooks.mdmarkdown2.5 KB
references/reference/06-operational-docs.mdmarkdown2.7 KB
references/reference/07-user-kb-approach.mdmarkdown1.8 KB
references/reference/07-user-kb-template.mdmarkdown3.1 KB
references/reference/07-user-writing-style.mdmarkdown2.0 KB
references/reference/08-agile-process.mdmarkdown3.2 KB
references/reference/08-tutorial-writing.mdmarkdown7.4 KB
references/reference/09-code-to-docs-detection.mdmarkdown2.5 KB
references/reference/09-code-to-docs-example.mdmarkdown3.2 KB
references/reference/09-code-to-docs-when.mdmarkdown1.7 KB
references/reference/09-code-to-docs-workflow.mdmarkdown2.0 KB
references/reference/10-quality-bonsai-pruning.mdmarkdown2.4 KB
references/reference/10-quality-checklist.mdmarkdown2.6 KB
references/reference/10-quality-metrics.mdmarkdown3.1 KB
references/reference/11-voice-and-style.mdmarkdown1.5 KB
references/reference/comprehensive-guide.mdmarkdown7.3 KB
references/templates/markdown/DESIGN_DOC.mdmarkdown4.5 KB
references/templates/markdown/api-openapi.yamlconfig17.7 KB
references/templates/markdown/architecture.mdmarkdown1.6 KB
references/templates/markdown/design-sdd.mdmarkdown143 B
references/templates/markdown/developer-tutorial.mdmarkdown7.1 KB
references/templates/markdown/getting-started.mdmarkdown6.5 KB
references/templates/markdown/howto-guide.mdmarkdown4.1 KB
references/templates/markdown/requirements-prd.mdmarkdown10.2 KB
references/templates/markdown/requirements-srs.mdmarkdown17.0 KB
references/templates/markdown/runbook.mdmarkdown8.8 KB
references/templates/markdown/user-manual.mdmarkdown4.8 KB
references/workflows/TOC.mdmarkdown1.9 KB
references/workflows/audit-workflow.mdmarkdown2.5 KB
references/workflows/brownfield-workflow.mdmarkdown3.6 KB
references/workflows/brownfield/fastapi-guide.mdmarkdown1.8 KB
references/workflows/brownfield/pulumi-guide.mdmarkdown1.9 KB
references/workflows/brownfield/springboot-guide.mdmarkdown2.2 KB
references/workflows/convert-workflow.mdmarkdown1.4 KB
references/workflows/design-workflow.mdmarkdown3.0 KB
references/workflows/diagram-workflow.mdmarkdown3.7 KB
references/workflows/greenfield-workflow.mdmarkdown3.4 KB
references/workflows/greenfield/advanced.mdmarkdown1.9 KB
references/workflows/greenfield/openapi-guide.mdmarkdown2.5 KB
references/workflows/greenfield/prd-guide.mdmarkdown2.0 KB
references/workflows/greenfield/srs-guide.mdmarkdown2.0 KB
references/workflows/runbook-workflow.mdmarkdown2.4 KB
references/workflows/tutorial-workflow.mdmarkdown2.3 KB
references/workflows/user-docs-workflow.mdmarkdown2.3 KB

references/examples/brownfield/database-failover-runbook.md

View on GitHub
# Runbook: PostgreSQL Primary Database Failover

**Owner**: Data Platform Team
**Risk Level**: Critical
**Last Updated**: 2025-01-18
**Last Tested**: 2025-01-15
**Approved By**: VP Engineering
**Version**: 2.1.0

---

## Quick Reference

| Attribute | Value |
|-----------|-------|
| **Execution Time** | ~15 minutes |
| **Impact Window** | 2-5 minutes downtime |
| **Rollback Time** | ~10 minutes |
| **Prerequisites** | Database admin access, VPN connection |

---

## Scope & Use Case

### When to Use This Runbook
Use when primary PostgreSQL database is unavailable and must failover to standby replica.

**Triggers**:
- Alert: "Primary DB Unresponsive" fires for >5 minutes
- Manual intervention required during maintenance
- Primary instance failure detected

### Expected Outcome
Standby replica promoted to primary, application reconnected, full read/write capability restored.

### What This Does NOT Cover
- Multi-region failover (see Runbook-DB-003)
- Planned maintenance failover (see Runbook-DB-002)

---

## Prerequisites

### Required Access
- [ ] AWS Admin access to production account (role: `DatabaseAdmin`)
- [ ] VPN connected to production network
- [ ] PagerDuty incident created

### Required Tools
- [ ] aws-cli 2.0+ configured with production profile
- [ ] psql client 14+
- [ ] kubectl 1.28+ with production cluster access

**Verify tools**:
```bash
aws sts get-caller-identity --profile production
psql --version
kubectl config current-context  # Should show: production
```

### System State Requirements
- [ ] Standby replica in sync (replication lag <10 seconds)
- [ ] No active database migrations running
- [ ] Backup completed within last 6 hours

### Communication
- [ ] Create P1 incident: https://company.pagerduty.com
- [ ] Notify #incidents channel: "DB failover in progress"
- [ ] Page on-call DBA if not already engaged

---

## Pre-Flight Checks

### Check 1: Verify Standby Health
```bash
aws rds describe-db-instances \
  --db-instance-identifier prod-db-replica \
  --query 'DBInstances[0].[DBInstanceStatus,ReplicationState]' \
  --output table
```

✅ **Pass**: Status=`available`, ReplicationState=`replicating`
❌ **Fail**: STOP - Standby not healthy, escalate to DBA team

---

### Check 2: Check Replication Lag
```bash
psql -h prod-db-replica.abc123.us-east-1.rds.amazonaws.com \
  -U admin -d postgres -c \
  "SELECT NOW() - pg_last_xact_replay_timestamp() AS replication_lag;"
```

✅ **Pass**: Lag <10 seconds
❌ **Fail**: Wait for replica to catch up or escalate

---

### Check 3: Confirm Recent Backup
```bash
aws rds describe-db-snapshots \
  --db-instance-identifier prod-db \
  --query 'DBSnapshots[0].[DBSnapshotIdentifier,SnapshotCreateTime]' \
  --output table
```

✅ **Pass**: Backup from within last 6 hours
❌ **Fail**: Trigger manual snapshot, wait for completion

---

## Step-by-Step Procedure

### Step 1: Put Application in Read-Only Mode
**What this does**: Prevents write attempts during failover

```bash
kubectl scale deployment/api-server --replicas=0 -n production
```

**Expected output**:
```
deployment.apps/api-server scaled
```

**Verification**:
```bash
kubectl get pods -n production | grep api-server
# Should show: No resources found
```

✅ **Success**: All API pods terminated
⚠️ **If failed**: Force delete pods, then proceed

---

### Step 2: Promote Standby to Primary
**What this does**: Converts read replica to standalone primary database

```bash
aws rds promote-read-replica \
  --db-instance-identifier prod-db-replica \
  --profile production
```

**Expected output**:
```json
{
    "DBInstance": {
        "DBInstanceIdentifier": "prod-db-replica",
        "DBInstanceStatus": "modifying"
    }
}
```

**Wait for promotion** (typically 3-5 minutes):
```bash
while true; do
  STATUS=$(aws rds describe-db-instances \
    --db-instance-identifier prod-db-replica \
    --query 'DBInstances[0].DBInstanceStatus' \
    --output text)
  echo "Status: $STATUS"
  [[ "$STATUS" == "available" ]] && break
  sleep 15
done
```

---

### Step 3: Update Application Configuration
**What this does**: Points app to new primary database

```bash
# Update Kubernetes secret with new endpoint
kubectl create secret generic db-connection \
  --from-literal=host=prod-db-replica.abc123.us-east-1.rds.amazonaws.com \
  --from-literal=port=5432 \
  --dry-run=client -o yaml | kubectl apply -f -
```

**Restart application pods**:
```bash
kubectl scale deployment/api-server --replicas=3 -n production
```

**Wait for pods to be ready**:
```bash
kubectl wait --for=condition=ready pod \
  -l app=api-server \
  -n production \
  --timeout=180s
```

---

### Step 4: Verify Database Connectivity
```bash
# Test write operation
psql -h prod-db-replica.abc123.us-east-1.rds.amazonaws.com \
  -U admin -d postgres -c \
  "CREATE TABLE failover_test (id serial, ts timestamp default now()); \
   DROP TABLE failover_test;"
```

✅ **Success**: Table created and dropped without errors

---

## Verification

### Check 1: Application Health
```bash
curl -s https://api.company.com/health | jq .
```

**Expected**:
```json
{
  "status": "healthy",
  "database": "connected",
  "uptime": 60
}
```

---

### Check 2: Monitor Metrics (10 minutes)
**Dashboards**:
- Error Rate: https://grafana.company.com/d/errors
  - Target: <0.1%
- Query Latency: https://grafana.company.com/d/db-perf
  - Target: P95 <50ms
- Connection Pool: https://grafana.company.com/d/db-connections
  - Target: <80% utilized

---

### Check 3: Sample Data Queries
```bash
psql -h prod-db-replica... -c "SELECT COUNT(*) FROM users;"
psql -h prod-db-replica... -c "SELECT COUNT(*) FROM orders WHERE created_at > NOW() - INTERVAL '1 hour';"
```

Verify counts match expected values.

---

## Rollback Procedure

### When to Rollback
- Error rate exceeds 5%
- Database writes failing
- Critical functionality broken

### Rollback Steps
**Note**: Rollback to old primary requires restoring from backup

1. **Scale down application**:
   ```bash
   kubectl scale deployment/api-server --replicas=0 -n production
   ```

2. **Restore from latest snapshot**:
   ```bash
   aws rds restore-db-instance-from-db-snapshot \
     --db-instance-identifier prod-db-restored \
     --db-snapshot-identifier <latest-snapshot> \
     --profile production
   ```

3. **Wait for restore** (~20 minutes)

4. **Update connection config** to point to restored instance

5. **Restart application**

---

## Troubleshooting

### Issue: Promotion Stuck in "modifying"
**Diagnosis**:
```bash
aws rds describe-events \
  --source-identifier prod-db-replica \
  --source-type db-instance \
  --duration 30
```

**Solution**: Wait up to 10 minutes. If still stuck, contact AWS support (Priority: Urgent).

---

### Issue: Application Can't Connect
**Diagnosis**:
```bash
# Check security group rules
aws ec2 describe-security-groups \
  --group-ids sg-abc123 \
  --query 'SecurityGroups[0].IpPermissions'

# Test connection from pod
kubectl run -it --rm debug --image=postgres:14 --restart=Never -- \
  psql -h prod-db-replica... -U admin -c '\conninfo'
```

**Solution**: Verify security group allows traffic from application subnet.

---

## Post-Execution

### Immediate
- [ ] Update incident with "Failover complete"
- [ ] Notify #incidents: "Service restored"
- [ ] Monitor for 30 minutes

### Within 24 Hours
- [ ] Review RDS events for issues
- [ ] Rebuild standby replica from new primary
- [ ] Document actual execution time: ____ minutes
- [ ] Update runbook if steps changed

### Within 1 Week
- [ ] Post-mortem meeting scheduled
- [ ] Root cause identified
- [ ] Action items created
- [ ] Share learnings with engineering

---

## Monitoring

**During Execution**:
- Database Health: https://cloudwatch.aws.amazon.com/db-health
- Application Metrics: https://grafana.company.com/d/app-overview
- Error Logs: https://kibana.company.com/app/discover

**Alert Channels**:
- Critical: PagerDuty rotation
- Warnings: #alerts-production Slack

---

## Escalation

| Contact | Role | When to Engage |
|---------|------|----------------|
| On-Call DBA | Database expert | Replication issues |
| AWS Support | Infrastructure | RDS API failures |
| Engineering Manager | Decision authority | >30 min outage |

---

**Last Reviewed**: 2025-01-15
**Next Review**: 2025-04-15
**Feedback**: data-platform@company.com
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall