hoangminhung/devops-ai-kit / claude/skills/disaster-recovery
disaster-recovery
VerifiedBackup and disaster-recovery standards — defining RTO/RPO, 3-2-1 and offsite copies, immutable backups (S3 Object Lock, Azure immutable blob storage, GCS retention), backup layers for Kubernetes objects, volumes and databases (GitOps, Velero, Longhorn, pgBackRest), restore drills, and backup monitoring. Use whenever designing backups, before any stateful or irreversible change, when defining RTO/RPO for a service, or when planning or reviewing restore procedures and DR drills.
Backup & disaster recovery standards
Basic rule: a backup that has never been test-restored is not a backup.
1. RTO / RPO
- MUST Each production service has an RTO (the longest time allowed to recover) and an RPO (the most data you accept to lose), written in the service's runbook or ADR.
- MUST The backup design must meet the written RPO (for example: with an RPO of 5 minutes, a nightly backup is not enough; you need WAL archiving / PITR).
2. Layers to back up
| Layer | Recommended method |
|---|---|
| Kubernetes objects | The GitOps repo is the source of truth. CONTEXT Velero when some objects are not in Git |
| Volumes | Longhorn backup to object storage, or CSI snapshot + Velero data mover |
| Databases | DB-level backup (pgBackRest via PGO, Barman via CNPG) with PITR, preferred over volume snapshots |
| Cloud infrastructure | IaC code in Git + a state backend with versioning |
| Secrets | Key Vault / Secrets Manager with soft-delete and purge protection; know how to recover |
3. Storage location & immutability
- SHOULD 3-2-1: at least 3 copies, on 2 types of media / systems, 1 copy in another place (another account / subscription / region).
- SHOULD Production: immutable backups, to protect against ransomware or a stolen admin account:
- AWS: S3 Object Lock.
- Azure: immutable storage for Blob (time-based retention).
- GCP: GCS bucket retention policy / lock.
- CONTEXT Compliance mode / a locked policy cannot be shortened or removed, not even by the root account. Test first with governance / unlocked mode, and calculate the storage cost carefully before you lock.
- MUST The credential that writes backups cannot delete old backups (when the platform lets you split these permissions).
4. Restore drills
- MUST Before any L3 change that touches data: test-restore the latest backup into a separate environment and check the data. Write the result in the plan's prerequisites.
- SHOULD Run drills regularly. Ideal: an automatic weekly pipeline: take the latest backup → restore into a sandbox → run data integrity smoke tests → report the result → delete the sandbox. Minimum: a manual drill every quarter.
- SHOULD Measure the real restore time and compare it with the RTO. If they do not match, update the RTO or improve the process.
5. Backup monitoring
- MUST Alert when a backup fails and when "there is no successful backup in N hours". A problem like "the job did not run" never creates a failure event.
- Example metrics (depends on the tool):
- Velero:
velero_backup_last_successful_timestamp; - pgBackRest / Longhorn: use the metrics of the matching exporter — check the metric names for the version that is running.
- Velero:
6. Drill report template (write it in the runbook)
| Date | Backup source | Restored into | Restore time | Data check | Result | Problems found |
|---|
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |