Skip to content

hoangminhung/devops-ai-kit / claude/skills/disaster-recovery

disaster-recovery

Verified

Backup and disaster-recovery standards — defining RTO/RPO, 3-2-1 and offsite copies, immutable backups (S3 Object Lock, Azure immutable blob storage, GCS retention), backup layers for Kubernetes objects, volumes and databases (GitOps, Velero, Longhorn, pgBackRest), restore drills, and backup monitoring. Use whenever designing backups, before any stateful or irreversible change, when defining RTO/RPO for a service, or when planning or reviewing restore procedures and DR drills.

Backup & disaster recovery standards

Basic rule: a backup that has never been test-restored is not a backup.

1. RTO / RPO

  • MUST Each production service has an RTO (the longest time allowed to recover) and an RPO (the most data you accept to lose), written in the service's runbook or ADR.
  • MUST The backup design must meet the written RPO (for example: with an RPO of 5 minutes, a nightly backup is not enough; you need WAL archiving / PITR).

2. Layers to back up

LayerRecommended method
Kubernetes objectsThe GitOps repo is the source of truth. CONTEXT Velero when some objects are not in Git
VolumesLonghorn backup to object storage, or CSI snapshot + Velero data mover
DatabasesDB-level backup (pgBackRest via PGO, Barman via CNPG) with PITR, preferred over volume snapshots
Cloud infrastructureIaC code in Git + a state backend with versioning
SecretsKey Vault / Secrets Manager with soft-delete and purge protection; know how to recover

3. Storage location & immutability

  • SHOULD 3-2-1: at least 3 copies, on 2 types of media / systems, 1 copy in another place (another account / subscription / region).
  • SHOULD Production: immutable backups, to protect against ransomware or a stolen admin account:
    • AWS: S3 Object Lock.
    • Azure: immutable storage for Blob (time-based retention).
    • GCP: GCS bucket retention policy / lock.
  • CONTEXT Compliance mode / a locked policy cannot be shortened or removed, not even by the root account. Test first with governance / unlocked mode, and calculate the storage cost carefully before you lock.
  • MUST The credential that writes backups cannot delete old backups (when the platform lets you split these permissions).

4. Restore drills

  • MUST Before any L3 change that touches data: test-restore the latest backup into a separate environment and check the data. Write the result in the plan's prerequisites.
  • SHOULD Run drills regularly. Ideal: an automatic weekly pipeline: take the latest backup → restore into a sandbox → run data integrity smoke tests → report the result → delete the sandbox. Minimum: a manual drill every quarter.
  • SHOULD Measure the real restore time and compare it with the RTO. If they do not match, update the RTO or improve the process.

5. Backup monitoring

  • MUST Alert when a backup fails and when "there is no successful backup in N hours". A problem like "the job did not run" never creates a failure event.
  • Example metrics (depends on the tool):
    • Velero: velero_backup_last_successful_timestamp;
    • pgBackRest / Longhorn: use the metrics of the matching exporter — check the metric names for the version that is running.

6. Drill report template (write it in the runbook)

DateBackup sourceRestored intoRestore timeData checkResultProblems found
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 321kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall