hoangminhung/devops-ai-kit / claude/skills/db-reliability
db-reliability
VerifiedDatabase reliability engineering standards for PostgreSQL (and MySQL) on Kubernetes — zero-downtime schema changes with expand/contract, safe DDL (lock_timeout, CREATE INDEX CONCURRENTLY, NOT VALID constraints), running migrations as Jobs/PreSync hooks instead of initContainers, connection pooling with PgBouncer, and operator-managed clusters (Crunchy PGO, CloudNativePG). Use whenever writing or reviewing schema migrations, deploying or tuning database clusters on Kubernetes, configuring poolers, or planning database upgrades.
DBRE (Database Reliability Engineering) standards
1. Schema changes without downtime (expand / contract)
- MUST Do not rename a column, change a data type or drop a column in one release on production. Use three phases:
- Expand: add the new column / table (nullable or with a default). The app writes to both old and new.
- Backfill: copy old data to the new place in small batches, with a way to stop / resume. Do not run one huge
UPDATEthat locks the table. - Contract: the app uses only the new column. Drop the old column in a later release.
- MUST PostgreSQL: set
SET lock_timeout = '5s'(or a suitable value) before DDL on a busy table. A DDL that waits for a lock blocks every query queued behind it, so it is better to fail early and retry than to hang the whole system. - MUST Create indexes on tables that have data with
CREATE INDEX CONCURRENTLY. This command cannot run inside a transaction, so the migration tool must run that step without a transaction. If it fails, the index is leftINVALID: you mustDROPit and create it again. - SHOULD Add constraints on big tables with
NOT VALID, then runVALIDATE CONSTRAINTin a separate step. - SHOULD Adding a column with a volatile default (for example a function) rewrites the whole table. A constant default is fast (PostgreSQL ≥ 11).
- MySQL: use
ALGORITHM=INPLACE, LOCK=NONEwhen supported. For big tables, consider gh-ost / pt-online-schema-change.
2. Running migrations on Kubernetes
- MUST Do not run migrations in the
initContainersof a Deployment withreplicas > 1: pods run at the same time and cause race conditions and deadlocks. - MUST A migration is a separate Kubernetes
Job, run by an Argo CDPreSynchook (withhook-delete-policy: BeforeHookCreation) or by a separate CD pipeline step, before the app rollout. - SHOULD The Job has a small
backoffLimit, anactiveDeadlineSeconds, and enough logs to see at which step the migration stopped.
3. Connection pooling
- MUST Apps with autoscaling connect to the DB through a pooler (PgBouncer for PostgreSQL, ProxySQL for MySQL), so the number of connections does not go over
max_connectionswhen the number of pods grows. - Calculate first:
max pods × pool size per pod≤PgBouncer pool, anddefault_pool_size × number of database/user pairs≤max_connectionsminus the part kept for admin / replication. - CONTEXT
transactionpooling mode saves the most connections, but it breaks session features (sessionSET, session advisory locks,LISTEN/NOTIFY, prepared statements on old PgBouncer versions). Check the PgBouncer and driver versions before you choose it. - Operators often include a pooler: PGO (
spec.proxy.pgBouncerinPostgresCluster), CloudNativePG (PoolerCRD). Check the fields for the operator version that is running.
4. Operator-managed database clusters
- MUST Make changes through the operator's CR (
PostgresCluster,Cluster…) in Git. Do not edit the StatefulSet / Pod the operator creates, because the operator will overwrite them. - MUST Production has at least 1 replica, DB-level backup (pgBackRest / Barman) to object storage outside the cluster, working PITR, and a tested restore.
- SHOULD Automatic failover (Patroni in PGO, the CNPG mechanism) has been tested at least once in a non-production environment.
- SHOULD Upgrade the major version with the operator's own method (PGO has the
PGUpgradeCRD; CNPG has its own method per version). This is always L3 and needs a backup and a rollback plan. - Put database PVCs on a suitable StorageClass (see
k8s-storage). If the operator already keeps DB replicas → consider fewer replicas at the Longhorn level.
5. Observability
- SHOULD Have metrics and alerts for:
- replication lag;
- connections compared with
max_connections; - long-running transactions / idle in transaction;
- deadlock rate;
- disk and WAL usage;
- latest successful backup.
- SHOULD Turn on
pg_stat_statementsto find heavy queries.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |