Skip to content

khuynh22/agent-dev-team / skills/release-and-rollback

release-and-rollback

Verified

Release and Rollback

Plan the rollback before the rollout. A change that cannot be undone or observed is not ready, whatever its tests say.

Process

  1. Answer the rollback question first. What single action undoes this, how long does it take, and can someone do it at 3am from a phone? If there is no answer, stop here and escalate; that is a design problem, not a release problem.

  2. Sequence so each step is independently reversible. The safe shape for a data change is four deploys, not one:

    1. Schema change that is backward compatible (add, do not rename or drop).
    2. Code that writes both shapes and reads the old one.
    3. Code that reads the new shape.
    4. Removal of the old shape, after usage is measured at zero.

    Four reversible deploys beat one irreversible one every time.

  3. Choose the rollout shape and the abort criterion together. Canary, percentage, ring, or flag. The abort criterion is a number measured from a named query, decided before the rollout starts. Deciding it in the moment, while looking at a graph you want to look good, does not work.

  4. Make it observable before it ships. You must be able to answer from telemetry alone: is the new path being taken, is it succeeding, how long does it take. Write the three queries down. See references/observability-checklist.md.

  5. Feature flags with a lifecycle. A flag has an owner, a default, a removal date, and someone who can flip it without a deploy. A flag with no removal plan becomes permanent configuration and doubles the state space forever.

  6. Write the runbook for the alert, not for the service. The reader is woken up and has five minutes.

  7. Check the pipeline. Reproducible build, pinned toolchain, no secret in scope for untrusted pull-request code, and a green run you can point at.

Release plan

## Release: <change>
- **Rollback:** <single action> — takes <duration> — executed in staging: yes/no
- **Steps:** <ordered, each independently reversible>
- **Flag:** <name, default, who can flip it, removal date>
- **Rollout:** <canary %, soak duration, then next stage>
- **Abort if:** <metric crosses <threshold>, from query `<q>`>
- **Observability:** <the three queries: taken / succeeding / duration>
- **Runbook:** <path>
- **Blast radius if wrong:** <who, how many, for how long>

Runbook

# Alert: <name>
**Means:** <the user-visible symptom>
**Check first:** <one dashboard or query>
**Common causes:** <ranked, each with its distinguishing signal>
**Mitigation:** <the action that stops the bleeding, before diagnosis>
**Escalate to:** <who, and when>

Migrations

  • Run it forward against a copy of realistic production data, not a fixture.
  • Execute the rollback at least once, somewhere real. An untested rollback is a plan, not a capability.
  • Long-running migrations run in batches with progress and a resume point.
  • Check locking on large tables. "It only adds a column" is true right up until it takes a lock and the site stops.

Verification

  • Rollback is a single documented action with a known duration, executed at least once.
  • The abort criterion is a number from a named query.
  • The three observability questions have concrete queries written down.
  • Every alert added has a runbook and an owner.
  • Every flag has a removal date.

Red flags

ThoughtReality
"We can roll forward if it breaks"Roll-forward under pressure turns a small outage into a long one.
"The migration is safe, it only adds a column"With a default, on a large table, with a lock. Check.
"We will watch it after deploy"Watch what? Name the query before you ship.
"It passed CI so it is ready"CI proves the code. It says nothing about rollback or observability.
"Alert on CPU"Alert on the symptom a user feels. CPU is a dashboard.
"We will remove the flag later"Give it a date now, or it is permanent.
"Rollback should work"Should is not evidence. Run it.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 197kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall