ccoalm/ccl-skills / skills/platform-release-engineering
platform-release-engineering
Verified发布 / 灰度 / canary / rollback / rollout / 环境泳道 / promotion gate / 发布值班 SOP(P0·P1 打断排班、是否回滚)→ design or review how a change moves from build to traffic and back safely, including rollout strategy, approval, rollback, secrets, config, and deploy control planes. Skip when the ask is the production release lifecycle — 上线范围确认 / 合并 main / 打 tag / 生产构建 / 发布后 reset → release-coordination; release document substance → release-doc-writer.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 38.5 KB |
| agents/openai.yaml | config | 266 B |
| references/canary-and-rollout-strategy.md | markdown | 9.6 KB |
| references/config-center-via-etcd.md | markdown | 13.3 KB |
| references/custom-control-plane-boundary.md | markdown | 14.6 KB |
| references/deploy-cli-concrete-recipe.md | markdown | 15.3 KB |
| references/deploy-pipeline.md | markdown | 10.7 KB |
| references/env-and-lane-matrix.md | markdown | 4.9 KB |
| references/lane-orchestration-control-plane.md | markdown | 22.2 KB |
| references/multi-region-and-cluster.md | markdown | 5.2 KB |
| references/promotion-gate-and-review.md | markdown | 14.1 KB |
| references/python-package-registry-release.md | markdown | 30.9 KB |
| references/rollback-playbook.md | markdown | 5.5 KB |
| references/secret-and-config-management.md | markdown | 20.1 KB |
| references/version-authority-and-deprecation.md | markdown | 5.1 KB |
references/rollback-playbook.md
View on GitHub# Rollback Playbook
## Rollback by strategy
| Rollout strategy | Rollback mechanism | Time to steady |
|---|---|---|
| Canary (traffic-shift) | Set canary weight to 0; drain | 30s - 2 min (mesh propagation) |
| Blue-green | Switch Service selector / VirtualService destination back | seconds |
| Rolling update | Re-deploy previous image digest | minutes (per-pod replacement) |
| In-place mesh policy | Re-apply previous policy YAML | seconds |
| Config-center flip | Set value back | < 1s (push update) |
| Static config in image | Re-deploy with previous image | minutes |
Choose strategy with rollback in mind. A 2-hour rolling update means a 2-hour rollback.
## The rollback command
```
plat rollback --service=<svc> [--to-version=<v>]
```
Without `--to-version`, rolls to the most recently promoted known-good version (the one that ran before the current).
Behavior:
1. Fetch previous known-good image digest from audit log.
2. Initiate the same workflow as a forward deploy, but with the previous digest.
3. Skip review for rollback (it's restoring known state) — unless the rollback target is itself in a known-bad list (e.g. an old version with a CVE).
4. Wait for steady state; verify with smoke probes.
5. Log the rollback event.
## What "previous known-good" means
A version is "known-good" if:
- It was promoted to 100% traffic for at least N hours without abort.
- No P0/P1 incidents attributed to it during that window.
The audit log filters versions by this. Don't naively pick "the version before this one"; that version might be the one that caused yesterday's incident.
## Data migration rollback
Data migrations are the hard case. Three patterns:
### Pattern 1: Forward-compatible (preferred)
Schema changes are designed so old code can read new schema and new code can read old schema. Then:
- Step 1: deploy schema migration.
- Step 2: deploy code that uses new schema.
- Rollback: revert code only; schema stays. Data is intact.
This is the default for non-emergency migrations. Examples:
- Adding a column with a default → forward-compatible.
- Adding a new table → forward-compatible.
- Renaming a column → NOT forward-compatible directly; do "add new + dual-write + cut over + drop old" over multiple releases.
### Pattern 2: Backward-replayable
Schema changes ship with a forward migration AND a back migration. If you must roll back:
- Run the back migration.
- Revert code.
This is dangerous for migrations that drop columns or merge tables (data loss). Use only when forward-compatibility is impossible.
### Pattern 3: Soft-launched migration
The migration is gated by a config flag. Rollback flips the flag without touching schema.
- Slow to clean up old paths.
- Safe rollback.
- Best for migrations of frequently-changing or risky tables.
## Rollback for mesh policy
VirtualService changes propagate via mesh control plane. Rollback:
- Revert the YAML.
- Mesh reloads (seconds).
- In-flight requests handled by old policy until next conn refresh.
For breaking mesh changes (mTLS trust domain), rollback may require coordinated steps across multiple namespaces. Document the order.
## Rollback for secrets
Secrets that have been rotated cannot be un-rotated (old secret is invalidated at source). If a rotation causes outage:
- Old secret is dead.
- Either: roll forward (deploy code that uses new secret).
- Or: rotate again to a third value, deploy code accepting both.
Plan rotation with rollback in mind: keep N-1 valid for a grace period.
## Drill schedule
A rollback path is not real until you've used it. Schedule:
- **Per-quarter**: pick one non-critical service, deploy a no-op change, rollback through the production path. Time the rollback. Note any manual steps that should be automated.
- **Per-major-platform-change**: any change to the rollback mechanism itself (control plane, mesh, registry) triggers a drill on a sample service.
- **Per-incident-postmortem**: if rollback failed or was slow during an incident, mandate a drill on a similar service within 30 days.
If you cannot drill rollback, you cannot rely on rollback.
## Incident-time rollback decision tree
```
P0 incident, suspected cause: recent deploy
│
▼
Was the change pure code / config? ───── yes ──► rollback first, investigate after
│ no
▼
Was the change a data migration? ────── yes ──► consult forward-compatibility pattern;
│ rollback may be unsafe; consider roll-forward fix
│
▼
Was the change mesh / IAM / secret? ─── yes ──► follow per-domain rollback playbook;
may require coordinated steps
```
Rule of thumb: if rollback is safe and the issue is severe, roll back. Investigation can happen with traffic restored.
## What goes wrong
- Audit log doesn't record digest, only tag → rolling back to "tag" pulls the new image again. Always record digest.
- Previous version's image evicted from registry → rollback target unavailable. Retain at least last 10 deployed digests for every prod service.
- Rollback runs but skips the canary-check task → if the previous version had a latent bug, you deploy it blind. Still run smoke probes after rollback.
- Mesh propagation slower than expected → rollback "succeeded" but some pods still see old policy. Verify via mesh access logs, not just control plane.
- Database migration not forward-compatible → code rolled back but DB now ahead. Service may crash on read. Plan migrations with this in mind.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |