ccoalm/ccl-skills / skills/platform-release-engineering
platform-release-engineering
Verified发布 / 灰度 / canary / rollback / rollout / 环境泳道 / promotion gate / 发布值班 SOP(P0·P1 打断排班、是否回滚)→ design or review how a change moves from build to traffic and back safely, including rollout strategy, approval, rollback, secrets, config, and deploy control planes. Skip when the ask is the production release lifecycle — 上线范围确认 / 合并 main / 打 tag / 生产构建 / 发布后 reset → release-coordination; release document substance → release-doc-writer.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 38.5 KB |
| agents/openai.yaml | config | 266 B |
| references/canary-and-rollout-strategy.md | markdown | 9.6 KB |
| references/config-center-via-etcd.md | markdown | 13.3 KB |
| references/custom-control-plane-boundary.md | markdown | 14.6 KB |
| references/deploy-cli-concrete-recipe.md | markdown | 15.3 KB |
| references/deploy-pipeline.md | markdown | 10.7 KB |
| references/env-and-lane-matrix.md | markdown | 4.9 KB |
| references/lane-orchestration-control-plane.md | markdown | 22.2 KB |
| references/multi-region-and-cluster.md | markdown | 5.2 KB |
| references/promotion-gate-and-review.md | markdown | 14.1 KB |
| references/python-package-registry-release.md | markdown | 30.9 KB |
| references/rollback-playbook.md | markdown | 5.5 KB |
| references/secret-and-config-management.md | markdown | 20.1 KB |
| references/version-authority-and-deprecation.md | markdown | 5.1 KB |
references/multi-region-and-cluster.md
View on GitHub# Multi-Region and Multi-Cluster
When and how to leave a single cluster, and what changes about release flow.
## Why leave a single cluster
Drivers (most common first):
- **Failure-domain isolation**: a single cluster outage takes down everything. DR needs separate failure domains.
- **Geographic latency**: users on another continent see > 200ms RTT.
- **Compliance / data residency**: certain data must stay in a jurisdiction.
- **Resource scale**: control-plane limits, node count, etcd size.
- **Noise isolation**: noisy workloads (heavy batch, stress tests) starve neighbors.
If none of these bite, stay single-cluster. Multi-cluster has real ongoing cost.
## Cluster topology patterns
### Active-passive (DR)
- Primary cluster serves all traffic.
- Secondary cluster runs the same workloads at minimum capacity (or scaled down to 0).
- Failover triggered by a documented condition (primary unreachable, latency > X, manual decision).
Recovery objective:
- RPO (recovery point): how much data loss is acceptable.
- RTO (recovery time): how long until secondary serves.
Test with DR drills (see below).
### Active-active
- Both clusters serve traffic concurrently.
- Geographic routing sends each user to the nearest cluster.
- Cross-cluster traffic happens for stateful services that pin to one region.
Higher complexity, lower RTO. Requires:
- Idempotent writes (or single-writer per record).
- Conflict resolution (CRDT, last-writer-wins, vector clocks — pick one and commit).
- Cross-region observability join.
### Per-purpose isolation
- Cluster A: latency-critical services.
- Cluster B: batch / async / stress.
- Cluster C: external-facing edge.
Routes drawn at ingress / mesh boundary. Easier than multi-region.
## Registry federation
If service A in cluster X needs to call service B in cluster Y:
- Both clusters' registries (Nacos / Consul) federate: writes in one are visible in the other (eventually).
- Mesh trust domain must be configured pairwise: A's identity is recognized in Y.
- VirtualService can target a remote-cluster subset.
Without federation, you fall back to direct hostname routing (k8s DNS doesn't cross clusters) — and lose lane-routing. Federation is the right answer.
## Mesh trust domains
mTLS identity is namespaced by trust domain. Cross-cluster mTLS requires:
- Each cluster has a distinct trust domain (e.g. `cluster-1.local`, `cluster-2.local`).
- Cross-cluster AuthorizationPolicy lists peer trust domains.
- Root CA either shared or bridged.
Misconfigured trust domain = "service A cannot reach service B" for unclear reasons.
## Multi-cluster lane semantics
A lane can live in one cluster or many:
| Scenario | Lane spec |
|---|---|
| `dev-alice` for a developer | one cluster |
| `prod` user traffic | sharded across regions, lane-tagged per region |
| `prod-canary` for global rollout | per-region canary lanes; promote independently |
Track which clusters each lane spans. Cross-cluster lane traffic uses the platform's chosen discovery path (registry federation, k8s-native EndpointSlice export, or mesh federation) plus mesh trust.
## DR drill
Annual minimum; quarterly preferred. Drill scope:
```
1. Pick a non-critical service.
2. Mark its primary cluster "unavailable" (without actually taking it down, if possible).
3. Trigger failover to secondary.
4. Run synthetic user traffic against the failed-over service.
5. Measure: RTO (time to first successful response), RPO (data delta), correctness.
6. Switch back.
7. Document gaps; file remediation tickets.
```
If RTO / RPO is worse than declared, fix the gap before claiming the SLA.
Reality check: most platforms have an undrilled DR plan that doesn't work. The first attempt usually surfaces 3-5 blockers. Plan for that.
## Release flow changes
Multi-cluster makes release harder:
- Image must be available in each region's registry (replicate or use a global registry with regional cache).
- Canary in one region first, then expand. Cross-region promotion is a separate gate.
- Config-center: per-region instance, replicated; or global with regional read replicas.
- Secret store: regional, with cross-region replication where compliant.
A change is "fully deployed" only when it's been canary'd and promoted in every region.
## Cross-region observability
Without it, you cannot reason about a global service.
- Traces stitched across regions (OTel collectors federate or push to a global backend).
- Metrics scraped per region, aggregated globally.
- Logs indexed per region with global search (federated query or central aggregation).
Cross-region trace gaps are common; verify in a drill.
## Cost
Multi-region doubles or triples:
- Compute capacity (idle secondary).
- Network egress (cross-region replication).
- Operational complexity (oncall, runbooks, drills).
If business doesn't need it, don't pay it.
## Verification
- Failover drill executed in the last quarter; gaps documented.
- Cross-region call traced end-to-end; trust domains and mesh policy verified.
- Lane `prod` has instances in expected regions; the chosen discovery view matches deployment.
- Image rollback can pull from the regional registry; not dependent on a single source.
- DR runbook exists, was practiced, and reaches steady state within stated RTO.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |