ccoalm/ccl-skills / skills/platform-release-engineering
platform-release-engineering
Verified发布 / 灰度 / canary / rollback / rollout / 环境泳道 / promotion gate / 发布值班 SOP(P0·P1 打断排班、是否回滚)→ design or review how a change moves from build to traffic and back safely, including rollout strategy, approval, rollback, secrets, config, and deploy control planes. Skip when the ask is the production release lifecycle — 上线范围确认 / 合并 main / 打 tag / 生产构建 / 发布后 reset → release-coordination; release document substance → release-doc-writer.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 38.5 KB |
| agents/openai.yaml | config | 266 B |
| references/canary-and-rollout-strategy.md | markdown | 9.6 KB |
| references/config-center-via-etcd.md | markdown | 13.3 KB |
| references/custom-control-plane-boundary.md | markdown | 14.6 KB |
| references/deploy-cli-concrete-recipe.md | markdown | 15.3 KB |
| references/deploy-pipeline.md | markdown | 10.7 KB |
| references/env-and-lane-matrix.md | markdown | 4.9 KB |
| references/lane-orchestration-control-plane.md | markdown | 22.2 KB |
| references/multi-region-and-cluster.md | markdown | 5.2 KB |
| references/promotion-gate-and-review.md | markdown | 14.1 KB |
| references/python-package-registry-release.md | markdown | 30.9 KB |
| references/rollback-playbook.md | markdown | 5.5 KB |
| references/secret-and-config-management.md | markdown | 20.1 KB |
| references/version-authority-and-deprecation.md | markdown | 5.1 KB |
references/env-and-lane-matrix.md
View on GitHub# Environment and Lane Matrix
## Why "lane" instead of "environment"
"Environment" suggests a small fixed set: `dev`, `staging`, `prod`. Real platforms grow:
- Per-developer lanes (one developer's in-progress version of a couple of services).
- Per-integration-test lanes (ephemeral, deleted after test run).
- Long-lived stages (`dev`, `pre`, `staging`, `prod`).
- Canary subsets within prod (`prod-canary-1pct`).
- Shadow traffic targets (`prod-shadow`).
- Stress-test lanes.
All share routing primitives (registry tag or k8s label plus mesh routing — see `platform-service-connectivity`). The release skill treats them uniformly: each lane is a deploy target.
## Lane object shape
```yaml
lane:
name: <unique-name>
parent: <fallback-lane> # if request lane has no instances, fall to parent
type: long-lived | ephemeral | canary | shadow | stress
expires_at: <timestamp> # required for ephemeral; absent for long-lived
owner: <team-or-individual>
services: [<service-names>] # services deployed in this lane
traffic_weight: <0-100> # only for canary/shadow subsets within a parent
protected: true | false # protected lanes cannot be deleted without explicit override
```
Long-lived lanes are protected. Ephemeral lanes auto-delete past expiry.
## Standard long-lived lanes
| Lane | Purpose | Traffic source |
|---|---|---|
| `prod` | live user traffic | external clients, ingress |
| `pre` | pre-production validation | internal smoke + load tests |
| `staging` | pre-pre, integration tests | CI |
| `dev` | shared dev environment | engineers, internal tools |
Some platforms add `gray` or `inner-prod` for internal employee dogfooding. Add only when there's a real signal it produces; don't proliferate.
## Per-developer lanes
A developer working on service A wants to test against the real `prod` versions of B, C, D, while running their own A. Lane mechanism:
1. Developer deploys their A under lane `dev-<name>`.
2. Their requests carry `<lane-header>: dev-<name>`.
3. Mesh VirtualService routes:
- A: to `dev-<name>` subset (their version).
- B, C, D: no `dev-<name>` subset → fall back to `prod` (or `dev` shared, depending on policy).
4. Developer's `dev-<name>` lane expires after N days (e.g. 7) unless renewed.
This lets one developer iterate without cluster-wide effect. Multiplied by team size, it works only if lane creation is cheap (control-plane API, not manual k8s).
## Ephemeral lanes for CI
Integration tests that need real services (not mocks) create a lane per test run:
```
lane name: it-<test-id>-<commit>
parent: pre
type: ephemeral
expires_at: now + 4h
services: [services-under-test]
```
CI deploys the under-test versions to the lane, runs tests with `<lane-header>: it-...`, then deletes the lane. Lane lifetime is bounded by test duration + grace.
## Canary lanes
A canary is a subset of `prod` with its own subset name:
```
lane name: prod-canary-v2.5.0
parent: prod
type: canary
traffic_weight: 1 # 1% to start
services: [the-one-being-rolled-out]
```
VirtualService weight shifts from canary to stable over the rollout window. See `canary-and-rollout-strategy.md`.
## Shadow lanes
Shadow receives a mirror of prod traffic without affecting user response:
```
lane name: prod-shadow
parent: prod
type: shadow
services: [services-being-shadowed]
```
Shadow target's responses are dropped. Metrics from shadow MUST be tagged so they don't contaminate prod SLIs.
## Cluster / region mapping
Lanes live in clusters. Common topologies:
| Topology | When |
|---|---|
| Single cluster, all lanes | Small platform; everything fits |
| Per-region cluster, lanes mirrored | Geographic latency or compliance |
| Per-lane-type cluster | Heavy stress / shadow isolation |
| Per-team cluster | Late stage; high cost; only when noise isolation is critical |
Document the mapping. Each lane declares its cluster. Cross-cluster lane traffic requires the platform's chosen cross-cluster discovery or mesh federation path (`platform-service-connectivity`).
## Lane policies
Beyond pure routing, lanes carry policy:
- **Resource quotas**: dev / canary lanes get smaller pod counts.
- **Data isolation**: prod and pre have different DB instances; sharing is a leak.
- **Network egress**: dev may have looser external access; prod may have stricter.
- **Observability retention**: prod traces kept 30 days; ephemeral lanes 1 day.
Policies are enforced by namespace labels + admission controllers + mesh AuthorizationPolicy.
## Verification
- `lane list` returns all lanes with type and owner. `lane get <name>` returns services + expiry + traffic weight.
- A new ephemeral lane is reachable end-to-end within minutes of creation.
- A lane deleted leaves no orphaned routing/discovery resources (Deployment, Service, VirtualService subset, registry tag or k8s label selector).
- A request with an unknown lane header falls back to the documented parent lane and logs the fallback (so silent fallback can be detected).
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |