ccoalm/ccl-skills / skills/platform-release-engineering
platform-release-engineering
Verified发布 / 灰度 / canary / rollback / rollout / 环境泳道 / promotion gate / 发布值班 SOP(P0·P1 打断排班、是否回滚)→ design or review how a change moves from build to traffic and back safely, including rollout strategy, approval, rollback, secrets, config, and deploy control planes. Skip when the ask is the production release lifecycle — 上线范围确认 / 合并 main / 打 tag / 生产构建 / 发布后 reset → release-coordination; release document substance → release-doc-writer.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 38.5 KB |
| agents/openai.yaml | config | 266 B |
| references/canary-and-rollout-strategy.md | markdown | 9.6 KB |
| references/config-center-via-etcd.md | markdown | 13.3 KB |
| references/custom-control-plane-boundary.md | markdown | 14.6 KB |
| references/deploy-cli-concrete-recipe.md | markdown | 15.3 KB |
| references/deploy-pipeline.md | markdown | 10.7 KB |
| references/env-and-lane-matrix.md | markdown | 4.9 KB |
| references/lane-orchestration-control-plane.md | markdown | 22.2 KB |
| references/multi-region-and-cluster.md | markdown | 5.2 KB |
| references/promotion-gate-and-review.md | markdown | 14.1 KB |
| references/python-package-registry-release.md | markdown | 30.9 KB |
| references/rollback-playbook.md | markdown | 5.5 KB |
| references/secret-and-config-management.md | markdown | 20.1 KB |
| references/version-authority-and-deprecation.md | markdown | 5.1 KB |
references/deploy-cli-concrete-recipe.md
View on GitHub# Deploy CLI — Concrete Recipe
Working implementation of the deploy CLI described in `custom-control-plane-boundary.md`. Distilled from a production CLI that ships every release of a multi-language platform.
## Surface — 5 commands
```
deploy-cli compile Compile source into a build artifact (tar.gz)
deploy-cli build Build a Docker image from the artifact, push to registry
deploy-cli deploy-config Preview the rendered k8s YAML (optionally save to file)
deploy-cli review Create an approval task, optionally wait for the result
deploy-cli deploy Apply the deploy via control plane; stream watch events via WebSocket
```
Each command:
- Takes a small set of named flags (parsed from struct tags via reflection — see CLI builder below).
- Calls the control plane's HTTP API.
- Returns 0 on success, non-zero on failure.
- Prints structured progress to stdout.
Importantly, the CLI **never** applies deploy/update state directly to k8s, service discovery, mesh control plane, or etcd. Deploy-config, review, deploy, rollback, and traffic-policy changes route through the control plane. Source compilation and image build may call pinned project build tools and the image registry behind typed CLI commands, but long-lived deploy/update behavior must stay declarative and auditable.
## Pipeline-name parsing convention
The CI/CD system passes a `pipeline_name` flag in the form `[lane_name] P.S.M XXXX` or `【lane_name】P.S.M XXXX` (Chinese brackets accepted). The CLI parses both `lane_name` and `PSM` from this single string:
```go
laneNameReg := regexp.MustCompile(`[\[【]([0-9a-zA-Z-]+)[]】]`)
psmReg := regexp.MustCompile(`[0-9a-zA-Z-_]+\.[0-9a-zA-Z-_]+\.[0-9a-zA-Z-_]+`)
// Validate: exactly one lane name + exactly one PSM in the string.
// Reject otherwise — "[a] [b] x.y.z" or "[a] x.y.z m.n.o" is ambiguous.
```
Why this convention: external CI/CD platforms (e.g. cloud-provider continuous pipeline) name their pipelines manually with a project-recognizable label. Embedding lane + PSM in the name means the CLI doesn't need separate flags for them, and a glance at the pipeline name tells operators what's deploying where.
If your platform passes lane and PSM as explicit flags, skip this parser. The convention is one less round-trip; explicit flags are clearer.
## Image naming convention
```
<registry-host>/<namespace>/<psm>:<lane-type>-<image-version>
```
Where:
- `registry-host`: private image registry hostname.
- `namespace`: separate namespace per environment tier (e.g. one for offline lanes, one for online lanes — two-tier organization for RBAC + cleanup policies).
- `psm`: service PSM as-is.
- `lane-type`: `prod` (online) or `dev` (offline) — derived from `lane.LaneType`.
- `image-version`: caller-supplied, typically `<git-commit-sha>` or `<release-tag>`.
This naming lets you:
- Filter all online images via the namespace prefix.
- Quickly identify the lane type from the tag.
- Garbage-collect dev images on a shorter retention than prod.
## Compile flow
```
deploy-cli compile --workspace <dir> --pipeline_name "[lane] x.y.z ..."
│
├─ parse pipeline_name → psm, lane
├─ control plane: GetLaneServiceDetail(lane, psm)
│ → buildScript (from cluster config, or service config, or default ./build.sh)
│ → runtime (from env RUNTIME, or cluster config, or default e.g. "go:1.25")
├─ execute embedded compile.sh with env vars:
│ CP_WORKSPACE, PSM, LANE, BUILD_SCRIPT, RUNTIME
│ → runs user's build script (sh "$BUILD_SCRIPT") in workspace
│ → packs ./output/* into output.tar.gz in workspace
├─ generate Dockerfile in workspace from embedded template:
│ FROM <registry>/<runtime>
│ USER <runtime-user>
│ WORKDIR /opt/<org>/app
│ ADD output.tar.gz .
│ ENTRYPOINT ["./bootstrap.sh"]
└─ done; tar.gz + Dockerfile ready in workspace
```
Embedded compile.sh:
```sh
#!/bin/bash
set -e
mkdir -p "$CP_WORKSPACE/.gomod"
cd "$CP_WORKSPACE"
rm -rf output/
sh "$BUILD_SCRIPT" # user's build script writes to ./output
cd output
shopt -s dotglob
tar cvzf output.tar.gz *
shopt -u dotglob
mv output.tar.gz "$CP_WORKSPACE"
```
The user's `build.sh` (in the service repo) is responsible for writing the actual artifact to `./output`. Compile orchestrates: cache setup, output packing, Dockerfile generation.
`bootstrap.sh` lives inside the user's output tarball — it is the service-specific entrypoint. Compile doesn't generate it.
## Build flow
```
deploy-cli build --pipeline_name "[lane] x.y.z ..." --image_version <v>
[--registry_user <u> --registry_pwd <p>]
│
├─ parse pipeline_name → psm, lane
├─ control plane: ListLane(lane) → lane type → image namespace + tag
├─ registry credentials from flag OR env (REGISTRY_USERNAME / REGISTRY_PASSWORD)
├─ registry host from env (REGISTRY_HOST) — required, no default
└─ execute embedded build.sh with env vars:
PSM, LANE, NAMESPACE, IMAGE, REGISTRY_USERNAME, REGISTRY_PASSWORD, REGISTRY_HOST
→ docker login → docker build -t $IMAGE -f Dockerfile .
→ docker push $IMAGE → docker rmi $IMAGE
```
Embedded build.sh:
```sh
#!/bin/bash
set -e
# Use --password-stdin to keep password out of process args / CI command logs.
echo "$REGISTRY_PASSWORD" | docker login -u "$REGISTRY_USERNAME" --password-stdin "$REGISTRY_HOST"
docker build -t "$IMAGE" -f Dockerfile .
docker push "$IMAGE"
docker rmi "$IMAGE"
```
**Never pass passwords with `-p` / `--password` flags** — they appear in `ps`, `/proc/<pid>/cmdline`, container event logs, and CI build output. `--password-stdin` is the universal correct form. Pair with:
- CI secret masking (most CI platforms scrub matching strings from logs).
- Short-lived registry tokens (rotate per CI run if the registry supports it).
- Image registry RBAC: build-time tokens have push-only scope to the target namespace, not org-wide.
Why bash not Go: docker CLI is the path of least friction; Go reimplementations of docker daemon interaction are fragile. Embed the script, execute with `exec.CommandContext`.
## Deploy-config flow (preview YAML)
```
deploy-cli deploy-config --pipeline_name "[lane] x.y.z ..." --image_version <v>
[--use_lane <override>] [--save_path <file>]
│
├─ parse pipeline_name → psm, lane
├─ compute image URI
├─ if special PSM (control plane itself): render embedded YAML template
│ with placeholders <psm> <appName> <laneName> <image>
├─ else: control plane: GetLaneServiceDeployConfig(lane, psm, image)
│ → returns server-rendered YAML
├─ print YAML to stdout
└─ if --save_path given: write to file
```
The preview lets operators eyeball the YAML before deploying — useful for "is this template change going to do what I think" review.
The "special PSM" branch handles the chicken-egg case: when deploying the control plane itself, the control plane API may be unavailable, so the CLI ships an embedded template for those services.
## Review flow
```
deploy-cli review --workspace_id <id> --pipeline_id <id> --pipeline_name "..."
--pipeline_run_id <id> [--need_approve <n>=1] [--expire <dur>=24h]
[--wait_result]
│
├─ validate: pipeline_run_id and pipeline_name required
├─ parse pipeline_name → psm, lane
├─ control plane: CreateReviewTask(
│ lane, psm, workspace_id, pipeline_id, pipeline_name, run_id,
│ expire, need_approve_num)
├─ control plane sends chat-platform card to reviewers (out-of-band)
└─ if --wait_result:
poll GetReviewTask(task_id) every 5s
tolerate up to 5 consecutive errors before failing
until: status != Pending OR elapsed > expire
on result: print approver list + decisions
```
The review task is bound to the **pipeline run id** — one approval per CI/CD pipeline run. The pipeline runs `review --wait_result`, which blocks until approved or expired; then runs `deploy`.
If approval expires or is rejected, `deploy` should not be called. The CI/CD orchestrator enforces this — typically by checking the review command's exit code.
## Deploy flow (the actual k8s apply)
```
deploy-cli deploy --pipeline_name "[lane] x.y.z ..." --image_version <v>
[--use_lane <override>]
│
├─ parse pipeline_name → psm, lane (or use --use_lane override)
├─ compute image URI
├─ open WebSocket to control plane: /v1/deploy_lane_service
│ query: ?psm=<psm>&name=<lane>&image=<image>&interactive=true
├─ control plane:
│ start k8s Deployment apply
│ start k8s.Watch on the Deployment
│ for each watch event: format as JSON → write to WebSocket
├─ CLI reads events from WebSocket with 5-min read deadline (reset per message):
│ event = { type, app_name, ready_replica, expect_replica,
│ condition_* (optional), is_result_event, is_success }
│ each event printed as a status line, e.g.:
│ [MODIFIED] <app>(2/3)
│ [MODIFIED-CONDITION] <app>(2/3), Available - True ...: Deployment available
├─ stop reading on is_result_event=true
└─ exit 0 if is_success, else non-zero
```
This gives the user real-time feedback. A deploy that gets stuck (e.g. ImagePullBackOff) surfaces conditions immediately, not after a timeout.
## CLI builder via reflection (generic)
A reusable pattern for adding new commands without boilerplate:
```go
func MustBuildCommand[P any](name, usage string, action func(*P) error) *cli.Command
```
Define a params struct with `json` + `usage` + `default` tags:
```go
type DeployParams struct {
PipelineName string `json:"pipeline_name" usage:"Pipeline name in '[lane] psm ...' format"`
ImageVersion string `json:"image_version" usage:"Image version tag"`
UseLane string `json:"use_lane" usage:"Override lane (optional)"`
}
var Deploy = MustBuildCommand[DeployParams]("deploy", "Deploy a service", deployHandler)
```
The builder uses reflection to:
- Read each field's `json` tag → flag name.
- Read `usage` tag → flag help text.
- Read `default` tag → default value.
- Detect field type (string/bool/int/int64/float64) → register the right flag type.
Supported types: `string`, `bool`, `int`, `int64`, `float64`. Slices via comma-separated strings, parsed with helpers (`GetStringSlice`, `GetInt64Slice`, etc.).
Adding a new command is one struct + one handler function + one `MustBuildCommand` line.
## Control plane host override
The CLI looks up the control plane at:
```
host = env.<CONTROL_PLANE_HOST> || <default-host>
lane_for_api = env.<CONTROL_PLANE_LANE> // optional; injected as request header
```
So a developer testing against a non-prod control plane (e.g. a dev-lane deployment of the control plane itself) sets one env var. The CLI's HTTP middleware adds the `<lane-header>` to outgoing requests, and the mesh routes them to the right lane of the control plane.
This lets the platform team dogfood the control plane in non-prod lanes before promoting it to prod.
**Security**: open host override is a real attack surface. A compromised CI environment that sets `<CONTROL_PLANE_HOST>` to an attacker-controlled URL would have the deploy CLI build + deploy via malicious rendered YAML. Production lanes MUST add safeguards:
- **Pinned host allowlist**: a build-time-baked list of allowed control-plane hosts. Override permitted only to hosts in the list (typically: prod, pre-prod, staging variants).
- **TLS identity validation**: pin the control-plane TLS cert fingerprint or require a specific CA; reject unexpected certs even on HTTPS.
- **Audience-bound tokens**: CLI's SSO token specifies the target host as audience; refuses to send the token to a different host. Prevents token reuse against a redirected control plane.
- **Break-glass approval for prod override**: setting `<CONTROL_PLANE_HOST>` for a prod lane requires an approval ticket; CLI refuses without it.
- **Audit log on override**: every CLI invocation with non-default host emits an audit entry to the platform's central audit log (not just local).
## Special-PSM bypass (chicken-and-egg)
A handful of platform services (typically the control plane itself + a secrets/auth service) are marked as "special PSM". For those:
- Image namespace logic uses lane-name prefix instead of querying the control plane (which may not be reachable when deploying itself).
- Deploy-config uses an embedded YAML template instead of calling the rendering API.
Limit the special-PSM set to the minimum necessary for self-bootstrap. Adding regular services here creates inconsistent paths and audit gaps.
## What this confirms about `custom-control-plane-boundary.md`
Mapping the recommended boundary to this concrete CLI:
| Boundary rule | CLI evidence |
|---|---|
| Deploy/update calls API; never k8s/etcd directly | ✅ Deploy-config, review, deploy, rollback, and traffic-policy changes go through `cloudapi.Client` HTTP |
| Audit every deploy event | ✅ control plane records each deploy; CLI just initiates |
| WebSocket for live deploy progress | ✅ `/deploy_lane_service` over WS with 5-min read deadline |
| Approval via chat platform | ✅ control plane sends card; CLI's `review --wait_result` polls task status |
| Stateless beyond audit + tasks | ✅ CLI is fully stateless; control plane holds task state |
| Single endpoint, single auth | ✅ `CONTROL_PLANE_HOST` env, lane-header optional |
| No business logic | ✅ CLI is pure orchestration |
| Break-glass kubectl open | ✅ Admin can bypass; CLI doesn't gatekeep cluster access |
| Generic helpers | ✅ Generic `MustBuildCommand[P]` for new commands |
The recommended boundary holds in practice. The CLI is small (~12 files, ~3.5kLOC), maintained by a small platform team, and ships every release.
## Pipeline integration (external CI/CD)
The CLI is designed to be called from a CI/CD pipeline runner (cloud-provider continuous-pipeline / GitLab CI / GitHub Actions / Tekton). A typical pipeline:
```yaml
stages:
- name: compile
script: deploy-cli compile --workspace . --pipeline_name "$PIPELINE_NAME"
- name: build
script: deploy-cli build --pipeline_name "$PIPELINE_NAME" --image_version "$VERSION"
- name: review
script: deploy-cli review --pipeline_run_id "$RUN_ID" --pipeline_name "$PIPELINE_NAME" --wait_result
timeout: 24h # matches review expire
- name: deploy
script: deploy-cli deploy --pipeline_name "$PIPELINE_NAME" --image_version "$VERSION"
```
The pipeline runner provides:
- `PIPELINE_NAME` (encodes lane + PSM)
- `VERSION` (image version, typically `$COMMIT_SHA`)
- `RUN_ID` (unique per pipeline run)
- `WORKSPACE_ID`, `PIPELINE_ID` (for review task linking)
The pipeline runner handles auth (image-registry credentials, control-plane SSO token) via environment variables. The CLI inherits them.
## Verification
- Run `compile + build + deploy-config + deploy` end-to-end against a dev lane; expect each stage idempotent (re-running same command produces same result).
- Override `CONTROL_PLANE_HOST` to a stub server; CLI calls hit the stub, no production traffic.
- Set `CONTROL_PLANE_LANE` to a dev lane; verify the mesh routes CLI's calls to the dev control plane.
- Kill the control plane mid-deploy; CLI surfaces the disconnection cleanly (not hung indefinitely).
- Add a new command via `MustBuildCommand[NewParams]("name", ...)`; expect compile + register + work without other code changes.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |