cogni-dao/cogni / .claude/skills/promote
promote
VerifiedPromote a sha to preview or production in cogni-template, OR diagnose why a promotion silently failed. Use this skill whenever the user says "/promote", "promote to preview", "promote to production", "ship to prod", "flight this sha", "deploy this PR to preview", "heal preview", "heal preview-operator", "prod is stuck on the old sha", or asks about preview/prod buildSha not advancing. Encodes the operator-API flight/promote endpoints (promotion runs as the operator, never a personal gh credential), lease semantics, affected-only + admin-merge gotchas (bug.0443), and the Monitor poll-loop you must arm so the dispatch isn't fire-and-forget. Trigger this even when the user only names the symptom ("preview is on stale sha", "production didn't update", "flight-preview hard-failed") — the diagnosis playbook lives here. Do NOT trigger for ordinary code/test/build work; this is exclusively for cd-pipeline operations.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 24.5 KB |
SKILL.md
View on GitHub---
name: promote
description: Promote a sha to preview or production in cogni-template, OR diagnose why a promotion silently failed. Use this skill whenever the user says "/promote", "promote to preview", "promote to production", "ship to prod", "flight this sha", "deploy this PR to preview", "heal preview", "heal preview-operator", "prod is stuck on the old sha", or asks about preview/prod buildSha not advancing. Encodes the operator-API flight/promote endpoints (promotion runs as the operator, never a personal gh credential), lease semantics, affected-only + admin-merge gotchas (bug.0443), and the Monitor poll-loop you must arm so the dispatch isn't fire-and-forget. Trigger this even when the user only names the symptom ("preview is on stale sha", "production didn't update", "flight-preview hard-failed") — the diagnosis playbook lives here. Do NOT trigger for ordinary code/test/build work; this is exclusively for cd-pipeline operations.
---
# Promote — preview + production playbook
## Nodes use the verb
A node ships to every env with one call. Only `env` changes. This skill does not own that journey.
```text
POST https://cognidao.org/api/v1/deploy/workloads {nodeId, env, workload, sourceSha}
GET https://cognidao.org/api/v1/deploy/workloads?nodeId=<id>&env=<env>
```
`candidate-a` takes the PR head. `preview` is automatic on merge. `production` is an explicit call with the validated merge commit. Authz: production needs `production_promoter`. preview needs `env_manager`. candidate-a needs `developer` (`can_flight`) or `env_manager` (bug.5574). Read [cicd-e2e-required-sequence](https://cognidao.org/knowledge/cicd-e2e-required-sequence), [deploy-a-declared-workload](https://cognidao.org/knowledge/deploy-a-declared-workload) and [node-service-delivery](https://cognidao.org/knowledge/node-service-delivery).
**Everything below is operator only.** The operator stays on k3s and Argo with its own deploy branches. It ships through `vcs/flight` and `deploy/promote`.
---
You are operating the operator release pipeline. The pipeline is multi-layered (deploy branches, Argo apps, verify-deploy matrix legs) and has several silent-success seams that can make a green workflow lie about what's deployed. **Trust `/version.buildSha` from outside the cluster**, not workflow conclusions and not `/readyz` (Service-level — the old pod can answer it during rolling cutover, returning the prior buildSha while CI thinks rollout is done).
## Promotion runs as the operator — call the API, never personal `gh`
`PROMOTION_RUNS_AS_THE_OPERATOR` ([node-ci-cd-contract.md](https://node-template.cognidao.org/knowledge/node-ci-sovereignty) § Env-promotion): every dispatch is an operator-GitHub-App action keyed off your registered Bearer token. **Do not `gh workflow run` a promote/flight — that uses a personal credential and is not allowed.** For the operator node, drive the ladder over HTTP:
| Step | Call | Authz (action → relation) | Result |
| ------------------------------ | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------- |
| Flight operator to candidate-a | `POST /api/v1/vcs/flight {nodeRef:{nodeId, sourceSha}}` | `node.flight` → `can_flight` (`developer`) | `202` + `candidate-flight.yml` dispatch (candidate-a only) |
| Promote operator to production | `POST /api/v1/deploy/promote {nodeId, env:"production", sourceSha?}` | `node.promote_production` → `can_promote_production` (`production_promoter`; `admin` inherits) | `200` dispatched; **app-digest only, `skip_infra=true`** |
| Reconcile prod infra | `POST /api/v1/deploy/infra-reconcile {nodeId, env:"production"}` | same production action, **operator node only** | `200` dispatched; current app pin preserved |
| Reconcile preview infra | `POST /api/v1/deploy/infra-reconcile {nodeId, env:"preview"}` | `node.manage_envs` → `can_manage_envs` (`env_manager`; `admin` inherits), **operator node only** | `200` dispatched; preview's own app pin preserved |
| Preview | automatic on node `main`-merge | ungated | no agent endpoint — see Preview below |
Need the production grant? Agent files `POST /api/v1/nodes/{id}/access-requests {role:"production_promoter"}`; the node owner approves once with `POST /api/v1/nodes/{id}/developers {agentUserId, decision:"approve", role}`. Result codes: `403 authz_denied` = no grant; `503 authz_unavailable` = the env's OpenFGA store is unbootstrapped (≠ denial); `502 dispatch_failed` = RBAC passed but that env's operator App isn't installed. Full playbook: hub entry [cicd-e2e-required-sequence](https://cognidao.org/knowledge/cicd-e2e-required-sequence).
Endpoints are advertised at `/.well-known/agent.json`. `sourceSha` is **optional** on promote — omit it and the operator resolves the catalog-pinned digest (no manual source-sha picking). The sections below are the **operator-internal mechanics + diagnosis** behind those endpoints; read them when a dispatch silently fails, not to hand-dispatch.
## When to use
- "promote to preview", "promote to production", "/promote", "ship to prod", "flight this sha", "heal preview", "deploy {PR} to preview/prod"
- "preview is still on old sha", "prod didn't update", "flight-preview hard-failed", "buildSha mismatch"
Do NOT use for code review, test fixes, or routine PR work.
## Ground truth — read first when in doubt
- `.github/workflows/flight-preview.yml` — auto preview trigger on `push:main`, manual recovery dispatch
- `.github/workflows/promote-and-deploy.yml` — does the actual promotion + verification (preview AND production)
- `.github/workflows/pr-build.yml` — `pull_request`, `merge_group`, `push:main` and operator `workflow_dispatch` (trusted fork) builds. Every event publishes one identity: `sha-<sourceSha>`. Preview uses `sha-<mainSha>`.
- `scripts/ci/flight-preview.sh` — dispatch the preview flight for the merged SHA (latest-wins; no lease)
- `scripts/ci/resolve-pr-build-images.sh` — single source of truth for "does the `sha-<sha>` image set exist for this merge"
- `scripts/ci/verify-buildsha.sh` — single source of truth for "is this image actually serving"; per-node /version probes + non-Ingress markers
- `scripts/ci/aggregate-decide-outcome.sh` — closes the silent-success seam (bug.0443)
- `scripts/ci/aggregate-rollup.sh` — computes `current-sha = merge-base` of per-node deploy-branch tips; merges per-node `source-sha-by-app.json` preserving unaffected entries
- `scripts/ci/lib/image-tags.sh` — `ALL_TARGETS` / `NODE_TARGETS` from `infra/catalog/*.yaml` (axiom 16, CATALOG_IS_SSOT). Source this; never hardcode target lists.
- Per-env deploy branches for the operator (operator only). The roster is LIVE STATE: read `GET /api/v1/nodes` and `infra/catalog/*.yaml`, never hardcode.
- `.promote-state/` files: `deploy/<env>` holds `current-sha` and `source-sha-by-app.json`; `deploy/<env>-operator` and `deploy/<env>-scheduler-worker` hold `source-sha-by-app.json` only
## Hostname rule (operator only)
- Operator preview → `https://preview.cognidao.org`. Operator production → `https://cognidao.org`.
- Nodes are not in this flow. A node ships through `POST /api/v1/deploy/workloads`; read its release with `GET /api/v1/deploy/workloads`.
- scheduler-worker / migrators → no Ingress, no /version. Use `kubectl rollout status` or trust the verify-deploy job's marker emission.
## Preview promotion (operator only)
**Auto path (default + only path):** a PR merged via merge-queue triggers `flight-preview.yml` on `push:main` → the operator promotes the proven digest. No action, no dispatch. Verify by watching (Monitoring section).
If preview is stuck, it is **not** healed by a hand-dispatch — it is healed by another merge (see the two gotchas below). There is no agent-facing preview-promote endpoint; preview rides the trust the node already earned at candidate-a.
### Affected-only gotcha (task.0376)
`flight-preview` only resolves targets that `pr-build` actually built (`RESOLVED_TARGETS`). Untouched targets are skipped. If preview-operator is broken and the merging PR was scheduler-worker-only, the heal flight resolves scheduler-worker only — preview-operator stays broken. **Fix:** open a follow-up PR that touches the broken node's paths (`nodes/<node>/app/` or its dir-local `AGENTS.md`) and merge it. The build for that merge publishes the missing `sha-<mainSha>` image for that node, and the auto-flight on the resulting `push:main` heals.
> **Migrations amplify this into a silent-success trap.** To land a **DB migration** (registry seed, RLS, backfill) on preview/prod you must deploy the operator image from the commit that CARRIES it — the migrate initContainer only runs migrations baked into the deployed image. Flighting **main head** to heal fails silently when main head is a non-operator change: affected-only skips operator, buildSha never moves, and BOTH workflows report `completed/success`. Find the real target with `git log --oneline <sha> -- nodes/operator/app` and flight THAT commit; verify in the VM Postgres (`SELECT max(id) FROM drizzle."__drizzle_migrations"`), never by workflow conclusion. Seen 2026-08-07: seed `0040` (#1978/`3c4f256`) reached preview+prod only after flighting `3c4f256`, not main head `a5f63fb` (#1980, provision-only). See `/provision-env` gotcha 21.
### Merge outside the queue (bug.0443)
A merge outside the queue (admin or direct) has no `merge_group` build. `flight-preview.yml` then waits up to 15 minutes for the `push:main` build of `pr-build.yml` to publish `sha-<mainSha>`. If the step "Hard-fail when no images found for resolved PR" fires, that build did not publish. Read the `pr-build.yml` run for that `push:main` first. Do NOT try to dispatch a deleted `build-multi-node.yml`.
## Production promotion (operator only)
Explicit, RBAC-gated, operator-dispatched. There is no production auto-trigger (`promote-to-production.yml` was removed in bug.0361).
```bash
# nodeId from the operator nodes registry; sourceSha optional (operator resolves
# the catalog-pinned digest). Bearer key — NEVER a personal gh credential.
curl -sS -X POST "$BASE/api/v1/deploy/promote" \
-H "Authorization: Bearer $COGNI_API_KEY" -H "Content-Type: application/json" \
-d '{"nodeId":"<node-uuid>","env":"production"}'
# 200 {"dispatched":true,...} → operator GitHub App dispatched promote-and-deploy.
# 403 authz_denied → request can_promote_production (grant loop, top of file).
```
`APP_PROMOTE_IS_NO_INFRA`: the API hard-sets `skip_infra=true` — promotion reconciles the **app digest only**. Per-node ESO `ExternalSecret`s are still materialized by the ungated `node-substrate` lane, so no-infra does **not** skip secret reconciliation. A merged substrate change (`infra/compose/**`, a VM-materialized OpenBao/ESO bridge secret, or edge/runtime topology) uses the separate operator-mediated lever:
```bash
curl -sS -X POST "$BASE/api/v1/deploy/infra-reconcile" \
-H "Authorization: Bearer $COGNI_API_KEY" -H "Content-Type: application/json" \
-d '{"nodeId":"<operator-node-uuid>","env":"production"}'
```
Swap `"production"` for `"preview"` to converge the preview lane instead (bug.5409) — same verb, same workflow, gated on `node.manage_envs`. This is shared-lane-only (`preview` | `production`) and operator-node-only. It accepts no SHA, ref, workflow, or mode: the adapter replays that lane's deployed app pin and the existing workflow consumes the merged infrastructure from `main`. The deployment parent is environment-scoped (`NODE_SUBMODULE_PARENT_{OWNER,REPO}`): candidate/test App operations stay in `cogni-test-org`; production App operations stay in `Cogni-DAO`. A `403 authz_denied` currently requires the same one-time production-promoter approval. That role reuse is a two-phase bootstrap bridge—the infra lever must first deploy the OpenFGA model before story.5028's dedicated `production_infra_promoter → can_reconcile_production_infra` grant can take over. Never replace it with personal `gh workflow run … -f skip_infra=false` or an SSH mutation. For node apps, use `POST /api/v1/deploy/workloads`; this verb owns only the shared state/edge substrate.
After dispatch, **always** confirm production `/version.buildSha` actually advanced — see Monitoring. Discover the dispatched run via the unauthenticated GitHub API (`GET /repos/Cogni-DAO/cogni/actions/workflows/promote-and-deploy.yml/runs`); no personal token required.
## Declared workloads
Node workloads do not use the flight and promote calls. They use the verb at the top of this file. For a node, read progress with `GET /api/v1/deploy/workloads` and prove the served `/version.buildSha`. The sequencer log is `{env="<env>",service="sequencer"} |= "node_deploy_step"`.
## Monitoring — Monitor tool, not eyeballing
Every promotion gets a Monitor armed before you walk away. The pattern emits one line per state change so you don't have to poll. Cover ALL terminal states (success, failure, cancelled, hung) — silence is not success.
```bash
# Copy-paste template. Set TARGET_SHA, FLIGHT_ID, and HOSTS for the env.
TARGET_SHA=<expected-buildsha-40-chars>
SHORT=${TARGET_SHA:0:12}
FLIGHT_ID=<flight-preview-or-promote-deploy-run-id-you-just-dispatched>
# Operator only. Nodes are not in this flow. For production use https://cognidao.org.
HOSTS="https://preview.cognidao.org"
# Self-discover the dispatch time so we can find the chained promote-and-deploy run.
DISPATCH_AT=$(gh run view "$FLIGHT_ID" --json createdAt --jq .createdAt)
PD_ID=""
prev_flight="" prev_pd=""
declare -A prev_bs=()
echo "watching flight=$FLIGHT_ID target=$SHORT hosts=$HOSTS dispatched=$DISPATCH_AT"
deadline=$(( $(date +%s) + 2400 ))
while [ "$(date +%s)" -lt "$deadline" ]; do
fs=$(gh run view "$FLIGHT_ID" --json status,conclusion --jq '"\(.status)/\(.conclusion)"' 2>/dev/null || echo "api-fail/")
[ "$fs" != "$prev_flight" ] && echo "[flight $FLIGHT_ID] $fs" && prev_flight="$fs"
if [ -z "$PD_ID" ]; then
PD_ID=$(gh run list --workflow=promote-and-deploy.yml --limit 5 --json databaseId,createdAt \
--jq "[.[] | select(.createdAt > \"$DISPATCH_AT\")] | .[0].databaseId // empty" 2>/dev/null || echo "")
[ -n "$PD_ID" ] && echo "[discovered promote-and-deploy] $PD_ID"
fi
if [ -n "$PD_ID" ]; then
ps=$(gh run view "$PD_ID" --json status,conclusion --jq '"\(.status)/\(.conclusion)"' 2>/dev/null || echo "api-fail/")
[ "$ps" != "$prev_pd" ] && echo "[promote-deploy $PD_ID] $ps" && prev_pd="$ps"
fi
all_match=1
for host in $HOSTS; do
bs=$(curl -sk --max-time 8 "$host/version" 2>/dev/null \
| python3 -c 'import json,sys
try: print(json.loads(sys.stdin.read()).get("buildSha","")[:12])
except: print("")' 2>/dev/null || echo "")
if [ "$bs" != "${prev_bs[$host]:-}" ]; then
echo "[$host /version buildSha] ${bs:-<empty>}"
prev_bs[$host]=$bs
fi
[ "$bs" = "$SHORT" ] || all_match=0
done
if [ "$all_match" = "1" ]; then
echo "[DONE] all hosts match $SHORT"
exit 0
fi
sleep 30
done
echo "[TIMEOUT] flight=$prev_flight promote=$prev_pd buildshas=$(declare -p prev_bs)"
exit 2
```
Wrap in a `Monitor` tool call with `timeout_ms` 2400000 (40 min) and a descriptive name like `"heal preview-operator: flight {ID} → promote-deploy → all preview /version → {SHORT}"`.
### When `/version.buildSha` doesn't advance — Loki, not SSH
`/version.buildSha` answers "is the new app pod serving" but says nothing
about why a host-side compose container (alloy, autoheal, db-backup,
alloy-k8s-events) is crash-looping. Don't SSH-tail. Both layers ship to the
same Loki:
```logql
# 1. App pod startup / env validation (k3s pod logs)
{namespace="cogni-<env>",pod=~"<service>-.*"} |~ "Error|EnvValidation|panic|started|ready"
# 2. Host-side compose container stdout (host alloy → Loki)
{env="<env>",service="<compose-svc>"} | json | level=~"error|warn"
# compose-svc ∈ {litellm, caddy, temporal, autoheal, db-backup, alloy-k8s-events}
# 3. Kubernetes Events stream (pod OOMKilled, probe failures, evictions, rollout)
{env="<env>",source="k8s-events"} | json
# Use this when a node restarted but app stderr is empty — kubelet's reason
# (OOMKilled, ImagePullBackOff, Liveness probe failed) only lives in events,
# not in container logs.
# 4. Argo control-plane (operator only) — "the operator app never appeared" / "stuck OutOfSync"
{namespace="argocd",source="k8s-events",kind="ApplicationSet"} | json
# msg: created/Deleted Application "<env>-<node>" — proves the AppSet IS
# generating (a `created`+`Deleted` pair on the same app = it generated then
# pruned: a catalog/overlay condition, NOT a stuck controller).
{namespace="argocd",source="k8s-events",kind="Application",reason=~"OperationStarted|OperationCompleted|Failed"} | json
# msg: "Initiated automated sync to <sha>" → "sync operation to <sha> succeeded"
# — Argo's reconcile state for that one app, by sha.
```
`source="k8s-events"` requires the `alloy-k8s-events` compose service from
PR #1233 to be deployed in the env you're querying. If empty: check
`{service="alloy-k8s-events", env="<env>"}` for crash logs first.
**Reaching for `kubectl get applications` / `kubectl describe applicationset`
is the anti-pattern.** The `applicationset-controller` and
`application-controller` emit every generate / create / prune / sync
transition as a k8s Event — already in Loki via stream #4 above. `kind=ApplicationSet`
answers "is the AppSet generating this app at all"; `kind=Application`
answers "is Argo syncing it to the expected sha." No SSH, no kubeconfig — which
is also the SOC2 posture (read-only, audited, zero standing cluster credential).
New compose services don't ship logs until added to the host-alloy allowlist
regex in `infra/compose/runtime/configs/alloy-config.{,metrics.}alloy`. If
`{service=<svc>}` returns silence post-deploy, fix the regex before
diagnosing further — it's not Loki, it's the filter.
## Preview is latest-wins (no lease)
There is **no preview review-lease** (removed 2026-06-16, `kill-preview-review-lease`). Every merge to main dispatches a fresh preview flight; the latest one wins. There is nothing to "unlock" and no `reviewing` hold to wait on — preview always tracks the latest merged SHA. Serialization under bursty merges is handled by the `flight-preview` and `promote-deploy-<env>-<nodes>` workflow concurrency groups. If preview looks stale, it's an affected-only or admin-merge image gap (see the two gotchas above) or a failed flight — not a lease; re-merge or re-dispatch `flight-preview.yml`.
## Failure modes — first-step diagnosis
| Symptom | First diagnosis |
| ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| flight-preview red at "Hard-fail when no images found for resolved PR" | The `push:main` build did not publish `sha-<mainSha>` within the 15-minute wait (bug.0443). Read that `pr-build.yml` run. |
| aggregate-preview red, "no cell reported promoted=true" | Same missing `sha-<mainSha>` build, OR the sha has zero images. Verify with `resolve-pr-build-images.sh` (preflight #2). |
| aggregate-production red, "Axiom 19 contradiction: scheduler-worker" | bug.0443 fix in `verify-buildsha.sh` is missing/reverted. The `NON_INGRESS_NODES` marker-emission block must be present. |
| verify-buildsha timeout 90s | Pod cutover incomplete; usually transient. Re-check `/version` directly in 60s. If still wrong, check Argo app revision matches deploy-branch tip. |
| `verify-deploy` green but `/version.buildSha` still old | CDN/edge cache, OR you hit `/readyz` (which the old pod still answers) instead of `/version`. Always use `/version.buildSha` for verification, never `/readyz`. |
| Production "succeeded" but only some targets advanced | Affected-only — the `nodes` input was scoped, or the source sha's image set did not cover all targets. Read `.promote-state/source-sha-by-app.json` on `deploy/production-operator` and `deploy/production-scheduler-worker`; `current-sha` lives only on `deploy/production`. Nodes have no deploy branch; read their release with `GET /api/v1/deploy/workloads`. |
| prod-pd: every `verify-deploy (<node>)` red at `Resolve cell state` with `bash: app-src/scripts/ci/<script>: No such file or directory` exit 127 | `source_sha` predates the verify-deploy script tree. Don't use `deploy/preview:.promote-state/current-sha` — `aggregate-rollup.sh` writes the merge-base of per-node tips, which under affected-only divergence regresses behind script additions. Re-dispatch using preflight #4's picker (newest preview-{node} promotion sha). |
## Discipline
- **`/version.buildSha` is the only deploy verification that matters.** CI conclusions can lie. `/readyz` can lie (old pod answers during rolling cutover).
- **Per-target truth, not per-workflow (operator only).** Check each operator deploy branch's `.promote-state/current-sha` independently. For a node, read `GET /api/v1/deploy/workloads`.
- **A merge outside the queue has no `merge_group` build.** Preview then waits on the `push:main` build. If that build fails, file a bug and ship a follow-up merge. Do not assume the gap heals itself.
- **Don't dispatch without arming a Monitor.** Fire-and-forget is how outages survive shift changes.
- **Catalog-as-SSOT for target lists.** Source `scripts/ci/lib/image-tags.sh`; never inline a node list. `resy` outlived the node by months precisely because it was inlined here.
- **Promotion runs as the operator, never personal `gh`.** Flight/promote via the API (`vcs/flight`, `deploy/promote`); the operator GitHub App dispatches. `gh workflow run` of a promote/flight uses a personal credential and is out of bounds. Read-only `gh run view/list` for watching a run is fine.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |