Skip to content

cogni-dao/cogni / .claude/skills/promote

promote

Verified

Promote a sha to preview or production in cogni-template, OR diagnose why a promotion silently failed. Use this skill whenever the user says "/promote", "promote to preview", "promote to production", "ship to prod", "flight this sha", "deploy this PR to preview", "heal preview", "heal preview-operator", "prod is stuck on the old sha", or asks about preview/prod buildSha not advancing. Encodes the operator-API flight/promote endpoints (promotion runs as the operator, never a personal gh credential), lease semantics, affected-only + admin-merge gotchas (bug.0443), and the Monitor poll-loop you must arm so the dispatch isn't fire-and-forget. Trigger this even when the user only names the symptom ("preview is on stale sha", "production didn't update", "flight-preview hard-failed") — the diagnosis playbook lives here. Do NOT trigger for ordinary code/test/build work; this is exclusively for cd-pipeline operations.

Promote — preview + production playbook

Nodes use the verb

A node ships to every env with one call. Only env changes. This skill does not own that journey.

POST https://cognidao.org/api/v1/deploy/workloads   {nodeId, env, workload, sourceSha}
GET  https://cognidao.org/api/v1/deploy/workloads?nodeId=<id>&env=<env>

candidate-a takes the PR head. preview is automatic on merge. production is an explicit call with the validated merge commit. Authz: production needs production_promoter. preview needs env_manager. candidate-a needs developer (can_flight) or env_manager (bug.5574). Read cicd-e2e-required-sequence, deploy-a-declared-workload and node-service-delivery.

Everything below is operator only. The operator stays on k3s and Argo with its own deploy branches. It ships through vcs/flight and deploy/promote.


You are operating the operator release pipeline. The pipeline is multi-layered (deploy branches, Argo apps, verify-deploy matrix legs) and has several silent-success seams that can make a green workflow lie about what's deployed. Trust /version.buildSha from outside the cluster, not workflow conclusions and not /readyz (Service-level — the old pod can answer it during rolling cutover, returning the prior buildSha while CI thinks rollout is done).

Promotion runs as the operator — call the API, never personal gh

PROMOTION_RUNS_AS_THE_OPERATOR (node-ci-cd-contract.md § Env-promotion): every dispatch is an operator-GitHub-App action keyed off your registered Bearer token. Do not gh workflow run a promote/flight — that uses a personal credential and is not allowed. For the operator node, drive the ladder over HTTP:

StepCallAuthz (action → relation)Result
Flight operator to candidate-aPOST /api/v1/vcs/flight {nodeRef:{nodeId, sourceSha}}node.flight → can_flight (developer)202 + candidate-flight.yml dispatch (candidate-a only)
Promote operator to productionPOST /api/v1/deploy/promote {nodeId, env:"production", sourceSha?}node.promote_production → can_promote_production (production_promoter; admin inherits)200 dispatched; app-digest only, skip_infra=true
Reconcile prod infraPOST /api/v1/deploy/infra-reconcile {nodeId, env:"production"}same production action, operator node only200 dispatched; current app pin preserved
Reconcile preview infraPOST /api/v1/deploy/infra-reconcile {nodeId, env:"preview"}node.manage_envs → can_manage_envs (env_manager; admin inherits), operator node only200 dispatched; preview's own app pin preserved
Previewautomatic on node main-mergeungatedno agent endpoint — see Preview below

Need the production grant? Agent files POST /api/v1/nodes/{id}/access-requests {role:"production_promoter"}; the node owner approves once with POST /api/v1/nodes/{id}/developers {agentUserId, decision:"approve", role}. Result codes: 403 authz_denied = no grant; 503 authz_unavailable = the env's OpenFGA store is unbootstrapped (≠ denial); 502 dispatch_failed = RBAC passed but that env's operator App isn't installed. Full playbook: hub entry cicd-e2e-required-sequence.

Endpoints are advertised at /.well-known/agent.json. sourceSha is optional on promote — omit it and the operator resolves the catalog-pinned digest (no manual source-sha picking). The sections below are the operator-internal mechanics + diagnosis behind those endpoints; read them when a dispatch silently fails, not to hand-dispatch.

When to use

  • "promote to preview", "promote to production", "/promote", "ship to prod", "flight this sha", "heal preview", "deploy {PR} to preview/prod"
  • "preview is still on old sha", "prod didn't update", "flight-preview hard-failed", "buildSha mismatch"

Do NOT use for code review, test fixes, or routine PR work.

Ground truth — read first when in doubt

  • .github/workflows/flight-preview.yml — auto preview trigger on push:main, manual recovery dispatch
  • .github/workflows/promote-and-deploy.yml — does the actual promotion + verification (preview AND production)
  • .github/workflows/pr-build.yml — pull_request, merge_group, push:main and operator workflow_dispatch (trusted fork) builds. Every event publishes one identity: sha-<sourceSha>. Preview uses sha-<mainSha>.
  • scripts/ci/flight-preview.sh — dispatch the preview flight for the merged SHA (latest-wins; no lease)
  • scripts/ci/resolve-pr-build-images.sh — single source of truth for "does the sha-<sha> image set exist for this merge"
  • scripts/ci/verify-buildsha.sh — single source of truth for "is this image actually serving"; per-node /version probes + non-Ingress markers
  • scripts/ci/aggregate-decide-outcome.sh — closes the silent-success seam (bug.0443)
  • scripts/ci/aggregate-rollup.sh — computes current-sha = merge-base of per-node deploy-branch tips; merges per-node source-sha-by-app.json preserving unaffected entries
  • scripts/ci/lib/image-tags.sh — ALL_TARGETS / NODE_TARGETS from infra/catalog/*.yaml (axiom 16, CATALOG_IS_SSOT). Source this; never hardcode target lists.
  • Per-env deploy branches for the operator (operator only). The roster is LIVE STATE: read GET /api/v1/nodes and infra/catalog/*.yaml, never hardcode.
  • .promote-state/ files: deploy/<env> holds current-sha and source-sha-by-app.json; deploy/<env>-operator and deploy/<env>-scheduler-worker hold source-sha-by-app.json only

Hostname rule (operator only)

  • Operator preview → https://preview.cognidao.org. Operator production → https://cognidao.org.
  • Nodes are not in this flow. A node ships through POST /api/v1/deploy/workloads; read its release with GET /api/v1/deploy/workloads.
  • scheduler-worker / migrators → no Ingress, no /version. Use kubectl rollout status or trust the verify-deploy job's marker emission.

Preview promotion (operator only)

Auto path (default + only path): a PR merged via merge-queue triggers flight-preview.yml on push:main → the operator promotes the proven digest. No action, no dispatch. Verify by watching (Monitoring section).

If preview is stuck, it is not healed by a hand-dispatch — it is healed by another merge (see the two gotchas below). There is no agent-facing preview-promote endpoint; preview rides the trust the node already earned at candidate-a.

Affected-only gotcha (task.0376)

flight-preview only resolves targets that pr-build actually built (RESOLVED_TARGETS). Untouched targets are skipped. If preview-operator is broken and the merging PR was scheduler-worker-only, the heal flight resolves scheduler-worker only — preview-operator stays broken. Fix: open a follow-up PR that touches the broken node's paths (nodes/<node>/app/ or its dir-local AGENTS.md) and merge it. The build for that merge publishes the missing sha-<mainSha> image for that node, and the auto-flight on the resulting push:main heals.

Migrations amplify this into a silent-success trap. To land a DB migration (registry seed, RLS, backfill) on preview/prod you must deploy the operator image from the commit that CARRIES it — the migrate initContainer only runs migrations baked into the deployed image. Flighting main head to heal fails silently when main head is a non-operator change: affected-only skips operator, buildSha never moves, and BOTH workflows report completed/success. Find the real target with git log --oneline <sha> -- nodes/operator/app and flight THAT commit; verify in the VM Postgres (SELECT max(id) FROM drizzle."__drizzle_migrations"), never by workflow conclusion. Seen 2026-08-07: seed 0040 (#1978/3c4f256) reached preview+prod only after flighting 3c4f256, not main head a5f63fb (#1980, provision-only). See /provision-env gotcha 21.

Merge outside the queue (bug.0443)

A merge outside the queue (admin or direct) has no merge_group build. flight-preview.yml then waits up to 15 minutes for the push:main build of pr-build.yml to publish sha-<mainSha>. If the step "Hard-fail when no images found for resolved PR" fires, that build did not publish. Read the pr-build.yml run for that push:main first. Do NOT try to dispatch a deleted build-multi-node.yml.

Production promotion (operator only)

Explicit, RBAC-gated, operator-dispatched. There is no production auto-trigger (promote-to-production.yml was removed in bug.0361).

# nodeId from the operator nodes registry; sourceSha optional (operator resolves
# the catalog-pinned digest). Bearer key — NEVER a personal gh credential.
curl -sS -X POST "$BASE/api/v1/deploy/promote" \
  -H "Authorization: Bearer $COGNI_API_KEY" -H "Content-Type: application/json" \
  -d '{"nodeId":"<node-uuid>","env":"production"}'
# 200 {"dispatched":true,...} → operator GitHub App dispatched promote-and-deploy.
# 403 authz_denied → request can_promote_production (grant loop, top of file).

APP_PROMOTE_IS_NO_INFRA: the API hard-sets skip_infra=true — promotion reconciles the app digest only. Per-node ESO ExternalSecrets are still materialized by the ungated node-substrate lane, so no-infra does not skip secret reconciliation. A merged substrate change (infra/compose/**, a VM-materialized OpenBao/ESO bridge secret, or edge/runtime topology) uses the separate operator-mediated lever:

curl -sS -X POST "$BASE/api/v1/deploy/infra-reconcile" \
  -H "Authorization: Bearer $COGNI_API_KEY" -H "Content-Type: application/json" \
  -d '{"nodeId":"<operator-node-uuid>","env":"production"}'

Swap "production" for "preview" to converge the preview lane instead (bug.5409) — same verb, same workflow, gated on node.manage_envs. This is shared-lane-only (preview | production) and operator-node-only. It accepts no SHA, ref, workflow, or mode: the adapter replays that lane's deployed app pin and the existing workflow consumes the merged infrastructure from main. The deployment parent is environment-scoped (NODE_SUBMODULE_PARENT_{OWNER,REPO}): candidate/test App operations stay in cogni-test-org; production App operations stay in Cogni-DAO. A 403 authz_denied currently requires the same one-time production-promoter approval. That role reuse is a two-phase bootstrap bridge—the infra lever must first deploy the OpenFGA model before story.5028's dedicated production_infra_promoter → can_reconcile_production_infra grant can take over. Never replace it with personal gh workflow run … -f skip_infra=false or an SSH mutation. For node apps, use POST /api/v1/deploy/workloads; this verb owns only the shared state/edge substrate.

After dispatch, always confirm production /version.buildSha actually advanced — see Monitoring. Discover the dispatched run via the unauthenticated GitHub API (GET /repos/Cogni-DAO/cogni/actions/workflows/promote-and-deploy.yml/runs); no personal token required.

Declared workloads

Node workloads do not use the flight and promote calls. They use the verb at the top of this file. For a node, read progress with GET /api/v1/deploy/workloads and prove the served /version.buildSha. The sequencer log is {env="<env>",service="sequencer"} |= "node_deploy_step".

Monitoring — Monitor tool, not eyeballing

Every promotion gets a Monitor armed before you walk away. The pattern emits one line per state change so you don't have to poll. Cover ALL terminal states (success, failure, cancelled, hung) — silence is not success.

# Copy-paste template. Set TARGET_SHA, FLIGHT_ID, and HOSTS for the env.
TARGET_SHA=<expected-buildsha-40-chars>
SHORT=${TARGET_SHA:0:12}
FLIGHT_ID=<flight-preview-or-promote-deploy-run-id-you-just-dispatched>
# Operator only. Nodes are not in this flow. For production use https://cognidao.org.
HOSTS="https://preview.cognidao.org"
# Self-discover the dispatch time so we can find the chained promote-and-deploy run.
DISPATCH_AT=$(gh run view "$FLIGHT_ID" --json createdAt --jq .createdAt)
PD_ID=""
prev_flight="" prev_pd=""
declare -A prev_bs=()
echo "watching flight=$FLIGHT_ID target=$SHORT hosts=$HOSTS dispatched=$DISPATCH_AT"
deadline=$(( $(date +%s) + 2400 ))
while [ "$(date +%s)" -lt "$deadline" ]; do
  fs=$(gh run view "$FLIGHT_ID" --json status,conclusion --jq '"\(.status)/\(.conclusion)"' 2>/dev/null || echo "api-fail/")
  [ "$fs" != "$prev_flight" ] && echo "[flight $FLIGHT_ID] $fs" && prev_flight="$fs"
  if [ -z "$PD_ID" ]; then
    PD_ID=$(gh run list --workflow=promote-and-deploy.yml --limit 5 --json databaseId,createdAt \
      --jq "[.[] | select(.createdAt > \"$DISPATCH_AT\")] | .[0].databaseId // empty" 2>/dev/null || echo "")
    [ -n "$PD_ID" ] && echo "[discovered promote-and-deploy] $PD_ID"
  fi
  if [ -n "$PD_ID" ]; then
    ps=$(gh run view "$PD_ID" --json status,conclusion --jq '"\(.status)/\(.conclusion)"' 2>/dev/null || echo "api-fail/")
    [ "$ps" != "$prev_pd" ] && echo "[promote-deploy $PD_ID] $ps" && prev_pd="$ps"
  fi
  all_match=1
  for host in $HOSTS; do
    bs=$(curl -sk --max-time 8 "$host/version" 2>/dev/null \
          | python3 -c 'import json,sys
try: print(json.loads(sys.stdin.read()).get("buildSha","")[:12])
except: print("")' 2>/dev/null || echo "")
    if [ "$bs" != "${prev_bs[$host]:-}" ]; then
      echo "[$host /version buildSha] ${bs:-<empty>}"
      prev_bs[$host]=$bs
    fi
    [ "$bs" = "$SHORT" ] || all_match=0
  done
  if [ "$all_match" = "1" ]; then
    echo "[DONE] all hosts match $SHORT"
    exit 0
  fi
  sleep 30
done
echo "[TIMEOUT] flight=$prev_flight promote=$prev_pd buildshas=$(declare -p prev_bs)"
exit 2

Wrap in a Monitor tool call with timeout_ms 2400000 (40 min) and a descriptive name like "heal preview-operator: flight {ID} → promote-deploy → all preview /version → {SHORT}".

When /version.buildSha doesn't advance — Loki, not SSH

/version.buildSha answers "is the new app pod serving" but says nothing about why a host-side compose container (alloy, autoheal, db-backup, alloy-k8s-events) is crash-looping. Don't SSH-tail. Both layers ship to the same Loki:

# 1. App pod startup / env validation (k3s pod logs)
{namespace="cogni-<env>",pod=~"<service>-.*"} |~ "Error|EnvValidation|panic|started|ready"

# 2. Host-side compose container stdout (host alloy → Loki)
{env="<env>",service="<compose-svc>"} | json | level=~"error|warn"
# compose-svc ∈ {litellm, caddy, temporal, autoheal, db-backup, alloy-k8s-events}

# 3. Kubernetes Events stream (pod OOMKilled, probe failures, evictions, rollout)
{env="<env>",source="k8s-events"} | json
# Use this when a node restarted but app stderr is empty — kubelet's reason
# (OOMKilled, ImagePullBackOff, Liveness probe failed) only lives in events,
# not in container logs.

# 4. Argo control-plane (operator only) — "the operator app never appeared" / "stuck OutOfSync"
{namespace="argocd",source="k8s-events",kind="ApplicationSet"} | json
# msg: created/Deleted Application "<env>-<node>" — proves the AppSet IS
# generating (a `created`+`Deleted` pair on the same app = it generated then
# pruned: a catalog/overlay condition, NOT a stuck controller).
{namespace="argocd",source="k8s-events",kind="Application",reason=~"OperationStarted|OperationCompleted|Failed"} | json
# msg: "Initiated automated sync to <sha>" → "sync operation to <sha> succeeded"
# — Argo's reconcile state for that one app, by sha.

source="k8s-events" requires the alloy-k8s-events compose service from PR #1233 to be deployed in the env you're querying. If empty: check {service="alloy-k8s-events", env="<env>"} for crash logs first.

Reaching for kubectl get applications / kubectl describe applicationset is the anti-pattern. The applicationset-controller and application-controller emit every generate / create / prune / sync transition as a k8s Event — already in Loki via stream #4 above. kind=ApplicationSet answers "is the AppSet generating this app at all"; kind=Application answers "is Argo syncing it to the expected sha." No SSH, no kubeconfig — which is also the SOC2 posture (read-only, audited, zero standing cluster credential). New compose services don't ship logs until added to the host-alloy allowlist regex in infra/compose/runtime/configs/alloy-config.{,metrics.}alloy. If {service=<svc>} returns silence post-deploy, fix the regex before diagnosing further — it's not Loki, it's the filter.

Preview is latest-wins (no lease)

There is no preview review-lease (removed 2026-06-16, kill-preview-review-lease). Every merge to main dispatches a fresh preview flight; the latest one wins. There is nothing to "unlock" and no reviewing hold to wait on — preview always tracks the latest merged SHA. Serialization under bursty merges is handled by the flight-preview and promote-deploy-<env>-<nodes> workflow concurrency groups. If preview looks stale, it's an affected-only or admin-merge image gap (see the two gotchas above) or a failed flight — not a lease; re-merge or re-dispatch flight-preview.yml.

Failure modes — first-step diagnosis

SymptomFirst diagnosis
flight-preview red at "Hard-fail when no images found for resolved PR"The push:main build did not publish sha-<mainSha> within the 15-minute wait (bug.0443). Read that pr-build.yml run.
aggregate-preview red, "no cell reported promoted=true"Same missing sha-<mainSha> build, OR the sha has zero images. Verify with resolve-pr-build-images.sh (preflight #2).
aggregate-production red, "Axiom 19 contradiction: scheduler-worker"bug.0443 fix in verify-buildsha.sh is missing/reverted. The NON_INGRESS_NODES marker-emission block must be present.
verify-buildsha timeout 90sPod cutover incomplete; usually transient. Re-check /version directly in 60s. If still wrong, check Argo app revision matches deploy-branch tip.
verify-deploy green but /version.buildSha still oldCDN/edge cache, OR you hit /readyz (which the old pod still answers) instead of /version. Always use /version.buildSha for verification, never /readyz.
Production "succeeded" but only some targets advancedAffected-only — the nodes input was scoped, or the source sha's image set did not cover all targets. Read .promote-state/source-sha-by-app.json on deploy/production-operator and deploy/production-scheduler-worker; current-sha lives only on deploy/production. Nodes have no deploy branch; read their release with GET /api/v1/deploy/workloads.
prod-pd: every verify-deploy (<node>) red at Resolve cell state with bash: app-src/scripts/ci/<script>: No such file or directory exit 127source_sha predates the verify-deploy script tree. Don't use deploy/preview:.promote-state/current-sha — aggregate-rollup.sh writes the merge-base of per-node tips, which under affected-only divergence regresses behind script additions. Re-dispatch using preflight #4's picker (newest preview-{node} promotion sha).

Discipline

  • /version.buildSha is the only deploy verification that matters. CI conclusions can lie. /readyz can lie (old pod answers during rolling cutover).
  • Per-target truth, not per-workflow (operator only). Check each operator deploy branch's .promote-state/current-sha independently. For a node, read GET /api/v1/deploy/workloads.
  • A merge outside the queue has no merge_group build. Preview then waits on the push:main build. If that build fails, file a bug and ship a follow-up merge. Do not assume the gap heals itself.
  • Don't dispatch without arming a Monitor. Fire-and-forget is how outages survive shift changes.
  • Catalog-as-SSOT for target lists. Source scripts/ci/lib/image-tags.sh; never inline a node list. resy outlived the node by months precisely because it was inlined here.
  • Promotion runs as the operator, never personal gh. Flight/promote via the API (vcs/flight, deploy/promote); the operator GitHub App dispatches. gh workflow run of a promote/flight uses a personal credential and is out of bounds. Read-only gh run view/list for watching a run is fine.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall