cogni-dao/cogni / .claude/skills/provision-env
provision-env
VerifiedProvision (or reprovision) a full environment — candidate-*, preview, or production — via the runner-owned `provision-env.yml` workflow. Covers the before (prereqs + minting secrets), the dispatch, the phase map, the hard-won gotchas that turn a "green" provision into a silent half-deploy, and the after (verify + decommission). Use when standing up a new env, reprovisioning a dead one, debugging a provision that "succeeded" but isn't serving, or when someone says "provision prod / candidate-b", "the new VM isn't up", "ImagePullBackOff after provision", "apex cut over to an empty VM", "525 after provision", "operator pod won't start". Sibling to `https://cognidao.org/knowledge/fork-quickstart-guide` (the forker's zero-to-green) — this is the operator's reference for ANY env on the hub.
provision-env — stand up an environment end-to-end
Operator lane only. This skill provisions the shared state substrate and the operator's own k3s and Argo apps. A node app ships with
POST /api/v1/deploy/workloads. See cicd-e2e-required-sequence.
The one mental model (read this first)
Provision builds the house. Promote fills the furniture. They are different steps.
- Provision (
provision-env.yml) = substrate: Cherry VM → k3s → Compose infra (postgres, doltgres, temporal, litellm, redis, alloy) → OpenBao + ESO → Caddy edge → DNS → seeds the operator's deploy branch → applies the operator's Argo apps (operator only). - Ship = fills the house. The operator uses
promote-and-deploy.yml(the/promoteskill; operator only). A node usesPOST /api/v1/deploy/workloads {nodeId, env, workload, sourceSha}; see cicd-e2e-required-sequence.
A provision is supposed to leave you at running apps (it seeds digests for nodes its seed-source runs). But it routinely leaves gaps that a green workflow hides — placeholder digests, missing local_certs. Those are bugs/gaps, not "by design." This skill is the list of what to check and fix.
/version.buildShafrom outside the cluster is the only "is it really live" signal. The workflow conclusion lies (it can pass with dead pods, or fail while Argo heals the cluster behind it).
BEFORE — prereqs (zero-spend, do all of them)
- Target GitHub environment exists + carries the minting secrets (the run dies at "Write
.env.bootstrap" if any are empty — this is the #1 first failure):- Per-env:
CHERRY_PROJECT_ID,CLOUDFLARE_API_TOKEN,CLOUDFLARE_ZONE_ID,GH_ADMIN_PAT,GH_ADMIN_USERNAME - Repo-level:
CHERRY_AUTH_TOKEN - Verify:
for k in CHERRY_PROJECT_ID CLOUDFLARE_API_TOKEN CLOUDFLARE_ZONE_ID GH_ADMIN_PAT GH_ADMIN_USERNAME; do gh api repos/<owner>/<repo>/environments/<env>/secrets/$k >/dev/null 2>&1 && echo "✅ $k" || echo "❌ $k MISSING"; done
- Per-env:
GH_ADMIN_PAT= a dedicated BOT, never a human token. Use the automation bot account (e.g.Cogni-1729), classic PAT withrepo, workflow, write:packages(admin:repo_hookif webhooks). Fine-grained PATs cannot reach org-owned repos from an outside collaborator — classic follows the account's actual access.GH_ADMIN_USERNAMEmust equalgh api user --jq .loginof that token.- App + external secrets present on the env (
OPENROUTER_API_KEY,EVM_RPC_URL,POSTHOG_*,POLYGON_RPC_URL, plus DoltHub / OAuth / Tavily / Privy / Langfuse if the node uses them). They reach the pod only if they're also in the runner env-block ofprovision-env.yml+NODE_BASELINE_KEYS(scripts/setup/lib/reconcile-secrets.sh). - Cloudflare token scope = DNS:Edit + Zone Settings:Edit (the run hard-fails at zero spend if only DNS:Edit).
- Generate + SAVE an encryption passphrase — you need it later to decrypt the init artifacts (kubeconfig, vm-key).
openssl rand -hex 24 > .local/<env>-init-passphrase.txt. - Production only: know your rollback anchor (the current live VM IP) before dispatch — provisioning repoints the apex (see Gotcha 1).
DISPATCH
gh workflow run provision-env.yml --repo <owner>/<repo> \
-f env=<candidate-a|candidate-b|preview|production> \
-f encryption_passphrase="$(cat .local/<env>-init-passphrase.txt)"
# run a branch's version with --ref <branch> (e.g. to test a renderer/secrets fix before main)
Modes (-f mode=)
A full provision is the default. The narrow, idempotent day-2 heal mode reconciles the OpenBao SA/role substrate on an existing VM without a full re-provision (verify-and-heal; a re-run is a no-op):
substrate-only— reconcile the env's OpenBao auth/role/SA layer on the existing VM (no tofu/VM/DNS). Seescripts/setup/reconcile-env-substrate.sh.
Born-observable is NOT a provision mode (it was
mode=observabilityuntil the substrate fold; superseded). The node-log proxy (GET /api/v1/nodes/{id}/observability/logs) depends on two per-env pieces, and both now come up on the normal flow — no bespoke lane, no "never hand-write" prohibition:
- Alloy
nodeId→nodeLoki-label config. Re-pushed + checksum-restarted by the substrate-readiness lane (scripts/ci/reconcile-node-substrate.sh→reconcile-alloy-config.remote.sh), which runs on every candidate-flight + promote — so even an app-only promote (skip_infra: true) lands it current. Idempotent; an unchanged config is a no-op.- Operator Grafana read creds at
cogni/<env>/operator/{GRAFANA_URL,GRAFANA_SERVICE_ACCOUNT_TOKEN}. The catalog row (infra/secrets-catalog.yaml,service: operator) + the operator ExternalSecret + provision Phase 5e (mints theglsa_read token from the one-timeGH_GRAFANA_CLOUD_ADMIN_TOKENand seeds both_sharedANDoperator) own this. A fresh provision wires it at birth.If an OLD env (provisioned before Phase 5e seeded the operator path) lacks the creds: re-run the standard provision, or seed
cogni/<env>/operator/GRAFANA_*through the normal secret path (secret-materialize.sh/ the substrate lane) — one Grafana stack serves every env (env is just a Loki label), so the sameglsa_read token + URL is correct everywhere. Spec:https://cognidao.org/knowledge/node-log-proxy-contract§"Born observable".
Phase map (where things die)
| Phase | Does | Common failure |
|---|---|---|
Write .env.bootstrap | maps GH secrets → runner | missing minting secret (empty GH_ADMIN_PAT) |
| Restore prior init-artifact | re-run idempotency | prior artifacts encrypted with a different passphrase → abort (delete prior VM+SSH key, or reuse passphrase) |
| 3 · Provision VM (tofu) | Cherry VM + DNS | adopt-vs-create; ephemeral tofu state on runner |
| 4b · DNS | cuts over apex + node subdomains to the new VM | production = LIVE cutover before apps exist (Gotcha 1) |
| 4b.5 · Seed deploy branches | seeds digests | placeholder digests (Gotcha 2); H7 divergence (Gotcha 4) |
| 5 · Compose | infra + Caddy | db-backup exit-137 — can be FATAL: aborts the whole deploy → no app layer (Gotcha 13) |
| 5b/5c · OpenBao + ESO + seed | secrets substrate | quoted secret values (Gotcha 6) |
| 5f · deploy-infra | Compose runtime | required-secret gaps (e.g. POSTHOG_*/POLYGON_RPC_URL if unset) |
| 7 · ApplicationSets | Argo deploys apps | placeholder digest → ImagePullBackOff |
9 · /readyz verify | 5-min/node | cascades red if scheduler-worker is down (Gotcha 11) |
GOTCHAS — every one of these cost hours
-
Apex cutover is in Phase 4b, BEFORE apps deploy (Phase 7). For
production,cogni_operator_domain_for_env(production) = cognidao.org(apex) — provisioning repoints the live apex to the new, app-less VM for 10–20 min. A failed run leaves the apex on a dead VM (a DNS time-bomb under resolver cache). Always have the old VM IP and roll the apex back if it goes sideways. Cleaner: blue-green — provision withDOMAIN=<env>-next.<root>override, validate the operator fully, then flip the apex by hand. -
Placeholder image digests = ImagePullBackOff that NEVER self-heals. Nodes the seed-source doesn't run get
ghcr.io/.../cogni-template:<env>-placeholder-<node>(a fake tag). Each node has its OWN image (same repo name, different digest) — do NOT reuse one node's digest for another (node-template's lacksnodes/operator/app/migrate.mjs→MODULE_NOT_FOUNDin the migrate init container). Fix: get the correct per-node digest from a healthy env (KUBECONFIG=.local/candidate-a-kubeconfig.yaml kubectl get deploy -n cogni-candidate-a -o custom-columns=NAME:.metadata.name,IMG:.spec.template.spec.containers[0].image), patch eachdeploy/<env>-<node>overlay (infra/k8s/overlays/<env>/<node>/kustomization.yaml: replacenewTag: "*placeholder*"withnewName + digest), push,kubectl -n argocd annotate app <env>-<node> argocd.argoproj.io/refresh=hard --overwrite. The real fix = provision seeds real per-node digests (or run/promote). -
local_certsmissing → Caddy 525 on every public host. The rendered Caddyfile has bare site blocks (no TLS directive) → Caddy attempts Let's Encrypt behind the Cloudflare proxy → challenge can't reach origin → no cert →tlsv1 internal error→ CF returns 525. The zone SSL mode isfull, so the fix islocal_certs(self-signed, CF Full accepts it). Manual: addlocal_certsinside the global{ }block of/opt/cogni-template-edge/configs/Caddyfile.tmplon the VM (before thelog {line),docker restart cogni-edge-caddy-1. Durable fix =scripts/ci/render-caddyfile.shemitslocal_certsin its global block. (A pre-existing env may "work" only because it got an LE cert during an old grey-cloud window.) -
H7 — preview/prod deploy-branch divergence. preview/production refuse force-update; Phase 4c commits
env-state.yamlon top of the seed, so the tip is always ahead → re-runs abort at Phase 4b.5. Cleardeploy/<env>*branches before re-dispatch (candidate-* auto-force, so this only bites preview/prod). They re-seed clean. -
Re-run trap. "Restore prior init-artifact keys" aborts if a previous run uploaded artifacts under a different passphrase. Deleting the Cherry VM + SSH key is NOT sufficient — the restore step keys off the prior run's GH
<env>-init-artifacts(1-day retention), not the VM. So a fresh-passphrase re-dispatch still finds the old artifact and diesdecrypt failed for <env>-vm-key. Fixes: (a) keep the SAME passphrase across retries of one env (simplest — never regenerate mid-cycle), or (b) delete the prior artifact:gh api repos/<owner>/<repo>/actions/artifacts --jq '.artifacts[]|select(.name|test("<env>-init-artifacts"))|.id'→gh api -X DELETE …/artifacts/<id>→ re-dispatch (fresh-key path). Seen 2026-06-04: regenerating the passphrase between candidate-b retries wasted a dispatch. -
Quoted secret values brick init.
gh secret setof a value that includes literal quotes ('https://...') stores the quotes → URL-validated fields (EVM_RPC_URL,POSTHOG_HOST) fail the Zod schema →/readyz503 init-loop..env.*files are quoted (fine forsource, not for verbatimgh secret set). Strip quotes before setting; operator-200 on the old prod proves nothing about the new secrets path. -
.env.bootstrapextraction traps. Lines carry inline comments and a commented stale value, andGITHUB_ADMIN_*(file) vsGH_ADMIN_*(GH secret) differ. Extract withgrep -m1 "^KEY=" f | cut -d= -f2- | awk '{print $1}' | sed "s/['\"]//g"(first field, quote-stripped) — naivecut -d= -f2-grabs the trailing comment. -
You CANNOT debug app failures from
gh run. The run shows phase text only; the real failure (ImagePullBackOff, initMODULE_NOT_FOUND, 525) lives in the cluster and Loki. Two access paths:- kubeconfig from the init artifacts (after the run ends):
gh run download <id> --name <env>-init-artifacts, decrypt each*.encwithopenssl enc -d -aes-256-cbc -pbkdf2 -iter 200000 -pass pass:<PP>, thenKUBECONFIG=.../<env>-kubeconfig.yaml kubectl .... - Loki (no cluster access needed, works mid-run): prod ships k8s-events to Grafana Loki —
{env="<env>"} |~ "(?i)imagepull|backoff|oom|evicted|context deadline". This is how you read the real pull error while Phase 9 is still grinding.
- kubeconfig from the init artifacts (after the run ends):
-
Argo
selfHeal=truereverts kubectl hacks.kubectl set image/editgets reverted to the deploy-branch state within ~3 min. Fix the deploy branch (push) — that's the source of truth. -
GHCR throttle ≠ placeholder. A fresh 6gb-shared VM pulling 5 images at once can transiently
BackOff(recovers). A placeholder tagBackOffs forever. Distinguish via Loki: a:<env>-placeholder-<node>tag is the never-heals case; a real@sha256:...that's slow is transient. -
/readyzcascade. Every node's/readyzasserts scheduler-worker connectivity. If scheduler-worker is down (placeholder/ImagePull), all nodes report 0/1 + Phase 9 fails everything — looks like 5 failures, is 1 root cause. Fix scheduler-worker → the rest go Ready. -
Postgres root-password drift wedges the WHOLE infra deploy (and looks like an alloy/observability failure — it isn't).
POSTGRES_PASSWORDis honored only at first-init of an empty volume. After a re-provision or aPOSTGRES_ROOT_PASSWORDrotation, the persistedpostgres_datavolume keeps the old password whiledeploy-infrarenders.envfrom the GH-secret SSOT →db-provision's TCP+scram auth fails, spins its full 120s, anddeploy-infra.shaborts at thedb-provisionstep. The ERR-trap diagnostic then dumpsalloy "unsupported protocol scheme"+ "unhealthy containers" — red herrings: alloy/s-ui/git-sync/repo-init/alloy-k8s-events have no healthcheck (Docker reportsnull, not unhealthy), and the empty-Loki-URL line is a transient of that one re-render. The real line is❌ Timed out waiting for Postgres+FATAL: password authentication failed for user "postgres". Downstream blast radius: the abort happens before deploy-infra's Step-7 k8s-Secret write, so the<node>-node-app-secretsSecret is never refreshed → every catalogued external key (GH_REVIEW_APP_*,DOLTHUB_*,PRIVY_*,LANGFUSE_*, OAuth) stays empty in the pod even though it's present in the GH env + carried in the promote-and-deploy env block. Fix is in git (idempotent, self-healing):deploy-infra.shreconciles the superuser password via the postgres image'slocal … trustunix socket (compose exec … ALTER USER) beforedb-provision, andpostgres-init/provision.shre-asserts the app/service role passwords on the "already exists" branch. Manual unwedge if you hit an unpatched env:docker exec cogni-runtime-postgres-1 psql -U postgres -c "ALTER USER postgres PASSWORD '<.env value>'"(and app_user / service user), then re-run the deploy. -
Phase 5f
db-backup(exit 137) aborts the ENTIRE deploy-infra → zero app layer.deploy-infra.sh:643runs$RUNTIME_COMPOSE --profile backup up --force-recreate --no-deps --abort-on-container-exit --exit-code-from db-backup db-backup. On a fresh/loaded VM thedb-backupcontainer is SIGKILLed (137) and--abort-on-container-exitkills deploy-infra before Step-7 → thecogni-<env>namespace, the<node>-node-app-secrets, and the per-node Argo Applications are never created. Symptom: argocd/openbao/external-secrets/reloader allSynced+Healthy, zero app pods, nocogni-<env>ns, URLs 502/000. Looks like total node failure; is one backup container. (Distinct from Gotcha 12, which aborts atdb-provision.) Seen 2026-06-04 — db-backup needs atimeout/retry or a skip-on-empty-DB guard so a flaky backup can't gate the whole app layer. -
A provision that dies late can upload an init-artifact INCONSISTENT with the live cluster — root + SSH lost despite a "good" artifact. Seen 2026-06-04:
<env>-init-artifactsdecrypted cleanly and itskubeconfigauthenticated to the live k3s, but the same artifact'sopenbao-init.jsonroot_tokendid not authenticate (bao token lookup -selffails on an unsealed OpenBao), itsvm-keydid not SSH (publickeydenied), ANDgenerate-rootwith itsunseal_keys_b64failed. A clean decrypt is NOT proof of valid custody. Validate immediately:kubectl -n openbao exec openbao-0 -- env BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=$RT bao token lookup -self. If it mismatches, the env has no recoverable root or SSH — surgical recovery is impossible; destroy + reprovision is the only path. (Likely cause: auto-init/auto-unseal racing the script'sbao operator init, or a VM/key regen between phases — capture happens at a different moment than the live state.) -
Reprovision reuses the Cherry IP but rotates BOTH the host key and the deploy key. Fresh VM at the same IP → (a) local
known_hostsMITM warning, clear withssh-keygen -R <ip>; (b) the GH-secret VM deploy key goes stale →candidate-flight-infra(every infra-lever workflow) fails at "Setup SSH" (publickey) until that secret is refreshed with the new<env>-vm-key. So you can't re-run deploy-infra via the workflow against a freshly-reprovisioned VM until you update the deploy-key secret — and if Gotcha 14 hit, you have no working key to update it with. -
Phase 1 of the DB-cred OpenBao port (
<env>-db-reader) validates via k8s-auth — NO root token needed. Even on a provision that died at 5f, if Phase 5b completed, the<env>-db-readerpolicy/role +db-provisionerSA are live. Prove the OpenBao read path with the kubeconfig alone:JWT=$(kubectl create token db-provisioner -n default); TOK=$(kubectl -n openbao exec openbao-0 -- env BAO_ADDR=http://127.0.0.1:8200 bao write -field=token auth/kubernetes/login role=<env>-db-reader jwt=$JWT)→bao kv list cogni/<env>succeeds (least-priv read). This is how Invariant-15 Phase 1 was deploy-verified on candidate-a despite the dead app layer. -
$REPO_ROOT(or any runner-only var) referenced INSIDE the remote heredoc is unbound on the VM →set -uaborts the whole deploy before Phase 7 → zero app layer.deploy-infra.shships a quoted-heredoc remote script (cat > … << 'EOF', runs on the VM via SSH);REPO_ROOTis assigned on the runner and never crosses. A barebash "$REPO_ROOT/scripts/…"in that block diesREPO_ROOT: unbound variableAFTER Step-7 secrets but BEFORE the ApplicationSets apply — looks identical to Gotchas 12/13 (substrate green, no app pods) but the tell isdeploy-infra-remote.sh: line NNN: <VAR>: unbound variablein the run log. Rule: any helper a remote-block needs must bescp'd to/tmp/(alongsideensure-temporal-namespace.sh) and invoked as/tmp/<name>.sh, never via a runner path. Seen 2026-06-04:sync-app-webhook-secret.shadded in #1482 with$REPO_ROOTsilently broke EVERY clean provision for a day (fixed: upload to/tmp+ call/tmp/sync-app-webhook-secret.sh). -
Phase 6 only applies the ExternalSecret leaves that EXIST on disk for that env — a missing per-(svc,env) leaf = that node never gets
<node>-env-secrets→ pod never Ready → Phase 9/readyzreds the provision. Phase 6 globs the single SSOT treenodes/<node>/k8s/external-secrets/<env>/(every node carries its own leaf — operator, resy, scheduler-worker, canary, node-template; the pre-wizard operator-domain tree was purged). For an ESO-migrated node (consumesenvFrom: <node>-env-secrets, not the deploy-infra bridge secret), if its leaf is absent for the target env, Phase 6 silently skips it (Applied Nwhere N < node-count) and that pod boots without config. The trap: this only bites the env that's missing the leaf — candidate-a had all 5 leaves so it looked done; candidate-b shipped only 3 (scheduler-worker/canary/node-template), missingexternal-secrets/candidate-b/{operator,resy}/, so a clean provision reached Phase 6/7 fine but red'd at Phase 9 on operator+resy/readyz. Diagnose via the run'sApplied N ExternalSecret manifest(s)count + the=== ExternalSecrets sync state ===dump (the missing nodes simply aren't listed). Fix = add the per-(svc,env) leaf (mirror an existing env, sednamespace:+dataFrom.extract.key:); it's committed → reproducible. Leaves are hand-authored per env today — a new env or node silently inherits this gap; the durable fix is to codegen the leaves from the catalog (same renderer class asrender-node-appset.sh). Seen 2026-06-04 (candidate-b clean-room). -
Per-env node membership ≠ the global catalog — Phase 9 must verify only the nodes the env DEPLOYS, or a correctly-provisioned subset-env reds.
NODE_TARGETS(fromscripts/ci/lib/image-tags.sh) = everytype:nodeininfra/catalog/*(operator, resy, canary, node-template). But an env deploys only what its ApplicationSet declares — equivalently, the nodes with an overlay underinfra/k8s/overlays/<env>/(the env-scoped set Phase 4b.5 already uses). A subset env (candidate-b = 3-nodetask.5094OpenBao proof: canary/node-template/scheduler-worker, NO operator/resy) provisions DBs + secrets + ExternalSecret leaves for ALL catalog nodes (harmless over-provision), but its AppSet creates no operator/resy Argo app → no pod → Phase 9 (looping the globalNODE_TARGETS) verifies operator/resy/readyz, fails after 5 min, and reds a provision whose declared nodes are all 1/1 + 200. The tell:Applied 5 ExternalSecret manifest(s)+ ESOSecretSynced Truefor operator/resy, yet no operator/resy pod inkubectl get podsand nocandidate-b-operatorArgo app. Fix = scope Phase 9's loop to[[ -d infra/k8s/overlays/<env>/<node> ]](skip undeclared). The deeper durable fix = make the per-env node set a single SSOT (the AppSet list) and derive NODE_TARGETS/DB/secret/verify scope from it, so a subset env doesn't over-provision. Seen 2026-06-04 (candidate-b); the oldCANARY IS REDbanner mislabels this (it fires for ANY node, not canary). -
A provision run is 30–60 min — re-engaging on completion without going silent is its OWN failure mode, and every passive monitor tried here MISSED. Exact ledger + the rule. Mechanisms tried on the 2026-06-10 preview rebuild and why each went silent:
gh run watch <id> --exit-statusviaBash run_in_background: true(the obvious "ping me when it finishes"). Fails: the background task is KILLED on every session resume / context-compaction boundary (theSessionStarthook fires repeatedly across a long wait). A killed task emits<task-notification status=killed>— NOT a completion — so the agent gets a "killed / Continue from where you left off" prompt, not the run result. Treated as terminal ⇒ silence.- Replying "No response requested" / any stand-down to that
status=killednotification. This was the silence. The killed-watcher notification is a RE-ARM trigger, never a terminal event, while the run is still pending. ScheduleWakeupas a far-future BACKSTOP (1200s) behind the watcher. Roles inverted — the unreliable background watcher was treated as primary, the durable wakeup as a 20-min backstop. When the run went green early and the watcher had already been killed, nothing re-invoked the agent for up to 20 min ⇒ effective silence (the human poked first).ScheduleWakeup(270s)self-paced 5-min polls (early runs). These worked — durably re-invoked — but burn a turn every ~5 min for an hour.
RULE:
ScheduleWakeupis the ONLY durable re-invoke for a provision/flight/promote wait — make it PRIMARY, never a backstop. (a) Match the delay to the phase you're gating on (a ~35-min provision → ~600–900s checks, not one 1200s shot). (b) Agh run watchbackground task is a nice-to-have fast path, never the sole signal — it WILL be killed across a resume. (c) Never end a turn while a run is pending without aScheduleWakeuparmed. (d) Astatus=killed/stoppedtask-notification means RE-ARM, not stand down. (e) Each wakeup's own prompt must say "re-arm before ending until<terminal condition>." Seen 2026-06-10: two silences on one preview rebuild — once from standing down on a killed watcher with no wakeup, once from a 20-min backstop that hadn't fired when the run went green. -
A migration only runs on the DEPLOYED operator image's commit — a stale-digest reprovision, OR healing by flighting a non-operator commit, leaves new migrations UNRUN despite a green run (silent success). The operator
migrateinitContainer applies only the migrations baked into whatever operator image thedeploy/<env>-operatorbranch pins (nodes/operator/app/src/adapters/server/db/migrations/). Two ways new DB state silently doesn't land: (a) a reprovision pins a digest that predates the migration (e.g. registry seed0040); (b) you heal by flighting main head, but main head is a non-operator change — flight/promote are affected-only (task.0376), so a commit touching 0nodes/operator/appfiles (a provision-scripts / docs PR) does NOT redeploy operator. Either way operator's/version.buildShastays stale and BOTHflight-preview+promote-and-deployreportcompleted/success(they verify whatever WAS deployed, not that your requested sha is serving). Diagnose in the VM Postgres, never by workflow conclusion:docker exec cogni-runtime-postgres-1 psql -U postgres -d cogni_operator -Atc 'SELECT max(id) FROM drizzle."__drizzle_migrations"'— behind main's migration count ⇒ the app-layer state that migration writes (registry rows, RLS, backfill) is missing. Fix: flight/promote the specific operator-AFFECTING commit that carries the migration (git log --oneline <sha> -- nodes/operator/appto find it), not main head; then verify in-DB. Seen 2026-08-07: registry seed0040(#1978 /3c4f256) reached preview+prod only after flighting3c4f256; flighting main heada5f63fb(#1980, provision-scripts-only) silently skipped operator (buildSha never moved, both workflows green). Cross-ref:/promote§ "Affected-only gotcha".
AFTER — verify + custody
KUBECONFIG=.local/prod-art/<env>-kubeconfig.yaml kubectl get pods -n cogni-<env> # all 1/1
curl -s https://<public-host>/readyz # 200 — and curl /version, confirm buildSha
- All pods 1/1, not just a green run.
- Custody (Gotcha 14 — this is what loses tokens): decrypt
<env>-openbao-init.json+<env>-vm-key+<env>-kubeconfig.yamland land them in the ONE canonical custody dir~/dev/cogni-template/.local/provision-creds/<env>/(gitignored,chmod 600) — keep that dir'sREADME.mdstatus table current (per-env: the live VM IP, a validated-✅ line, and the file list — root token / kubeconfig / SSH key). Validate the root token against the live cluster (bao token lookup -self) before trusting it. Do NOT delete the GH artifact until the token is validated — its 1-day retention is the only backup. (If you have no password-manager CLI, this durable dir + validation IS the custody bar; the artifact expiring + no local copy = unrecoverable env.)- ⚠️ Scatter trap. The provision workflow writes decrypted artifacts into whatever workspace's
.local/it ran from (.local/prod-art/…, a Conductor workspace.local/, etc.). Those are throwaway working copies that go stale the next reprovision — do NOT let them become a second source of truth. Immediately promote the validated set into~/dev/cogni-template/.local/provision-creds/<env>/(the non-workspace clone) and treat every other.local/copy as disposable. An agent that greps a random workspace.local/will find a stale VM IP / dead kubeconfig; the canonical dir is the only address to publish.
- ⚠️ Scatter trap. The provision workflow writes decrypted artifacts into whatever workspace's
- Production: keep the old VM as the rollback anchor until the new apex is proven, then decommission (delete the old Cherry VM, prune stale DNS).
Linked guides
https://cognidao.org/knowledge/fork-quickstart-guide— forker's zero-to-green (the human seam: bot PAT + minting secrets)https://cognidao.org/knowledge/provision-an-environment— env stand-up + the known e2e gaps/promote— fill real per-node digests (the app layer; production is human-gated)/cicd-secrets-expert— minting vs runtime secrets, OpenBao/ESO, the quote/extraction traps/devops-expert— pipeline + VM SSH policy (candidate-a write-OK, never prod).github/workflows/provision-env.yml·scripts/setup/provision-env-vm.sh·scripts/ci/render-caddyfile.sh·scripts/setup/lib/reconcile-secrets.sh— verify claims against code
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |