Skip to content

cogni-dao/cogni / .claude/skills/cicd-secrets-expert

cicd-secrets-expert

Verified

Secrets architecture reference for node-template — when to use OpenBao+ESO vs GitHub env secrets, which operation pattern fits which write/rotate/add flow, the load-bearing invariants from the spec, the YAML catalog + Zod-loader the script consumes, and where to find the canonical implementation. Use when adding/rotating a secret, designing a service that consumes secrets, debugging an ExternalSecret or writer-role login, deciding between substrate and Compose-infra routing, touching `pnpm secrets:set` / `scripts/secrets/` / `scripts/lib/secrets-catalog-loader.ts` / `nodes/<node>/.cogni/secrets-catalog.yaml` / `infra/secrets-catalog.yaml` / `infra/k8s/argocd/{openbao,external-secrets}/` / per-node ExternalSecret manifests, or evaluating any new workflow that touches secret values. Triggers: 'add a secret', 'add a node secret', 'rotate a key', 'OpenBao', 'ESO', 'ExternalSecret', 'secrets-catalog', 'catalog tier', 'A1', 'A2', 'B-tier', 'writer role', 'bao login', 'vault-action', 'vault-config-operator', 'secrets-manage', 'secret in GH env vs OpenBao', 'where do I put this credential', 'per-node catalog'.

CI/CD Secrets Expert

Route the work through operator-architect first; this skill adds the secrets architecture rules on top of it.

One-page reference for anyone touching secrets in node-template. Read this BEFORE the spec; this points at what to actually read.

Node self-serve secrets — the value-write path

A node-owner sets a secret with only an API key — POST <env-host>/api/v1/nodes/<id>/secrets {env, key, value} → 200 written, version, written to cogni/<env>/<node>/<KEY>, then ESO + Reloader carry it into the pod. The node-dev contract + the LIVE per-env status are single-sourced in the hub guide node-self-serve-secrets (GET /api/v1/knowledge/node-self-serve-secrets) — not snapshotted in git. Design canon: https://cognidao.org/knowledge/node-secret-write-design. The durable, non-obvious facts (why this skill exists):

  • Per-env sealed operators — NO cross-env write. Each env runs its own operator pod; the OpenBao policy (cogni/<env>/*, _system/_shared denied) makes a candidate-a token physically unable to write prod. The auth chain: operator pod's projected SA token (audience: cogni-openbao) → auth/kubernetes/login (role <env>-node-secrets-writer, SA operator-secrets-writer@cogni-<env>) → KV put/patch. No kube/SSH; caller holds only an API key.
  • RBAC is granular: can_manage_secrets ← secrets_manager, NOT developer. developer → can_flight (logs/flight); secrets_manager is a distinct grant for secret writes (both ∪ admin). Requesting developer and expecting secret-write is the trap — 403 authz_denied is the right answer to the wrong role.
  • env is an explicit validated body param (D1, #1754), == the operator's own DEPLOY_ENVIRONMENT else 409 wrong_operator_env. Node resolved via resolveNodeRef (status-agnostic) through the shared withNodeRbac seam (src/app/_lib/node-rbac.ts, #1771).
  • Per-env provisioning + the three 5xx (diagnose by status, not by guess): the write needs, per env, the <env>-node-secrets-writer OpenBao role + the operator overlay (OPENBAO_NODE_SECRETS_WRITER_ROLE via envFrom + a projected SA token, audience: cogni-openbao) + Reloader present + the OpenFGA model carrying the checked relation. Failure modes: 503 secrets_plane_config_missing = OPENBAO_NODE_SECRETS_WRITER_ROLE unset on the pod; 502 at write = the OpenBao role/login is missing (vs the env var); 503 authz_unavailable = the env's OpenFGA model lacks the checked relation. That last one is a recurring silent-drift class (task.5049): bootstrap-openfga.sh only runs inside deploy-infra (skip_infra=false), so normal promotes never re-bootstrap — a new RBAC relation (e.g. can_manage_secrets, #1627) silently never reaches an env, and secrets 503 while older actions 403. Fix: POST the current rbac-model.json to that env's OpenFGA store + repoint cogni/<env>/operator OPENFGA_AUTHORIZATION_MODEL_ID + let Reloader roll the operator. (Live per-env readiness: recall the hub guide, above.)
  • source: agent keys are off-limits (denylist node-secrets-reserved.data.ts, #1753) — they're substrate-minted per node; clobbering one breaks the node.
  • Control-plane framing: OperatorSecretsPlanePort is the secrets row of the typed operator control plane (cicd-platform-boundary.md) — sibling of OperatorDeployPlanePort; one port per substrate (Argo/OpenBao/Cherry-Akash), unified by the (node_id, env)+OpenFGA contract. A node is a bundle of services (node → services → deployments, node BaaS substrate model), not one deployable.

North star

proj.agentic-fork-bootstrap — easy-start guide for a forking dev that uses OpenBao. Every PR is measured against the forker's manual-command count. If your change adds a manual step to fork-quickstart.md, that's debt — try a workflow first.

Load-bearing invariants — gate every secrets decision

Load-bearing subset — canonical numbering is secrets-management Invariants 1–16; the rows below are the ones that gate day-to-day secrets work (7, 10–12, 14 omitted — read the spec for the full set):

#RuleWhere it bites
1PATH = cogni/<env>/<service>/<KEY>; <service> = catalog nameNew service → new ExternalSecret dir
2ONE ExternalSecret per (service, env) with dataFrom: extract; target <service>-env-secretsAdding keys = NO YAML edit
3Pod envFrom: secretRef: name: <service>-env-secrets once per containerPod spec set ONCE at service creation
4NO secret value in git — everBase64-in-YAML = immediate rotate + audit
5OpenBao is runtime SSOT; VM .env files are rendered views, not authoritiesDon't seed runtime values in two places
6RBAC via path policy (eso-reader, <env>-writer) bound to k8s SAsPhase 5b.3 + 5b.4 of provision-env-vm.sh
8Every access audited via OpenBao audit device → LokiPipeline not built yet — bug.0445 follow-up
9Three entry points only: CLI / workflow_dispatch / operator API (POST /api/v1/nodes/<id>/secrets, OpenFGA can_manage_secrets — shipped #1627). Never raw bao kv putSee decision tree below
13NO_OPERATOR_ROOT_TOKEN_ON_LAPTOP — bootstrap window only; day-2 uses writer-role JWT.local/<env>-openbao-root-token is never read post-Phase-5b
15Pod-facing DB role material is OpenBao-owned, even when Compose renders a copyNo DB password authority in GitHub env or VM .env
16New-node secret materialization precedes substrate reconcile/assertGenerate safe agent values before first flight

Authority model — classify by three axes

Do not use tier as the authority model. Every secret has:

AxisValuesQuestion
originagent · human · derivedWho can produce the bytes?
custodyopenbao · github-env · repo-configWhich system is authoritative?
consumerspod · compose · ci · externalWhere does the value get used?

Hard rule: if a value is consumed by a pod, provisions a pod-facing role, or must agree with a pod-facing value, custody is OpenBao. GitHub Environment Secrets may bootstrap access and carry CI-only credentials. VM .env files are rendered views for Compose, never authorities.

Examples:

  • POSTGRES_ROOT_PASSWORD: Compose/bootstrap-only for now, because pods should not consume it.
  • APP_DB_PASSWORD, APP_DB_SERVICE_PASSWORD, APP_DB_READONLY_PASSWORD, DOLTGRES_PASSWORD, DOLTGRES_READER_PASSWORD, DOLTGRES_WRITER_PASSWORD: OpenBao custody; Compose renders copies to create roles. (DOLTGRES_PASSWORD is the env Doltgres superuser the pod itself authenticates as — Doltgres RBAC is table-DML-only, so no per-node role; why + value-shape in https://node-template.cognidao.org/knowledge/database-migration-contract §5.2.)
  • OPENFGA_DB_PASSWORD: OpenBao custody. OpenFGA is the first shared-infra DB given a dedicated, non-root login role (its own openfga role, OpenBao-sourced — Phase A dropped root from the datastore DSN). That's the hardening direction, not a snowflake: litellm / postgres are still the un-hardened root-owned legacy. Consequence — shared-infra DBs (openfga, litellm) are provisioned in the INFRA_ONLY pass only; the per-node provision pass must not touch them (it holds no shared-infra password). litellm survives the per-node pass only by being root-owned (no password to demand); openfga's dedicated role exposed that the per-node pass should never provision shared infra. provision.sh now gates openfga to INFRA_ONLY; gating litellm the same way is the fuller fix. Shared-infra DB-cred rule (learned the hard way, fleet reprovision 2026-08-04 — applies to any NEW one): (1) a shared-infra DB-role password (OPENFGA_DB_PASSWORD, TEMPORAL_DB_PASSWORD) is NOT a catalog entry (infra ≠ node; the loader allowlist is node-domain) — provision-env-vm.sh Phase 5c generates it (randHex 32) + seeds it set-once (reuse the existing OpenBao value on re-provision, never regenerate → the created role can't diverge, the 2026-06-11 temporal-drift class). (2) deploy-infra MUST read it via the ungated openbao_get_field (${env}-db-reader seam) — NOT source_openbao_runtime_key, which is gated on operator-env-secrets (OPENBAO_RUNTIME_SSOT=true = "established ESO env"). That gate is false on a fresh provision (no namespace/secret yet), so a required read there silently skips and the .env render trips set -u → <VAR>: unbound variable → deploy-infra aborts before the app layer, leaving a 0-byte runtime .env (a red-herring symptom). Wiring a consumer (compose DSN + provision.sh role) without also wiring the producer (Phase 5c seed) + the ungated read leaves the env unprovisionable-from-clean but green on established re-deploys — latent until the next from-scratch provision. (3) The superuser password MUST be minted set-once (randHex), NEVER derived from another secret. DOLTGRES_PASSWORD was the one left out of the 2026-08-04 set-once fix: deploy-infra fell back to derive_secret doltgres-root = sha256(POSTGRES_ROOT_PASSWORD), so a root rotation (or genesis-derive ≠ a later value) drifted the rendered password from the volume's superuser, which is baked at first-init and IMMUTABLE (Doltgres can't ALTER its superuser; postgres/temporal can, via the local-trust socket — that's why they have an in-place Step-6.5 reconcile and doltgres cannot). Fix = seed set-once at cogni/<env>/operator/DOLTGRES_PASSWORD, mirroring OPENFGA/TEMPORAL → the superuser can never drift. (4) NEVER "heal" a credential drift by recreating/wiping the DB volume — destroying data to fix a password is a footgun (a transient auth-probe miss then nukes a live store). Prevent drift with (3); an already-drifted volume is recovered by an explicit operator action (wipe only if provably empty; otherwise re-pin the SSOT to the volume's value), never an automatic deploy-path behavior.
  • Public URLs / owner slugs / feature modes: repo-config, not OpenBao.

Biggest systemic shortcoming — the .env/_shared split-brain

One logical shared secret becomes three physical copies — _shared SSOT, per-node inherited copy (ESO→pod), VM .env render (Compose) — written by three writers with no lockstep. They drift: the recurring root of secrets outages (litellm / temporal / doltgres / billing 401·28P01). The tell: pods have no .env and rarely drift; every .env-bound Compose service is the surface. Never heal toward .env (Invariant 15); the endgame is to eliminate the copies (purge the _shared bucket + purge the server .env → Bao Agent / k8s). Mechanism, evidence, three-layer fix, falsification gate → node-self-serve-secrets.

Routing tree — where does the value render?

TierConsumed byRender pathCustody
A1k8s pod baseline (anything under nodes/<n>/app/, every fork)OpenBao cogni/<env>/<service>/* → ESO → k8s Secret → envFromOpenBao
A2k8s pod node-specific (any downstream <node>)OpenBao cogni/<env>/<node>/* → ESO → k8s Secret → envFromOpenBao
BCompose-infra service (postgres, litellm, temporal, redis, alloy, caddy)Rendered to VM .env or future Bao AgentOpenBao unless CI/bootstrap-only
DCI-only (workflow consumption, never runtime)GH Env Secret → workflow env: blockGH Environment Secrets
ERepo-level CI (cross-env, one value per repo)GH Repo SecretGH Repo Secrets
FLocal dev only.env.local (gitignored)Operator's laptop
GDerived from repo state at provision time (e.g. nodes/* listing)Computed by loader; written alongside other catalog valuesAuto-generated

Full tier definitions + invariants: secrets-routing-tiers. Identity and RBAC layers: operator-identity-model, operator-rbac-capability-map. Routing checklist (file-by-file propagation): .claude/commands/env-update.md §0.5.

Runtime env triage

serverEnv() validates process env; it does not decide the source of truth. Classify first:

  • Secret: leaking it requires rotation or incident response. Route through OpenBao/ESO or the proper GH secret tier.
  • Plain config: owner slugs, repo names, public URLs, feature modes, and routing values. Route through GitOps ConfigMaps or repo config.

For either path, k8s object presence is not process proof. Pods read env only at startup, so prove: source object -> Deployment envFrom -> restarted pod env -> public health. Treat old Argo/workflow failures as leads until live cluster checks agree.

App flight substrate assertions are read-only (operator lane)

A node workload on the deploy verb gets its runtime env from OpenBao, and the node's .cogni/secrets-catalog.yaml is the ONLY place a node declares a secret (bug.5254): the app gets the standard node keys plus every catalog entry whose workloads names the app service; a worker gets only the entries whose workloads names it. Repo-spec secret_refs is retired: a repo-spec that still carries it is refused (422 secret_ref_refused, a fixed message pointing at the catalog), never stripped. The verb reads the node catalog at sourceSha and refuses an invalid one (422 secrets_catalog_refused). When the release records, the operator creates a missing source: agent key the release needs (create-if-absent; only if it custodies the lane and the caller holds node.manage_secrets; otherwise the release waits on missing_keys). secret-materialize.sh below reads nodes/<n>/.cogni/secrets-catalog.yaml only inside the operator monorepo; it never sees a sovereign node's catalog. The call carries no secret value (deploy-a-declared-workload). The rest of this section describes the operator flight lane (operator only).

candidate-flight.yml uses scripts/ci/assert-target-substrate.sh as a preflight for selected app rollouts. That gate may verify a Deployment-consumed k8s Secret exists and its matching ExternalSecret is Ready, but it must not seed OpenBao, patch GitHub secrets, run deploy-infra.sh, or repair Compose/env state. A missing secret is a substrate failure. For a wizard-created ordinary node, there should be zero per-node human secret values. The environment must already have the substrate/runtime inputs classified in https://node-template.cognidao.org/knowledge/secrets-routing-tiers. If an environment-bank value is missing, repair that bank; do not pass values through candidate-flight inputs or store them in the wizard.

"Read-only" applies to assert, not the whole flight. Distinct from the read-only assertion, candidate-flight.yml / promote-and-deploy.yml run a materialize-substrate job first — the sole OpenBao writer in the flight (scripts/ci/secret-materialize.sh, <env>-writer token) — which generates the node's source: agent secrets at cogni/<env>/<node>/* idempotently before reconcile/assert, including the per-node DB creds (app_<node>/service_<node> passwords) and the composed Postgres DSNs (DATABASE_URL/DATABASE_SERVICE_URL; only DOLTGRES_URL is still deferred) since #1584 (canonical: https://cognidao.org/knowledge/operator-pipeline-contract Axiom 22, SUBSTRATE_IS_RECONCILED_BEFORE_PROMOTION). Since #1584 reconcile-substrate is read-only on OpenBao (<env>-db-reader token, zero bao kv put/patch) and assert-target-substrate.sh is read-only too — so only materialize-substrate writes. bug.5007 status (corrected 2026-06-24): the openbao-operator ServiceAccount that materialize-substrate uses to mint <env>-writer does exist on prod (default ns, 21d), and operator-secrets-writer@cogni-production exists too (6d) — the "SA absent on prod" framing is stale. The self-serve write path is proven (operator returns 200). The original bug.5007 promote-pipeline e2e (materialize-substrate green on a real prod promote) is the part still worth a direct confirm; the SA/role prerequisite it blocked on is in place.

The target shape matters. Today the implemented branch is type=node; a future type=service branch should assert the service's declared Secret / ExternalSecret / ConfigMap contract without inheriting node DNS, Caddy, NodePort, or node-DB assumptions.

Decision tree — how do I write / rotate the value?

OperationRight patternToday's reality
Add new secret SHAPE (service X consumes key A)PR → vault-config-operator CRD → Argo reconcilesNot built; tracked in proj.agentic-fork-bootstrap Walk
Materialize a new wizard node's runtime secretsGenerate/derive node-local values and inherit only explicit org/env grants; no per-node human values for ordinary nodesShipped: materialize-substrate is its own flight job (scripts/ci/secret-materialize.sh, <env>-writer token), idempotent read-once→diff→write-missing (#1582/#1585; https://cognidao.org/knowledge/operator-pipeline-contract Axiom 22). The shared-bank / explicit inheritFrom-grant model is the follow-up
Rotate AUTO-GENERATED value (e.g., AUTH_SECRET)rotate-secret.yml workflow with env-protection; auto-generates value; human approves event, never sees valueNot built; do manual openssl rand + pnpm secrets:set per secrets-rotate.md
Rotate VENDOR-MINTED value (OpenAI key, Cherry token)Operator API route POST /api/v1/nodes/[id]/secrets, OpenFGA can_manage_secrets — caller holds only an API keyShipped + live on candidate-a AND production (operator returns 200). preview still 503 authz_unavailable (OpenFGA store not resolvable by the pod). For preview/cross-env: CLI + admin/kube custody. See "Who holds secret-write authority" above
Candidate-a experimentationpnpm secrets:set <env> <service> <KEY> via port-forward + writer-role JWTShipped — see secrets-add-new.md
Dynamic DB credentialsOpenBao DB engine, no human in loopFuture (Crawl row 3 of proj.security-hardening)

Rotation is not done when the write returns 200. "Written to OpenBao" and "the consumer holds it" are two facts with an ExternalSecret refreshInterval between them. A pod that reads its credential once at start is indistinguishable from healthy while holding a revoked one — proven live 2026-09-15: a rotated Akash key sat in OpenBao while the actuator ran 1/1 Running on a dead credential.

Rotate in four steps, never three:

write -> kubectl annotate externalsecret <name> force-sync=$(date +%s) --overwrite
      -> Reloader restarts the consumer (already automatic, Invariant 11)
      -> VERIFY the consumer (read it back; bug.5142 credential fingerprint)

Do NOT shorten refreshInterval to compensate. The class table in secrets-management.md is the sanctioned set (1h routine / 24h external API / 15m dynamic DB / 5m critical+wallet). 1m was proposed twice and reverted: 60x the OpenBao reads forever, against a Shamir 1-of-1 OpenBao that has already OOMKilled (bug.5011). And do not make the operator push the sync — it has no ServiceAccount and no k8s identity, so that means giving the internet-facing app RBAC to patch ExternalSecrets.

The killer rule: no human types a secret VALUE into a UI in production. Auto-generated, vendor-minted via operator-app, or dynamic. Form-input is the anti-pattern.

Who holds secret-write authority — TODAY vs TARGET (READ THIS)

The honest answer to "can a node add its own secret without Derek's GitHub/kube creds?" is somewhere in between, closer to Derek/GitHub-admin-controlled today. Do not claim "operator-app RBAC" where it isn't yet shipped. Three distinct authorities — keep them separate:

AuthorityWhat it decidesTODAY (custody)TARGET
Secret declaration (shape)"node Y is allowed key KEY_X"GitHub repo merge authority — whoever can merge a PR to the catalog. Guarded by required CI + Cogni Git PR Review + branch protections. No value in the PR.Same (PR-gated). This rung is fine.
Generated value (source: agent)random bytes minted for node YGitHub Actions materialize-substrate job: the flight workflow SSHes the env VM and mints an <env>-writer OpenBao token via the in-cluster openbao-operator SA. Real and shipped for candidate-a/preview; the openbao-operator SA also exists on prod (21d) — the old "prod lacks the writer SA (bug.5007)" line is stale. "Who can cause it" = whoever can merge/dispatch the env's deploy workflow → GitHub merge authority again.In-cluster Job (SA-bound <env>-writer), zero SSH (design §North-star Stage 2).
Human/vendor value (source: human: OpenAI key, Cherry token)the byte value a vendor mintedOperator-controlled CLI/kube path (pnpm secrets:set + short-lived writer JWT via port-forward). In this workspace that effectively means Derek/admin-held repo + kube custody.Operator app route + OpenFGA RBAC (see below) — prototyped, not merged.

The gap that keeps us on Derek's creds: there is no shipped operator-app secrets route and no .github/workflows/secret-set.yml. Every value-write today bottoms out in GitHub-merge authority or operator/admin kube custody — i.e. Derek.

The precise PR/design that closes it — the value-write complement to the flight triangle (developer → can_flight → operator-held GitHub App creds), one rung over with its own least-privilege role (secrets_manager → can_manage_secrets → operator-held OpenBao writer; distinct from developer, mirroring production_promoter):

  • Design: https://cognidao.org/knowledge/node-secret-write-design (operator-mediated, OpenFGA-authorized). Sibling: node-wizard-secret-setting.md (shape, shipped).
  • PR: #1627 (prototype) — POST /api/v1/nodes/[id]/secrets + OperatorSecretsPlanePort + OpenBaoSecretsAdapter (operator pod self-logins with its OWN projected SA token, audience: cogni-openbao) + can_manage_secrets OpenFGA relation + build-time A2 allowlist. Caller holds only an API key — no kubeconfig, no vault token.
  • Per-env readiness is live state — recall the hub guide node-self-serve-secrets, don't trust a git snapshot. The provisioning prereqs + the 503 authz_unavailable OpenFGA-model-drift class are above; the durable cross-env fix is task.5049.

Three defense-in-depth gates in #1627: OpenFGA can_manage_secrets (fail-closed) → catalog A2 allowlist (build-time typed module) → OpenBao policy explicit-deny on _system/_shared.

Validation status + the road to a proper e2e: https://cognidao.org/knowledge/node-secret-write-design — the layered proof (L1 candidate-a authz done; L2 full write needs the allowlist codegen + the OpenBao writer-role DRY fix; L3 human-in-loop deployed e2e). Load-bearing gotcha: the write env is operator-stamped (deployEnv = env.DEPLOY_ENVIRONMENT), so cogni/<operator-env>/<node>/* is the only path it can write — a prod operator cannot write a candidate-a node's secret. Any cross-env "set my candidate-a secret from the prod operator" needs an explicit authz-scoped targetEnv param (not in #1627).

Placement is not provisioning — the empty-node-path class (READ THIS)

Symptom: a node deploys fine in one environment and its ExternalSecret fails in another with could not get secret data from provider / cannot find secret data for key: "AUTH_SECRET" — failing at spec.data[0], i.e. the node's OpenBao path is empty, not one key short.

⚠️ FIXED — rule 1 below shipped. Do not diagnose a live node from the Historical cause. An Akash-placed node is materialized today; see "As-built" before concluding anything. This section is kept for the invariant and the failure signature, not as a description of current behaviour.

Historical cause (superseded): the node was moved to external compute (deployment_provider.<env>: akash) and thereby dropped out of the job that provisions it. node-substrate — which runs secret-materialize — was matrixed on k3s_node_targets_json, so flipping a node's provider silently removed it from the population that got served. Nothing errored; nobody created its secrets.

As-built (verify before you trust either version): the population was split, not narrowed. Both lanes run the same script — scripts/ci/run-node-substrate.sh, whose header calls itself "the ONE per-node substrate runner" and whose case "$DEPLOYMENT_PROVIDER" accepts k3s|akash. It calls secret-materialize.sh for either provider.

lanejob / writermatrix
k3snode-substratek3s_node_targets_json
Akashnode-deploy sequencer (POST /deploy/workloads)none (no flight job, story.5137)

The trap that outlived the bug: the JOB node-substrate and the SCRIPT run-node-substrate.sh share a name but serve different populations. Tracing the job alone shows a k3s-only matrix and looks like proof that Akash nodes are never materialized — a wrong, confident conclusion. The disconfirming evidence is cheap: check which rows declare deployment_provider.candidate-a: akash and carry complete buckets including LITELLM_VIRTUAL_KEY. If Akash nodes were never materialized, those could not exist. Grep for every run-node-substrate.sh call site, not just the obvious job — there are two.

Why the original defect hid: the first external nodes were k3s nodes first. They kept the secrets they already had, so test and preview looked healthy while inheriting from a k3s past. The defect only surfaced in an environment where that node never ran on k3s. Do not treat a green lower environment as evidence.

Two rules:

  1. Branch the steps, never the population. Every type: node target gets materialized regardless of provider; only provider-specific work (k3s Caddy/NodePort edge reconcile vs external DNS reconcile) is selected inside the job. The typed planner already emits the correct wider node list — prefer it over the k3s-only one.
  2. A provider that needs a different secret needs a catalog entry, not a rename. External compute must not receive fleet-shared credentials (they would reach a rented third-party host), so it consumes a node-scoped equivalent — e.g. LITELLM_VIRTUAL_KEY in place of the shared LITELLM_MASTER_KEY. That is correct, but it is only real once the catalog declares it with a generator. A key that has a consumer and no declaration gets hand-seeded once and silently carried forever (CATALOG_IS_THE_ONE_READER).

Fast triage: read key NAMES only, never values — bao kv get -format=json cogni/<env>/<node> and compare the environment that works against the one that does not. If the failing path is empty rather than short, this is your class.

Spec: https://cognidao.org/knowledge/secrets-management invariants 17-18; https://cognidao.org/knowledge/node-baas-substrate-model § Placement vs Provisioning.

Dual-plane secrets — the silent-webhook-fail class (READ THIS)

Some secrets must byte-equal a value held by an external system, not merely exist. The operator's GH_WEBHOOK_SECRET must equal the GitHub App's webhook secret; an OAuth *_CLIENT_SECRET must equal the provider's app config. These are dual-plane: one copy in our pod, one on the external plane — they only work if identical.

The trap (live bug, preview 2026-06-03): GH_WEBHOOK_SECRET was source: agent with no syncTo, generated randHex 32 every provision → it could never match the App's webhook secret → every webhook failed HMAC verification, silently (a level:40 warn component:webhook-route msg:"webhook verification failed", no alert, no 5xx). The App just looks dead — no PR reviews, no node-wizard. deploy-infra re-applying the Secret was the breaking path, not the healing one — a generated value never equals an externally-held one unless something pushes it there. That push is what syncTo declares.

syncTo is an external mirror, not custody. origin says who can produce the bytes, custody says where the value is authoritative, and syncTo says whether a copy must also be pushed to an external system.

axisvaluesanswers
source:agent (we generate, generate: required) · human (un-generatable; vendor-minted, human supplies once, e.g. GH_REVIEW_APP_PRIVATE_KEY_BASE64)where the value originates
syncTo: (opt)github-app-webhook · (unset)does it also live in an external system we must keep in lockstep

GH_WEBHOOK_SECRET is source: agent + syncTo: github-app-webhook — we generate it (origin is internal; calling it source: external lied about that), and it ALSO must byte-match the App. The materializer/provisioner generates it into OpenBao; the infra lane then pushes it (scripts/secrets/sync-app-webhook-secret.sh: App JWT → PATCH /app/hook/config):

APPID + GH_REVIEW_APP_PRIVATE_KEY_BASE64 → RS256 JWT (iss=APPID, exp≤10m)
PATCH /app/hook/config  -d '{"secret":"<generated GH_WEBHOOK_SECRET>"}'   # endpoint EXISTS

No-human-secret done right: agent generates, agent pushes, zero human, self-healing (provisioning owns both copies → every infra-lever deploy re-converges). Do NOT make it source: human/carry — that drags a human into the App's "Change secret" field for a value that's ours to generate.

⚠️ Sync only fires on the infra lever (deploy-infra via candidate-flight-infra / provision-env), NOT on app-lever promotes (candidate-flight/flight-preview/promote-and-deploy = Argo image bump, never touches the Secret). assert-target-substrate.sh is also read-only: it can fail a flight when the Deployment-consumed Secret / ExternalSecret is absent or not Ready, but it does not heal the value. In an established ESO environment, deploy-infra must source GH_WEBHOOK_SECRET from OpenBao before invoking the sync and must fail closed if the App PATCH fails; otherwise the pod plane and GitHub App plane can silently diverge again. Fresh/plain-Secret bootstrap may still use workflow env input until the operator ESO target exists.

Heal-proof test = redeploy twice; a PR on the test repo must still post a cogni-git-review review.

OpenBao availability — sealed = DOWN, not secure (READ THIS)

Counterintuitive but load-bearing: a sealed OpenBao serves nothing — it 503s every ESO sync, every materialize/reconcile-substrate auth/kubernetes/login, and openfga's config load. OpenBao must run UNSEALED ~100% of the time; unsealed-and-running is the correct operational state. The seal protects data at rest (stolen storage = encrypted/useless), not runtime access — that's auth methods + path policy (Invariant 6) + in-cluster-only (no Ingress). So "keep it sealed / minimize unsealed time" is backwards: an unexpected seal is an outage, not a safe state.

The fragility (bug.5011, prod outage 2026-06-11 02:25): prod OpenBao is Shamir 1-of-1, no auto-unseal. It OOMKilled on the 6 GB box → resealed → secret plane down (openfga 503, node-substrate promote failures) until a human kubectl exec … bao operator unseal. Any restart (OOM, reboot, chart upgrade) reseals it.

Recorded decision (bug.5011) — do not re-litigate without the trigger:

  1. Memory headroom — OpenBao limit 512Mi→1Gi (#1617) so it isn't container-OOMKilled; #1616 lean-prod cut node pressure too. Mitigates the OOM trigger only.
  2. Sealed-state → Loki alert — the no-silent-outage control: detect an unexpected seal, page, unseal in minutes. Free, no vendor. This is the durable fix, because you cannot prevent every reseal.
  3. Auto-unseal (KMS) DEFERRED behind a written trigger: reseals recur post-#1617, OR a SOC2 engagement starts. KMS is the audit-correct control but it is our first AWS/GCP dependency on a Cherry+OSS stack — premature at 0 users. If a real SOC2 deadline lands, flip it to now.
  4. REJECTED: in-cluster stored-key unseal sidecar — key-next-to-lock, fails SOC2.

Anti-patterns — instant reject

  • Human typing a secret VALUE into a production UI or GitHub workflow input. See killer rule.
  • Treating tier: B, GitHub Environment Secrets, or VM .env as authority for a runtime secret. If it feeds a pod or a pod-facing role, OpenBao owns it.
  • A flight or deploy call accepting secret values as inputs. The operator flight may invoke materialization for safe generated/derived node values and fail if the environment's required DAO/org bank is missing; it must not carry values.
  • A dual-plane secret (must byte-match an external system — GitHub App webhook secret, OAuth client secret) declared with no syncTo: — it silently fails verification forever and deploy-infra re-breaks it every run. Add syncTo: (keep source: agent if we generate the value). See "Dual-plane secrets" above.
  • Generic catch-all workflow (secrets-manage.yml-shaped). Per-operation only.
  • ssh root@vm kubectl ... or ssh root@vm bao .... Use local kubectl + port-forward + writer-role JWT.
  • Treating k8s Secret or ConfigMap presence as proof that a running pod has the value. Prove the process after rollout.
  • Treating a failed app-flight substrate assertion as permission to run deploy-infra from the app flight. Heal secrets/substrate through the owning secrets or infra lane; keep app flight read-only until image promotion.
  • Re-exporting .local/<env>-openbao-root-token after Phase 5b — violates Invariant 13.
  • bao kv put instead of bao kv patch (replaces sibling keys).
  • bao login -method=kubernetes in OpenBao CLI 2.5.x — that subcommand doesn't exist; use raw API: bao write auth/kubernetes/login role=X jwt=Y.
  • Per-secret ExternalSecret YAML — violates Invariant 2.
  • valueFrom: secretKeyRef per env var in pod spec — violates Invariant 3.
  • Base64-in-git "encryption" — violates Invariant 4.
  • Sealed Secrets / SOPS+ksops — explicitly rejected per proj.security-hardening Design Notes.
  • Editing scripts/setup-secrets.ts to add/remove a SECRETS array entry — there isn't one. The script loads YAML via scripts/lib/secrets-catalog-loader.ts. Edit the appropriate secrets-catalog.yaml instead.
  • ExternalSecret manifests under infra/k8s/secrets/external-secrets/<env>/<node>/ — that pre-wizard tree is PURGED. Remote-source/wizard nodes carry their leaf in the node repo at k8s/external-secrets/<env>/, mounted by the operator as nodes/<node>/k8s/external-secrets/<env>/ when pinned. Only the cluster-scoped ClusterSecretStore remains under the old dir.
  • Treat the operator overlay-local ExternalSecret as a legacy operator exception, not the node standard. infra/k8s/overlays/<env>/operator/external-secret.yaml is Argo-owned because the operator Deployment patch lives in that overlay; source name env-secrets renders through namePrefix: operator- to object operator-env-secrets, while spec.target.name stays operator-env-secrets.

The catalog (per-node YAML + Zod loader)

scripts/setup-secrets.ts does NOT hold a hardcoded SECRETS array. It calls a Zod-validated loader that walks YAML catalogs:

FileDomainHolds
infra/secrets-catalog.yamloperator-domain_shared, _system, B/D/E/G entries, A2 placeholders, and (today) all node A1/A2 entries — node-template's catalog was migrated here by task.5094
nodes/<node>/.cogni/secrets-catalog.yamlnode-domain (single-node-scope: a node engineer adds their node's secret in ONE PR)A1/A2 entries the node owns; service: auto-fills from parent dir. Loader-supported but currently empty — no node populates it yet (post-task.5094 consolidation)
scripts/lib/secrets-catalog-loader.tsoperator-domain (substrate)Walker (walks both paths above) + uniqueness + service-allowlist assertions; the entry Zod schema lives in the published @cogni-dao/node-declarations package, re-exported by @cogni/repo-spec (shared with the node-deploy verb, bug.5254)

To add a node secret today: in the operator monorepo, edit infra/secrets-catalog.yaml — node entries are consolidated there post-task.5094. In a sovereign node repo on the deploy verb, add one entry to the node's own .cogni/secrets-catalog.yaml with workloads: [<repo-spec service>]; that is the whole declaration (no repo-spec edit). To add an operator-domain secret (B/D/E/G, or _shared cross-cutting): edit infra/secrets-catalog.yaml. Operator-domain PR. The loader rejects at module load: missing tier, name collision across files, per-node service: mismatch with parent dir, unknown service: value not in the allowlist (_shared/_system + present nodes + canonical-future-domain names from https://node-template.cognidao.org/knowledge/node-ci-sovereignty).

Files to read by topic

If you're doing…Read
Understanding the tier system + invariantssecrets-routing-tiers
Designing node-wizard secret behaviorhttps://cognidao.org/knowledge/node-secret-materialize-phases
The layered authority model (Identity → DAO)operator-identity-model
Adding a new secret to your nodeEdit nodes/<node>/.cogni/secrets-catalog.yaml + read https://cognidao.org/knowledge/break-glass-secret-write
Adding a _shared / B / D / E / G secretEdit infra/secrets-catalog.yaml
Rotating an existing secrethttps://cognidao.org/knowledge/break-glass-secret-write
Following the bootstrap flowhttps://cognidao.org/knowledge/fork-quickstart-guide
Provisioning a new env (candidate-*, fork)Dispatch .github/workflows/provision-env.yml (-f env=<env> -f encryption_passphrase=…). Full walkthrough — 7 minting secrets + init-artifact custody — in https://cognidao.org/knowledge/fork-quickstart-guide §6. The runner owns the tofu+bao+kubectl session; the operator's laptop never holds substrate creds.
Adding a new service (new k8s Deployment)https://cognidao.org/knowledge/run-node-formation-wizard + add ExternalSecret under nodes/<node>/k8s/external-secrets/<env>/
Touching substrate provisioningscripts/setup/provision-env-vm.sh Phases 5b.1–5b.5
Touching the CLIscripts/secrets/set-secret.sh + test scripts/ci/tests/set-secret.test.sh
Touching the loader / catalog schema@cogni-dao/node-declarations (entry Zod schema + sovereign-node check; Release pin in packages/repo-spec/package.json) and scripts/lib/secrets-catalog-loader.ts (walker)
Touching any node's ExternalSecretRemote-source/wizard nodes: nodes/<node>/k8s/external-secrets/<env>/ (per-node, node-domain) — the single repo-wide convention. Operator legacy overlay: keep infra/k8s/overlays/<env>/operator/external-secret.yaml aligned until operator moves to the same node-domain leaf. No aggregator; only the cluster-scoped ClusterSecretStore lives under infra/k8s/secrets/external-secrets/.
Touching substrate Argo Applicationsinfra/k8s/argocd/{openbao,external-secrets}-application.yaml
Touching the env-var classification routing.claude/commands/env-update.md — k8s app vs Compose-infra split
Granting Grafana/Loki query access (dev, validator, or a gate)node-log-proxy-contract — one admin root → scoped consumers. Dev node-read = operator query-PROXY pinned to {node="<id>"}; the dev holds NO token (a returned token's reach escapes the per-node OpenFGA check → dormant env-wide leak; never hand a dev a Grafana token). Real blocker: node Loki streams have no node label yet (app/env/service only). assertLive gates on the PUBLIC run-carries rung; the operator's gate holds no query token. Out of MVP scope: label-scoped glc_ / per-dev SA minting. Cross-substrate plane + health scorecard: read-fleet-health.
Designing a new workflow that handles secretsThis file + proj.agentic-fork-bootstrap anti-patterns. Run it past the killer rule.

When to escalate

Surface to operator before writing code if:

  • Adding a NEW entry point that isn't already CLI / workflow_dispatch / operator-MCP — Invariant 9 lists the only three sanctioned shapes.
  • Changing eso-reader policy or <env>-writer role binding — affects every consumer.
  • Bumping OpenBao or ESO chart version — rotation drill required (see secrets-rotate.md §Upgrade discipline).
  • Anything that smells like Invariant 4 (NO_VALUE_IN_GIT) — finding a value in YAML / commit message / PR diff / chat is always a rotate-now event.
  • Designing a workflow where humans type values into a form — recheck against the killer rule before building.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall