cogni-dao/cogni / .claude/skills/cicd-secrets-expert
cicd-secrets-expert
VerifiedSecrets architecture reference for node-template — when to use OpenBao+ESO vs GitHub env secrets, which operation pattern fits which write/rotate/add flow, the load-bearing invariants from the spec, the YAML catalog + Zod-loader the script consumes, and where to find the canonical implementation. Use when adding/rotating a secret, designing a service that consumes secrets, debugging an ExternalSecret or writer-role login, deciding between substrate and Compose-infra routing, touching `pnpm secrets:set` / `scripts/secrets/` / `scripts/lib/secrets-catalog-loader.ts` / `nodes/<node>/.cogni/secrets-catalog.yaml` / `infra/secrets-catalog.yaml` / `infra/k8s/argocd/{openbao,external-secrets}/` / per-node ExternalSecret manifests, or evaluating any new workflow that touches secret values. Triggers: 'add a secret', 'add a node secret', 'rotate a key', 'OpenBao', 'ESO', 'ExternalSecret', 'secrets-catalog', 'catalog tier', 'A1', 'A2', 'B-tier', 'writer role', 'bao login', 'vault-action', 'vault-config-operator', 'secrets-manage', 'secret in GH env vs OpenBao', 'where do I put this credential', 'per-node catalog'.
CI/CD Secrets Expert
Route the work through operator-architect first; this skill adds the secrets architecture rules on top of it.
One-page reference for anyone touching secrets in node-template. Read this BEFORE the spec; this points at what to actually read.
Node self-serve secrets — the value-write path
A node-owner sets a secret with only an API key — POST <env-host>/api/v1/nodes/<id>/secrets {env, key, value} → 200 written, version, written to cogni/<env>/<node>/<KEY>, then ESO + Reloader carry it into the pod.
The node-dev contract + the LIVE per-env status are single-sourced in the hub guide node-self-serve-secrets (GET /api/v1/knowledge/node-self-serve-secrets) — not snapshotted in git. Design canon: https://cognidao.org/knowledge/node-secret-write-design. The durable, non-obvious facts (why this skill exists):
- Per-env sealed operators — NO cross-env write. Each env runs its own operator pod; the OpenBao policy (
cogni/<env>/*,_system/_shareddenied) makes a candidate-a token physically unable to write prod. The auth chain: operator pod's projected SA token (audience: cogni-openbao) →auth/kubernetes/login(role<env>-node-secrets-writer, SAoperator-secrets-writer@cogni-<env>) → KV put/patch. No kube/SSH; caller holds only an API key. - RBAC is granular:
can_manage_secrets ← secrets_manager, NOTdeveloper.developer → can_flight(logs/flight);secrets_manageris a distinct grant for secret writes (both ∪admin). Requestingdeveloperand expecting secret-write is the trap — 403authz_deniedis the right answer to the wrong role. envis an explicit validated body param (D1, #1754), == the operator's ownDEPLOY_ENVIRONMENTelse409 wrong_operator_env. Node resolved viaresolveNodeRef(status-agnostic) through the sharedwithNodeRbacseam (src/app/_lib/node-rbac.ts, #1771).- Per-env provisioning + the three 5xx (diagnose by status, not by guess): the write needs, per env, the
<env>-node-secrets-writerOpenBao role + the operator overlay (OPENBAO_NODE_SECRETS_WRITER_ROLEviaenvFrom+ a projected SA token,audience: cogni-openbao) + Reloader present + the OpenFGA model carrying the checked relation. Failure modes:503 secrets_plane_config_missing=OPENBAO_NODE_SECRETS_WRITER_ROLEunset on the pod;502at write = the OpenBao role/login is missing (vs the env var);503 authz_unavailable= the env's OpenFGA model lacks the checked relation. That last one is a recurring silent-drift class (task.5049):bootstrap-openfga.shonly runs insidedeploy-infra(skip_infra=false), so normal promotes never re-bootstrap — a new RBAC relation (e.g.can_manage_secrets, #1627) silently never reaches an env, and secrets503while older actions403. Fix: POST the currentrbac-model.jsonto that env's OpenFGA store + repointcogni/<env>/operatorOPENFGA_AUTHORIZATION_MODEL_ID+ let Reloader roll the operator. (Live per-env readiness: recall the hub guide, above.) source: agentkeys are off-limits (denylistnode-secrets-reserved.data.ts, #1753) — they're substrate-minted per node; clobbering one breaks the node.- Control-plane framing:
OperatorSecretsPlanePortis the secrets row of the typed operator control plane (cicd-platform-boundary.md) — sibling ofOperatorDeployPlanePort; one port per substrate (Argo/OpenBao/Cherry-Akash), unified by the(node_id, env)+OpenFGAcontract. A node is a bundle of services (node → services → deployments, node BaaS substrate model), not one deployable.
North star
proj.agentic-fork-bootstrap — easy-start guide for a forking dev that uses OpenBao. Every PR is measured against the forker's manual-command count. If your change adds a manual step to fork-quickstart.md, that's debt — try a workflow first.
Load-bearing invariants — gate every secrets decision
Load-bearing subset — canonical numbering is secrets-management Invariants 1–16; the rows below are the ones that gate day-to-day secrets work (7, 10–12, 14 omitted — read the spec for the full set):
| # | Rule | Where it bites |
|---|---|---|
| 1 | PATH = cogni/<env>/<service>/<KEY>; <service> = catalog name | New service → new ExternalSecret dir |
| 2 | ONE ExternalSecret per (service, env) with dataFrom: extract; target <service>-env-secrets | Adding keys = NO YAML edit |
| 3 | Pod envFrom: secretRef: name: <service>-env-secrets once per container | Pod spec set ONCE at service creation |
| 4 | NO secret value in git — ever | Base64-in-YAML = immediate rotate + audit |
| 5 | OpenBao is runtime SSOT; VM .env files are rendered views, not authorities | Don't seed runtime values in two places |
| 6 | RBAC via path policy (eso-reader, <env>-writer) bound to k8s SAs | Phase 5b.3 + 5b.4 of provision-env-vm.sh |
| 8 | Every access audited via OpenBao audit device → Loki | Pipeline not built yet — bug.0445 follow-up |
| 9 | Three entry points only: CLI / workflow_dispatch / operator API (POST /api/v1/nodes/<id>/secrets, OpenFGA can_manage_secrets — shipped #1627). Never raw bao kv put | See decision tree below |
| 13 | NO_OPERATOR_ROOT_TOKEN_ON_LAPTOP — bootstrap window only; day-2 uses writer-role JWT | .local/<env>-openbao-root-token is never read post-Phase-5b |
| 15 | Pod-facing DB role material is OpenBao-owned, even when Compose renders a copy | No DB password authority in GitHub env or VM .env |
| 16 | New-node secret materialization precedes substrate reconcile/assert | Generate safe agent values before first flight |
Authority model — classify by three axes
Do not use tier as the authority model. Every secret has:
| Axis | Values | Question |
|---|---|---|
origin | agent · human · derived | Who can produce the bytes? |
custody | openbao · github-env · repo-config | Which system is authoritative? |
consumers | pod · compose · ci · external | Where does the value get used? |
Hard rule: if a value is consumed by a pod, provisions a pod-facing role, or
must agree with a pod-facing value, custody is OpenBao. GitHub Environment
Secrets may bootstrap access and carry CI-only credentials. VM .env files are
rendered views for Compose, never authorities.
Examples:
POSTGRES_ROOT_PASSWORD: Compose/bootstrap-only for now, because pods should not consume it.APP_DB_PASSWORD,APP_DB_SERVICE_PASSWORD,APP_DB_READONLY_PASSWORD,DOLTGRES_PASSWORD,DOLTGRES_READER_PASSWORD,DOLTGRES_WRITER_PASSWORD: OpenBao custody; Compose renders copies to create roles. (DOLTGRES_PASSWORDis the env Doltgres superuser the pod itself authenticates as — Doltgres RBAC is table-DML-only, so no per-node role; why + value-shape in https://node-template.cognidao.org/knowledge/database-migration-contract §5.2.)OPENFGA_DB_PASSWORD: OpenBao custody. OpenFGA is the first shared-infra DB given a dedicated, non-root login role (its ownopenfgarole, OpenBao-sourced — Phase A dropped root from the datastore DSN). That's the hardening direction, not a snowflake:litellm/postgresare still the un-hardened root-owned legacy. Consequence — shared-infra DBs (openfga, litellm) are provisioned in theINFRA_ONLYpass only; the per-node provision pass must not touch them (it holds no shared-infra password).litellmsurvives the per-node pass only by being root-owned (no password to demand); openfga's dedicated role exposed that the per-node pass should never provision shared infra.provision.shnow gates openfga toINFRA_ONLY; gatinglitellmthe same way is the fuller fix. Shared-infra DB-cred rule (learned the hard way, fleet reprovision 2026-08-04 — applies to any NEW one): (1) a shared-infra DB-role password (OPENFGA_DB_PASSWORD,TEMPORAL_DB_PASSWORD) is NOT a catalog entry (infra ≠ node; the loader allowlist is node-domain) —provision-env-vm.shPhase 5c generates it (randHex 32) + seeds it set-once (reuse the existing OpenBao value on re-provision, never regenerate → the created role can't diverge, the 2026-06-11 temporal-drift class). (2)deploy-infraMUST read it via the ungatedopenbao_get_field(${env}-db-readerseam) — NOTsource_openbao_runtime_key, which is gated onoperator-env-secrets(OPENBAO_RUNTIME_SSOT=true= "established ESO env"). That gate is false on a fresh provision (no namespace/secret yet), so arequiredread there silently skips and the.envrender tripsset -u→<VAR>: unbound variable→ deploy-infra aborts before the app layer, leaving a 0-byte runtime.env(a red-herring symptom). Wiring a consumer (compose DSN +provision.shrole) without also wiring the producer (Phase 5c seed) + the ungated read leaves the env unprovisionable-from-clean but green on established re-deploys — latent until the next from-scratch provision. (3) The superuser password MUST be minted set-once (randHex), NEVER derived from another secret.DOLTGRES_PASSWORDwas the one left out of the 2026-08-04 set-once fix:deploy-infrafell back toderive_secret doltgres-root = sha256(POSTGRES_ROOT_PASSWORD), so a root rotation (or genesis-derive ≠ a later value) drifted the rendered password from the volume's superuser, which is baked at first-init and IMMUTABLE (Doltgres can'tALTERits superuser; postgres/temporal can, via the local-trust socket — that's why they have an in-place Step-6.5 reconcile and doltgres cannot). Fix = seed set-once atcogni/<env>/operator/DOLTGRES_PASSWORD, mirroring OPENFGA/TEMPORAL → the superuser can never drift. (4) NEVER "heal" a credential drift by recreating/wiping the DB volume — destroying data to fix a password is a footgun (a transient auth-probe miss then nukes a live store). Prevent drift with (3); an already-drifted volume is recovered by an explicit operator action (wipe only if provably empty; otherwise re-pin the SSOT to the volume's value), never an automatic deploy-path behavior.- Public URLs / owner slugs / feature modes: repo-config, not OpenBao.
Biggest systemic shortcoming — the .env/_shared split-brain
One logical shared secret becomes three physical copies — _shared SSOT, per-node inherited copy (ESO→pod), VM .env render (Compose) — written by three writers with no lockstep. They drift: the recurring root of secrets outages (litellm / temporal / doltgres / billing 401·28P01). The tell: pods have no .env and rarely drift; every .env-bound Compose service is the surface. Never heal toward .env (Invariant 15); the endgame is to eliminate the copies (purge the _shared bucket + purge the server .env → Bao Agent / k8s). Mechanism, evidence, three-layer fix, falsification gate → node-self-serve-secrets.
Routing tree — where does the value render?
| Tier | Consumed by | Render path | Custody |
|---|---|---|---|
| A1 | k8s pod baseline (anything under nodes/<n>/app/, every fork) | OpenBao cogni/<env>/<service>/* → ESO → k8s Secret → envFrom | OpenBao |
| A2 | k8s pod node-specific (any downstream <node>) | OpenBao cogni/<env>/<node>/* → ESO → k8s Secret → envFrom | OpenBao |
| B | Compose-infra service (postgres, litellm, temporal, redis, alloy, caddy) | Rendered to VM .env or future Bao Agent | OpenBao unless CI/bootstrap-only |
| D | CI-only (workflow consumption, never runtime) | GH Env Secret → workflow env: block | GH Environment Secrets |
| E | Repo-level CI (cross-env, one value per repo) | GH Repo Secret | GH Repo Secrets |
| F | Local dev only | .env.local (gitignored) | Operator's laptop |
| G | Derived from repo state at provision time (e.g. nodes/* listing) | Computed by loader; written alongside other catalog values | Auto-generated |
Full tier definitions + invariants: secrets-routing-tiers.
Identity and RBAC layers: operator-identity-model, operator-rbac-capability-map.
Routing checklist (file-by-file propagation): .claude/commands/env-update.md §0.5.
Runtime env triage
serverEnv() validates process env; it does not decide the source of truth.
Classify first:
- Secret: leaking it requires rotation or incident response. Route through OpenBao/ESO or the proper GH secret tier.
- Plain config: owner slugs, repo names, public URLs, feature modes, and routing values. Route through GitOps ConfigMaps or repo config.
For either path, k8s object presence is not process proof. Pods read env only at
startup, so prove: source object -> Deployment envFrom -> restarted pod env ->
public health. Treat old Argo/workflow failures as leads until live cluster
checks agree.
App flight substrate assertions are read-only (operator lane)
A node workload on the deploy verb gets its runtime env from OpenBao, and the node's .cogni/secrets-catalog.yaml is the ONLY place a node declares a secret (bug.5254): the app gets the standard node keys plus every catalog entry whose workloads names the app service; a worker gets only the entries whose workloads names it. Repo-spec secret_refs is retired: a repo-spec that still carries it is refused (422 secret_ref_refused, a fixed message pointing at the catalog), never stripped. The verb reads the node catalog at sourceSha and refuses an invalid one (422 secrets_catalog_refused). When the release records, the operator creates a missing source: agent key the release needs (create-if-absent; only if it custodies the lane and the caller holds node.manage_secrets; otherwise the release waits on missing_keys). secret-materialize.sh below reads nodes/<n>/.cogni/secrets-catalog.yaml only inside the operator monorepo; it never sees a sovereign node's catalog. The call carries no secret value (deploy-a-declared-workload). The rest of this section describes the operator flight lane (operator only).
candidate-flight.yml uses scripts/ci/assert-target-substrate.sh as a
preflight for selected app rollouts. That gate may verify a Deployment-consumed
k8s Secret exists and its matching ExternalSecret is Ready, but it must not seed
OpenBao, patch GitHub secrets, run deploy-infra.sh, or repair Compose/env
state. A missing secret is a substrate failure. For a wizard-created ordinary
node, there should be zero per-node human secret values. The environment
must already have the substrate/runtime inputs classified in
https://node-template.cognidao.org/knowledge/secrets-routing-tiers.
If an environment-bank value is missing, repair that bank; do not pass values
through candidate-flight inputs or store them in the wizard.
"Read-only" applies to assert, not the whole flight. Distinct from the
read-only assertion, candidate-flight.yml / promote-and-deploy.yml run a
materialize-substrate job first — the sole OpenBao writer in the flight
(scripts/ci/secret-materialize.sh, <env>-writer token) — which generates the
node's source: agent secrets at cogni/<env>/<node>/* idempotently before
reconcile/assert, including the per-node DB creds (app_<node>/service_<node>
passwords) and the composed Postgres DSNs (DATABASE_URL/DATABASE_SERVICE_URL;
only DOLTGRES_URL is still deferred) since #1584 (canonical:
https://cognidao.org/knowledge/operator-pipeline-contract Axiom 22,
SUBSTRATE_IS_RECONCILED_BEFORE_PROMOTION). Since #1584 reconcile-substrate is
read-only on OpenBao (<env>-db-reader token, zero bao kv put/patch) and
assert-target-substrate.sh is read-only too — so only materialize-substrate
writes. bug.5007 status (corrected 2026-06-24): the openbao-operator
ServiceAccount that materialize-substrate uses to mint <env>-writer does
exist on prod (default ns, 21d), and operator-secrets-writer@cogni-production
exists too (6d) — the "SA absent on prod" framing is stale. The self-serve write
path is proven (operator returns 200). The original bug.5007 promote-pipeline e2e
(materialize-substrate green on a real prod promote) is the part still worth a direct
confirm; the SA/role prerequisite it blocked on is in place.
The target shape matters. Today the implemented branch is type=node; a future
type=service branch should assert the service's declared Secret /
ExternalSecret / ConfigMap contract without inheriting node DNS, Caddy, NodePort,
or node-DB assumptions.
Decision tree — how do I write / rotate the value?
| Operation | Right pattern | Today's reality |
|---|---|---|
| Add new secret SHAPE (service X consumes key A) | PR → vault-config-operator CRD → Argo reconciles | Not built; tracked in proj.agentic-fork-bootstrap Walk |
| Materialize a new wizard node's runtime secrets | Generate/derive node-local values and inherit only explicit org/env grants; no per-node human values for ordinary nodes | Shipped: materialize-substrate is its own flight job (scripts/ci/secret-materialize.sh, <env>-writer token), idempotent read-once→diff→write-missing (#1582/#1585; https://cognidao.org/knowledge/operator-pipeline-contract Axiom 22). The shared-bank / explicit inheritFrom-grant model is the follow-up |
Rotate AUTO-GENERATED value (e.g., AUTH_SECRET) | rotate-secret.yml workflow with env-protection; auto-generates value; human approves event, never sees value | Not built; do manual openssl rand + pnpm secrets:set per secrets-rotate.md |
| Rotate VENDOR-MINTED value (OpenAI key, Cherry token) | Operator API route POST /api/v1/nodes/[id]/secrets, OpenFGA can_manage_secrets — caller holds only an API key | Shipped + live on candidate-a AND production (operator returns 200). preview still 503 authz_unavailable (OpenFGA store not resolvable by the pod). For preview/cross-env: CLI + admin/kube custody. See "Who holds secret-write authority" above |
| Candidate-a experimentation | pnpm secrets:set <env> <service> <KEY> via port-forward + writer-role JWT | Shipped — see secrets-add-new.md |
| Dynamic DB credentials | OpenBao DB engine, no human in loop | Future (Crawl row 3 of proj.security-hardening) |
Rotation is not done when the write returns 200. "Written to OpenBao" and "the consumer holds it" are two facts with an ExternalSecret refreshInterval between them. A pod that reads its credential once at start is indistinguishable from healthy while holding a revoked one — proven live 2026-09-15: a rotated Akash key sat in OpenBao while the actuator ran 1/1 Running on a dead credential.
Rotate in four steps, never three:
write -> kubectl annotate externalsecret <name> force-sync=$(date +%s) --overwrite
-> Reloader restarts the consumer (already automatic, Invariant 11)
-> VERIFY the consumer (read it back; bug.5142 credential fingerprint)
Do NOT shorten refreshInterval to compensate. The class table in secrets-management.md is the sanctioned set (1h routine / 24h external API / 15m dynamic DB / 5m critical+wallet). 1m was proposed twice and reverted: 60x the OpenBao reads forever, against a Shamir 1-of-1 OpenBao that has already OOMKilled (bug.5011). And do not make the operator push the sync — it has no ServiceAccount and no k8s identity, so that means giving the internet-facing app RBAC to patch ExternalSecrets.
The killer rule: no human types a secret VALUE into a UI in production. Auto-generated, vendor-minted via operator-app, or dynamic. Form-input is the anti-pattern.
Who holds secret-write authority — TODAY vs TARGET (READ THIS)
The honest answer to "can a node add its own secret without Derek's GitHub/kube creds?" is somewhere in between, closer to Derek/GitHub-admin-controlled today. Do not claim "operator-app RBAC" where it isn't yet shipped. Three distinct authorities — keep them separate:
| Authority | What it decides | TODAY (custody) | TARGET |
|---|---|---|---|
| Secret declaration (shape) | "node Y is allowed key KEY_X" | GitHub repo merge authority — whoever can merge a PR to the catalog. Guarded by required CI + Cogni Git PR Review + branch protections. No value in the PR. | Same (PR-gated). This rung is fine. |
Generated value (source: agent) | random bytes minted for node Y | GitHub Actions materialize-substrate job: the flight workflow SSHes the env VM and mints an <env>-writer OpenBao token via the in-cluster openbao-operator SA. Real and shipped for candidate-a/preview; the openbao-operator SA also exists on prod (21d) — the old "prod lacks the writer SA (bug.5007)" line is stale. "Who can cause it" = whoever can merge/dispatch the env's deploy workflow → GitHub merge authority again. | In-cluster Job (SA-bound <env>-writer), zero SSH (design §North-star Stage 2). |
Human/vendor value (source: human: OpenAI key, Cherry token) | the byte value a vendor minted | Operator-controlled CLI/kube path (pnpm secrets:set + short-lived writer JWT via port-forward). In this workspace that effectively means Derek/admin-held repo + kube custody. | Operator app route + OpenFGA RBAC (see below) — prototyped, not merged. |
The gap that keeps us on Derek's creds: there is no shipped operator-app secrets route and no .github/workflows/secret-set.yml. Every value-write today bottoms out in GitHub-merge authority or operator/admin kube custody — i.e. Derek.
The precise PR/design that closes it — the value-write complement to the flight triangle (developer → can_flight → operator-held GitHub App creds), one rung over with its own least-privilege role (secrets_manager → can_manage_secrets → operator-held OpenBao writer; distinct from developer, mirroring production_promoter):
- Design:
https://cognidao.org/knowledge/node-secret-write-design(operator-mediated, OpenFGA-authorized). Sibling:node-wizard-secret-setting.md(shape, shipped). - PR: #1627 (prototype) —
POST /api/v1/nodes/[id]/secrets+OperatorSecretsPlanePort+OpenBaoSecretsAdapter(operator pod self-logins with its OWN projected SA token,audience: cogni-openbao) +can_manage_secretsOpenFGA relation + build-time A2 allowlist. Caller holds only an API key — no kubeconfig, no vault token. - Per-env readiness is live state — recall the hub guide
node-self-serve-secrets, don't trust a git snapshot. The provisioning prereqs + the503 authz_unavailableOpenFGA-model-drift class are above; the durable cross-env fix istask.5049.
Three defense-in-depth gates in #1627: OpenFGA can_manage_secrets (fail-closed) → catalog A2 allowlist (build-time typed module) → OpenBao policy explicit-deny on _system/_shared.
Validation status + the road to a proper e2e: https://cognidao.org/knowledge/node-secret-write-design — the layered proof (L1 candidate-a authz done; L2 full write needs the allowlist codegen + the OpenBao writer-role DRY fix; L3 human-in-loop deployed e2e). Load-bearing gotcha: the write env is operator-stamped (deployEnv = env.DEPLOY_ENVIRONMENT), so cogni/<operator-env>/<node>/* is the only path it can write — a prod operator cannot write a candidate-a node's secret. Any cross-env "set my candidate-a secret from the prod operator" needs an explicit authz-scoped targetEnv param (not in #1627).
Placement is not provisioning — the empty-node-path class (READ THIS)
Symptom: a node deploys fine in one environment and its ExternalSecret fails in another with
could not get secret data from provider / cannot find secret data for key: "AUTH_SECRET" — failing at
spec.data[0], i.e. the node's OpenBao path is empty, not one key short.
⚠️ FIXED — rule 1 below shipped. Do not diagnose a live node from the Historical cause. An Akash-placed node is materialized today; see "As-built" before concluding anything. This section is kept for the invariant and the failure signature, not as a description of current behaviour.
Historical cause (superseded): the node was moved to external compute
(deployment_provider.<env>: akash) and thereby dropped out of the job that provisions it.
node-substrate — which runs secret-materialize — was matrixed on k3s_node_targets_json, so flipping a
node's provider silently removed it from the population that got served. Nothing errored; nobody created
its secrets.
As-built (verify before you trust either version): the population was split, not narrowed. Both lanes
run the same script — scripts/ci/run-node-substrate.sh, whose header calls itself "the ONE per-node
substrate runner" and whose case "$DEPLOYMENT_PROVIDER" accepts k3s|akash. It calls
secret-materialize.sh for either provider.
| lane | job / writer | matrix |
|---|---|---|
| k3s | node-substrate | k3s_node_targets_json |
| Akash | node-deploy sequencer (POST /deploy/workloads) | none (no flight job, story.5137) |
The trap that outlived the bug: the JOB node-substrate and the SCRIPT run-node-substrate.sh share a
name but serve different populations. Tracing the job alone shows a k3s-only matrix and looks like proof
that Akash nodes are never materialized — a wrong, confident conclusion. The disconfirming evidence is
cheap: check which rows declare deployment_provider.candidate-a: akash and carry complete
buckets including LITELLM_VIRTUAL_KEY. If Akash nodes were never materialized, those could not exist.
Grep for every run-node-substrate.sh call site, not just the obvious job — there are two.
Why the original defect hid: the first external nodes were k3s nodes first. They kept the secrets they already had, so test and preview looked healthy while inheriting from a k3s past. The defect only surfaced in an environment where that node never ran on k3s. Do not treat a green lower environment as evidence.
Two rules:
- Branch the steps, never the population. Every
type: nodetarget gets materialized regardless of provider; only provider-specific work (k3s Caddy/NodePort edge reconcile vs external DNS reconcile) is selected inside the job. The typed planner already emits the correct wider node list — prefer it over the k3s-only one. - A provider that needs a different secret needs a catalog entry, not a rename. External compute must
not receive fleet-shared credentials (they would reach a rented third-party host), so it consumes a
node-scoped equivalent — e.g.
LITELLM_VIRTUAL_KEYin place of the sharedLITELLM_MASTER_KEY. That is correct, but it is only real once the catalog declares it with a generator. A key that has a consumer and no declaration gets hand-seeded once and silently carried forever (CATALOG_IS_THE_ONE_READER).
Fast triage: read key NAMES only, never values —
bao kv get -format=json cogni/<env>/<node> and compare the environment that works against the one that
does not. If the failing path is empty rather than short, this is your class.
Spec: https://cognidao.org/knowledge/secrets-management invariants 17-18; https://cognidao.org/knowledge/node-baas-substrate-model
§ Placement vs Provisioning.
Dual-plane secrets — the silent-webhook-fail class (READ THIS)
Some secrets must byte-equal a value held by an external system, not merely exist. The operator's GH_WEBHOOK_SECRET must equal the GitHub App's webhook secret; an OAuth *_CLIENT_SECRET must equal the provider's app config. These are dual-plane: one copy in our pod, one on the external plane — they only work if identical.
The trap (live bug, preview 2026-06-03): GH_WEBHOOK_SECRET was source: agent with no syncTo, generated randHex 32 every provision → it could never match the App's webhook secret → every webhook failed HMAC verification, silently (a level:40 warn component:webhook-route msg:"webhook verification failed", no alert, no 5xx). The App just looks dead — no PR reviews, no node-wizard. deploy-infra re-applying the Secret was the breaking path, not the healing one — a generated value never equals an externally-held one unless something pushes it there. That push is what syncTo declares.
syncTo is an external mirror, not custody. origin says who can produce
the bytes, custody says where the value is authoritative, and syncTo says
whether a copy must also be pushed to an external system.
| axis | values | answers |
|---|---|---|
source: | agent (we generate, generate: required) · human (un-generatable; vendor-minted, human supplies once, e.g. GH_REVIEW_APP_PRIVATE_KEY_BASE64) | where the value originates |
syncTo: (opt) | github-app-webhook · (unset) | does it also live in an external system we must keep in lockstep |
GH_WEBHOOK_SECRET is source: agent + syncTo: github-app-webhook — we generate it (origin is internal; calling it source: external lied about that), and it ALSO must byte-match the App. The materializer/provisioner generates it into OpenBao; the infra lane then pushes it (scripts/secrets/sync-app-webhook-secret.sh: App JWT → PATCH /app/hook/config):
APPID + GH_REVIEW_APP_PRIVATE_KEY_BASE64 → RS256 JWT (iss=APPID, exp≤10m)
PATCH /app/hook/config -d '{"secret":"<generated GH_WEBHOOK_SECRET>"}' # endpoint EXISTS
No-human-secret done right: agent generates, agent pushes, zero human, self-healing (provisioning owns both copies → every infra-lever deploy re-converges). Do NOT make it source: human/carry — that drags a human into the App's "Change secret" field for a value that's ours to generate.
⚠️ Sync only fires on the infra lever (deploy-infra via candidate-flight-infra / provision-env), NOT on app-lever promotes (candidate-flight/flight-preview/promote-and-deploy = Argo image bump, never touches the Secret). assert-target-substrate.sh is also read-only: it can fail a flight when the Deployment-consumed Secret / ExternalSecret is absent or not Ready, but it does not heal the value. In an established ESO environment, deploy-infra must source GH_WEBHOOK_SECRET from OpenBao before invoking the sync and must fail closed if the App PATCH fails; otherwise the pod plane and GitHub App plane can silently diverge again. Fresh/plain-Secret bootstrap may still use workflow env input until the operator ESO target exists.
Heal-proof test = redeploy twice; a PR on the test repo must still post a cogni-git-review review.
OpenBao availability — sealed = DOWN, not secure (READ THIS)
Counterintuitive but load-bearing: a sealed OpenBao serves nothing — it 503s every ESO sync, every materialize/reconcile-substrate auth/kubernetes/login, and openfga's config load. OpenBao must run UNSEALED ~100% of the time; unsealed-and-running is the correct operational state. The seal protects data at rest (stolen storage = encrypted/useless), not runtime access — that's auth methods + path policy (Invariant 6) + in-cluster-only (no Ingress). So "keep it sealed / minimize unsealed time" is backwards: an unexpected seal is an outage, not a safe state.
The fragility (bug.5011, prod outage 2026-06-11 02:25): prod OpenBao is Shamir 1-of-1, no auto-unseal. It OOMKilled on the 6 GB box → resealed → secret plane down (openfga 503, node-substrate promote failures) until a human kubectl exec … bao operator unseal. Any restart (OOM, reboot, chart upgrade) reseals it.
Recorded decision (bug.5011) — do not re-litigate without the trigger:
- Memory headroom — OpenBao limit 512Mi→1Gi (#1617) so it isn't container-OOMKilled; #1616 lean-prod cut node pressure too. Mitigates the OOM trigger only.
- Sealed-state → Loki alert — the no-silent-outage control: detect an unexpected seal, page, unseal in minutes. Free, no vendor. This is the durable fix, because you cannot prevent every reseal.
- Auto-unseal (KMS) DEFERRED behind a written trigger: reseals recur post-#1617, OR a SOC2 engagement starts. KMS is the audit-correct control but it is our first AWS/GCP dependency on a Cherry+OSS stack — premature at 0 users. If a real SOC2 deadline lands, flip it to now.
- REJECTED: in-cluster stored-key unseal sidecar — key-next-to-lock, fails SOC2.
Anti-patterns — instant reject
- Human typing a secret VALUE into a production UI or GitHub workflow input. See killer rule.
- Treating
tier: B, GitHub Environment Secrets, or VM.envas authority for a runtime secret. If it feeds a pod or a pod-facing role, OpenBao owns it. - A flight or deploy call accepting secret values as inputs. The operator flight may invoke materialization for safe generated/derived node values and fail if the environment's required DAO/org bank is missing; it must not carry values.
- A dual-plane secret (must byte-match an external system — GitHub App webhook secret, OAuth client secret) declared with no
syncTo:— it silently fails verification forever anddeploy-infrare-breaks it every run. AddsyncTo:(keepsource: agentif we generate the value). See "Dual-plane secrets" above. - Generic catch-all workflow (
secrets-manage.yml-shaped). Per-operation only. ssh root@vm kubectl ...orssh root@vm bao .... Use local kubectl + port-forward + writer-role JWT.- Treating k8s Secret or ConfigMap presence as proof that a running pod has the value. Prove the process after rollout.
- Treating a failed app-flight substrate assertion as permission to run
deploy-infrafrom the app flight. Heal secrets/substrate through the owning secrets or infra lane; keep app flight read-only until image promotion. - Re-exporting
.local/<env>-openbao-root-tokenafter Phase 5b — violates Invariant 13. bao kv putinstead ofbao kv patch(replaces sibling keys).bao login -method=kubernetesin OpenBao CLI 2.5.x — that subcommand doesn't exist; use raw API:bao write auth/kubernetes/login role=X jwt=Y.- Per-secret ExternalSecret YAML — violates Invariant 2.
valueFrom: secretKeyRefper env var in pod spec — violates Invariant 3.- Base64-in-git "encryption" — violates Invariant 4.
- Sealed Secrets / SOPS+ksops — explicitly rejected per
proj.security-hardeningDesign Notes. - Editing
scripts/setup-secrets.tsto add/remove a SECRETS array entry — there isn't one. The script loads YAML viascripts/lib/secrets-catalog-loader.ts. Edit the appropriatesecrets-catalog.yamlinstead. - ExternalSecret manifests under
infra/k8s/secrets/external-secrets/<env>/<node>/— that pre-wizard tree is PURGED. Remote-source/wizard nodes carry their leaf in the node repo atk8s/external-secrets/<env>/, mounted by the operator asnodes/<node>/k8s/external-secrets/<env>/when pinned. Only the cluster-scopedClusterSecretStoreremains under the old dir. - Treat the operator overlay-local ExternalSecret as a legacy operator exception, not the node standard.
infra/k8s/overlays/<env>/operator/external-secret.yamlis Argo-owned because the operator Deployment patch lives in that overlay; source nameenv-secretsrenders throughnamePrefix: operator-to objectoperator-env-secrets, whilespec.target.namestaysoperator-env-secrets.
The catalog (per-node YAML + Zod loader)
scripts/setup-secrets.ts does NOT hold a hardcoded SECRETS array. It calls a Zod-validated loader that walks YAML catalogs:
| File | Domain | Holds |
|---|---|---|
infra/secrets-catalog.yaml | operator-domain | _shared, _system, B/D/E/G entries, A2 placeholders, and (today) all node A1/A2 entries — node-template's catalog was migrated here by task.5094 |
nodes/<node>/.cogni/secrets-catalog.yaml | node-domain (single-node-scope: a node engineer adds their node's secret in ONE PR) | A1/A2 entries the node owns; service: auto-fills from parent dir. Loader-supported but currently empty — no node populates it yet (post-task.5094 consolidation) |
scripts/lib/secrets-catalog-loader.ts | operator-domain (substrate) | Walker (walks both paths above) + uniqueness + service-allowlist assertions; the entry Zod schema lives in the published @cogni-dao/node-declarations package, re-exported by @cogni/repo-spec (shared with the node-deploy verb, bug.5254) |
To add a node secret today: in the operator monorepo, edit infra/secrets-catalog.yaml — node entries are consolidated there post-task.5094. In a sovereign node repo on the deploy verb, add one entry to the node's own .cogni/secrets-catalog.yaml with workloads: [<repo-spec service>]; that is the whole declaration (no repo-spec edit).
To add an operator-domain secret (B/D/E/G, or _shared cross-cutting): edit infra/secrets-catalog.yaml. Operator-domain PR.
The loader rejects at module load: missing tier, name collision across files, per-node service: mismatch with parent dir, unknown service: value not in the allowlist (_shared/_system + present nodes + canonical-future-domain names from https://node-template.cognidao.org/knowledge/node-ci-sovereignty).
Files to read by topic
| If you're doing… | Read |
|---|---|
| Understanding the tier system + invariants | secrets-routing-tiers |
| Designing node-wizard secret behavior | https://cognidao.org/knowledge/node-secret-materialize-phases |
| The layered authority model (Identity → DAO) | operator-identity-model |
| Adding a new secret to your node | Edit nodes/<node>/.cogni/secrets-catalog.yaml + read https://cognidao.org/knowledge/break-glass-secret-write |
Adding a _shared / B / D / E / G secret | Edit infra/secrets-catalog.yaml |
| Rotating an existing secret | https://cognidao.org/knowledge/break-glass-secret-write |
| Following the bootstrap flow | https://cognidao.org/knowledge/fork-quickstart-guide |
| Provisioning a new env (candidate-*, fork) | Dispatch .github/workflows/provision-env.yml (-f env=<env> -f encryption_passphrase=…). Full walkthrough — 7 minting secrets + init-artifact custody — in https://cognidao.org/knowledge/fork-quickstart-guide §6. The runner owns the tofu+bao+kubectl session; the operator's laptop never holds substrate creds. |
| Adding a new service (new k8s Deployment) | https://cognidao.org/knowledge/run-node-formation-wizard + add ExternalSecret under nodes/<node>/k8s/external-secrets/<env>/ |
| Touching substrate provisioning | scripts/setup/provision-env-vm.sh Phases 5b.1–5b.5 |
| Touching the CLI | scripts/secrets/set-secret.sh + test scripts/ci/tests/set-secret.test.sh |
| Touching the loader / catalog schema | @cogni-dao/node-declarations (entry Zod schema + sovereign-node check; Release pin in packages/repo-spec/package.json) and scripts/lib/secrets-catalog-loader.ts (walker) |
| Touching any node's ExternalSecret | Remote-source/wizard nodes: nodes/<node>/k8s/external-secrets/<env>/ (per-node, node-domain) — the single repo-wide convention. Operator legacy overlay: keep infra/k8s/overlays/<env>/operator/external-secret.yaml aligned until operator moves to the same node-domain leaf. No aggregator; only the cluster-scoped ClusterSecretStore lives under infra/k8s/secrets/external-secrets/. |
| Touching substrate Argo Applications | infra/k8s/argocd/{openbao,external-secrets}-application.yaml |
| Touching the env-var classification routing | .claude/commands/env-update.md — k8s app vs Compose-infra split |
| Granting Grafana/Loki query access (dev, validator, or a gate) | node-log-proxy-contract — one admin root → scoped consumers. Dev node-read = operator query-PROXY pinned to {node="<id>"}; the dev holds NO token (a returned token's reach escapes the per-node OpenFGA check → dormant env-wide leak; never hand a dev a Grafana token). Real blocker: node Loki streams have no node label yet (app/env/service only). assertLive gates on the PUBLIC run-carries rung; the operator's gate holds no query token. Out of MVP scope: label-scoped glc_ / per-dev SA minting. Cross-substrate plane + health scorecard: read-fleet-health. |
| Designing a new workflow that handles secrets | This file + proj.agentic-fork-bootstrap anti-patterns. Run it past the killer rule. |
When to escalate
Surface to operator before writing code if:
- Adding a NEW entry point that isn't already CLI / workflow_dispatch / operator-MCP — Invariant 9 lists the only three sanctioned shapes.
- Changing
eso-readerpolicy or<env>-writerrole binding — affects every consumer. - Bumping OpenBao or ESO chart version — rotation drill required (see
secrets-rotate.md§Upgrade discipline). - Anything that smells like Invariant 4 (NO_VALUE_IN_GIT) — finding a value in YAML / commit message / PR diff / chat is always a rotate-now event.
- Designing a workflow where humans type values into a form — recheck against the killer rule before building.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |