Skip to content

phuryn/product-engineering / app-stack/skills/environments

environments

Verified

Adding a variable or secret, 'set up production', a domain or a provider token, 'which database is this', a leaked key: dev and production for an AI-built app, backups, incidents.

references/variables.md

View on GitHub
# Variables and secrets

How configuration enters an app, the contract that keeps a missing, misplaced or half-set value from reaching a build, where every copy of a secret ends up, and how to change one without breaking whatever still holds the old value. Learned on Next.js, where `NEXT_PUBLIC_*` is the only prefix that reaches the browser and is compiled into the bundle at build; on another framework, find its equivalent public prefix and the moment it is inlined — the rules follow from that moment.

## Every new variable, in the same change

1. **A placeholder in `.env.example`.**
2. **A row in the variables doc** (columns below).
3. **A line in the env schema** that fails when it is missing (or says it is optional, below).

**A value you generate rather than fetch is generated now** — written into `.env` and named in your summary. Nobody can look it up, so a placeholder left empty is a boot failure with no instruction attached to it. For production, generate it afresh into the production operator file outside the development checkout, never any `.env*` file in it, single-quoted, since the shell reads that file; never copy development's.

## The variables doc

One row per variable, added when it is added to `.env.example`. The `shipping-artifacts` skill defines the doc for a reviewer; these columns are a good default:

| Name | Used by | Scope | Source | Rotation | Risk if leaked |
|---|---|---|---|---|---|

- **Scope** is `server`, `client`, `test`, `build/test` or `both`. **No secret is ever client-side**: only `NEXT_PUBLIC_*` reaches the browser, so every `NEXT_PUBLIC_*` row must be safe to be public.
- **Source** says where a person gets it (the provider's dashboard, by path), or *generated by the agent* with the command, or *harness*.
- **Rotation** says what rotating it costs, not only where — "every visitor becomes a new visitor", "all tokens invalid".
- **Risk if leaked** is ranked: none (public by design) · low · medium · high · **critical**.

## The kinds of variable, and how each is handled

Grouped by kind, with a few names as examples. The pattern, not the name, is what carries to another product.

- **The app's own public URL** (`APP_URL`): absolute links, the Origin check, the `secure` flag on cookies, and the base for `og:image`, `robots.txt` and `sitemap.xml`. **Set it on the deploy**: unset, those fall back to the host's own URL and then to localhost, and a static page bakes the wrong value into its metadata at build time. **Every URL the product hands out is lasting public configuration**: a link in a sent email, a badge pasted into someone's README or an embed keeps the host it was made with, so production's value is permanent once used, and a temporary address (a tunnel) never goes into a deployed runtime's link base.
- **Generated secrets** — a cookie-signing secret for the anonymous public (`PARTICIPANT_COOKIE_SECRET`), an HMAC pepper for API tokens (`API_TOKEN_PEPPER`), an origin-seal secret: generated by the agent, for example `node -e "console.log(require('crypto').randomBytes(48).toString('base64url'))"`. Their rotation is a product event: rotating the cookie secret makes every visitor a new visitor (their rows are orphaned, not lost); rotating the pepper invalidates every token. Risk: high. **A secret for an architecture not adopted stays empty** (the origin-seal secret, unused while there is no proxy in front of the host), with the row saying so.
- **Keys that cannot be replaced without breaking stored data**: an encryption key for credentials the product stores, a token pepper, a key that tags issued URLs, a link-signing secret. Losing the encryption key makes every stored credential unreadable, and there is no rotation without a re-encryption pass. So list them in the variables doc as irreplaceable, **keep a copy off the operator's machine**, and mark them secret on the host. (`2026-10-24`, a monitoring product's inventory found three such keys stored as *plain* variables on its production site, and every other copy on one laptop.)
- **The database URL and its secret key** (`SUPABASE_URL`, `SUPABASE_SECRET_KEY`): server only; the secret key bypasses row-level security, so its risk is **critical — the full database**. A second runtime gets its own database login and URL, never this key (`supabase-production`).
- **The identity provider's keys**: a publishable key needed at **build** (`NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY`, low, public by design) and a secret key (`CLERK_SECRET_KEY`, **critical — impersonation**). Two keys of different risk side by side on one dashboard page, pasted by hand: the schema refuses a secret-shaped value in the public variable (below, and `clerk-production`).
- **A webhook signing secret** (`CLERK_WEBHOOK_SECRET`): high — forged events (for deletion webhooks, forged deletions). Stays empty, with no webhook route, until the capability that needs it arrives.
- **Optional feature switches** — an analytics project id, a legal contact address: **empty means the feature is off**, not broken. An empty analytics id means no script loads at all; an empty contact address means the legal pages show only the postal address, which is better than a mailbox that does not exist, except where the legal pages must give an email address or a telephone number before the front door is shared (`legal-pages`, rule 9). Turning one on can carry a dashboard obligation beside it (cookieless analytics also needs cookies switched off in the analytics project, because the privacy page promises both halves).
- **Development-only switches** — a simulated paid plan, a redirect that sends every email to the provider's discard address: gated on something only the dev server sets (`NODE_ENV=development` from `next dev`) **and** their own flag, so neither a deploy nor a test runner can turn them on; pinned by a unit test in both directions; and visible on screen while on. **The production build and the production deploy of every runtime refuse a non-empty value**: a convention alone lets a copied `.env` send every customer's email to one address. (`2026-10-23`, a monitoring product's backlog: nothing refused its email redirect in a production build or a Worker's production deploy, and the pre-go-live box for it was still unchecked on `2026-10-24`.) **A local eval switch that must work under a production build is a distinct case:** `next start` sets `NODE_ENV=production`, so the env schema refuses that switch unless the app's own URL is loopback (`localhost`, `127.0.0.1`, `[::1]`), with a unit test that accepts loopback and refuses a deployed URL under `NODE_ENV=production`. (`2026-11-07`, a live Q&A product: its routing evals needed a production build, and its adoption pass found an any-caller flag guarded only by a comment and a docs row.)
- **A pair** — an email provider's key plus the sender address: **both set or both empty**, and the schema refuses a half-configured one.
- **A model provider's key** (`ANTHROPIC_API_KEY`), **only when the product embeds agents**: commented out in `.env.example`, and only such a product's schema has a line for it, so a product without agents never needs it. Absent on a developer's laptop, the local SDKs may fall back to the developer's own login; **a deploy always sets it**. Risk: high — model spend on the account, up to the provider's spend limit.
- **Test-only values** — a publishable database key used only by the grant-denial tests, the test account's address, the identity provider's fixed verification code: **not in the runtime schema**; their harness checks them (`test-harness`). A test-only key that is absent makes its suite skip loudly; configured and rejected, it fails.
- **Harness values** — the browser suites' port and build directory: set by the harness, risk none.
- **Operator values** — a provider's API token, a database admin connection string, an SSH key path: read only by the operator's scripts, never by a runtime, and **one per environment** (SKILL.md, separate key sets).

## The env schema's contract

Runtime configuration is validated in one schema module (zod, for example); a required value missing is a **refusal to build and to boot**.

- **It runs from the build config (`next.config.ts`) at build and at dev start, not only on the first request.** A `NEXT_PUBLIC_*` value is inlined into the client bundle when the build runs, so a check at boot comes after the leak — and static chunks are served without any request-time check at all.
- **Keep the schema module pure** (no `server-only`) so the build config and the unit suite can import it, with a server-only module wrapping it for the app.
- **An empty string counts as unset.** A `KEY=` line copied from `.env.example` otherwise passes a plain string check, so drop empty values before parsing rather than allowing `""` field by field.
- **A failure lists every problem in one message and never echoes a value.**
- **Each key's shape is checked where the vendor gives one** (on Clerk `pk_(test|live)_` and `sk_(test|live)_`, on Supabase `sb_secret_`), and **every `NEXT_PUBLIC_*` key refuses a secret-shaped value.** A unit test pins the refusal, and that the message does not contain the secret. (`2026-10-10`, a product's `.env` held the Clerk secret key in the publishable variable; caught before anything was built with it.)
- **A production build refuses every development-only value** (§ The kinds of variable), and a local eval switch unless the app's own URL is loopback.
- **A host that scopes secrets away from the build** (Netlify's variable scopes) still gets the `NEXT_PUBLIC_*` half and whatever the build itself reads validated at build; the server-only remainder is checked at boot.
- **Required and optional are different things, and the schema says which.** An optional value absent turns a feature off rather than breaking anything, and the unit suite pins that in both directions.
- **Test-only and build-only values are checked by their harness or script, never by the runtime schema.**

**If a build or a dev server ever ran with a secret in a public variable, that secret was served** — inlined into every client chunk, or handed out by `next dev`. Rotate it (§ Rotating).

## Setting values on the host

- **The host does not read `.env`.** On Netlify every variable the deploy needs is set in the site's own environment, and `NEXT_PUBLIC_*` values must be there **at build time** (`release-gate`, `references/netlify.md`).
- **Any variable change takes effect only on a new deploy**, server-only ones included, and a `NEXT_PUBLIC_*` or other build-time value only on a rebuild, because it was compiled in (`release-gate`, rollback).
- **Mark every secret as secret.** On Netlify, a plain variable can be read back by anyone with access to the site or an account token; one marked *Contains secret values* is write-only once saved (observed `2026-10-24`). A write-only value means the operator's file may be the only readable copy (§ Where every copy lives).
- **A secret never goes on a command line.** Pass it on stdin (`wrangler secret put`, `wrangler secret bulk`) or name the environment variable that holds it; an argument lands in shell history and process listings. Where a tool takes the secret only as an argument (the database CLI's `--db-url`), expand it from the operator file's variable (`--db-url "$DEV_DB_URL"`), so the value never reaches history or a transcript, and run it only where nobody else can read the process list, which shows it while the command runs: the operator's own machine, or, for a development URL only, a CI job's own runner with command echo off; never a shared host. Production's URL stays on the operator's machine (`supabase-production`). A password a session printed is rotated the same day. (`2026-10-23` and `2026-10-24`, a monitoring product: two development login passwords printed by a session and by an operator command, both rotated; a command that put a password on the command line was replaced.)
- **Provider work done by an agent runs on a token the owner issues for it**: scoped to one environment where the vendor allows that, and recorded; otherwise short-lived and removed afterwards (what to ask each provider for: `domain-and-provider-access.md`), and any scratch dump that held values is deleted in the same session. (`2026-10-24`, a monitoring product: a seven-day host token, removed when the work was done; the session's dumps of both sites' variables deleted.)
- **At go-live the product branch's deploy gets development values**, or it writes to production (`environments-and-branches.md § Branches`).

## Where every copy lives

**Every holder keeps its own copy.** The app's host, each Worker, each box's root-only environment file and a pooling proxy's configuration each hold one, and most of them are write-only. Changing a value in one place changes nothing anywhere else until that holder is set and redeployed. Once the product runs more than its host builds, the variables doc carries a table with one row per secret:

| Secret | Made by | Copies in development | Copies in production | Read at runtime by | If it is lost |
|---|---|---|---|---|---|

- **"Copies" includes the operator's files**: the `.env` in each operator checkout, any record of production's generated values, and each builder worktree's copy. Read them by name, compare values in memory, and never print one.
- **"If it is lost"** says whether the vendor can simply issue another, or whether stored data breaks (§ The kinds of variable).
- **Also list the copies that shouldn't exist** (stale worktrees, `.env.bak`), and the copies that are missing (a value that exists only on the host and at the vendor, so a rebuilt site needs a new one).
- **A deploy that uploads secrets from the operator's `.env` puts that file's value back.** After a rotation, the operator file changes *before* the next deploy, or the deploy silently undoes the rotation.

(`2026-10-24`, a monitoring product: the table it wrote listed about two dozen secrets spread over its host, its Workers, its boxes and three operator files, with production's operator files all on one laptop.)

## Rotating

**A planned rotation revokes last:**

1. **Make the new value beside the old one**, where the vendor allows two at once (Supabase's and Clerk's secret keys do; a chat bot's token revoked at the vendor dies at once, so have every holder ready before you revoke it).
2. **Put it in every holder** in the copies table, operator files included, **and deploy each one.** A push redeploys only the app (`release-gate`).
3. **Check it works** through a path that uses each holder (a real email, a real ping), not only the app.
4. **Then revoke the old value.** Revoking first breaks whatever still holds it; a holder you missed fails silently, later.

Write each rotation in the environment log with the date and what changed.

(`2026-10-24`, a monitoring product rotated its identity provider's production secret on the app's host only. Its alert-sending Worker kept the old key, which still worked, and the production operator files most likely held it too, so a redeploy would have uploaded it again. Deleting the old key would have stopped members' alert email with nothing to notice it.)

- **A key that seals or tags stored data cannot be rotated by this order alone**: it needs a pass that re-encrypts or reissues what it protects, so it is rotated only after a leak, as a planned product event.
- **A link-signing secret breaks every link already sent**, an email's unsubscribe link included, so it is rotated only after a leak.
- **A leaked secret is the same order, done at once**; if it is being used against you, revoke first and accept that every holder fails until it has the new value (`data-and-incidents.md § A leaked key`).

## Pre-go-live checklist

- [ ] The production identity instance and the production database project have their own keys; none shared with development, operator credentials included.
- [ ] Every secret that a local secrets file (`.env`, `.dev.vars` and their usual copies) ever set in history is rotated, and none of those files is tracked now but a template that holds no secret, such as `.env.example`. Make the clone whole first: `git rev-parse --is-shallow-repository` prints `false` (otherwise run `git fetch --unshallow`), and once the code is on a code host, `git fetch --tags origin '+refs/heads/*:refs/remotes/origin/*' '+refs/pull/*/head:refs/remotes/origin/pull/*'` brings every branch and tag, and on GitHub every pull request; on GitLab, adding `'+refs/merge-requests/*/head:refs/remotes/origin/merge-requests/*'` brings every open merge request and each one closed or merged in the last 14 days (GitLab's docs, read on `2026-10-29`). In that clone, `git --no-replace-objects log --exclude=refs/stash --all --full-history --diff-merges=separate --oneline --name-only -- ':(top,glob)**/.env*' ':(top,glob)**/*.env' ':(top,glob)**/.env*/**' ':(top,glob)**/.dev.vars*'` (git 2.31 or later) lists every such file a branch, tag or pull request reaches, templates included. It can't list a commit no ref reaches any more, which the host may still serve by its id (a branch force-pushed over or deleted, a tag moved or deleted, a GitLab merge request past its 14 days, a commit pushed to a fork, a fork since deleted): once the code has been on a code host, rotate every secret still live that such a file could have set, operator and test-only values included, listed or not. A development or test key is a secret; a placeholder, or a value public by design, is not; judge each without printing it. Rotate as for a leaked key (`data-and-incidents.md § A leaked key`), a password inside a connection string included; a key that seals or tags stored data is rotated together with its re-encryption or reissue pass (§ Rotating). Never rewrite history to hide them: clones and forks keep the commits.
- [ ] Every `NEXT_PUBLIC_*` value is safe to be public.
- [ ] Every variable the deploy needs is set in the host's production environment, the app's public URL included, and every secret is marked secret; generated secrets generated afresh for production. At go-live they go in only after the product branch is pinned to development values, and a fresh production deploy follows (`release-gate`, go-live step 3).
- [ ] No development-only value is set on any production holder, and the production build refuses one; a local eval switch is refused unless the app's own URL is loopback.
- [ ] The irreplaceable keys have a copy off the operator's machine.
- [ ] `.env.example` and the variables doc list every variable the code reads (also a release precondition — `release-gate`).
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall