Skip to content

phuryn/product-engineering / ai-shipping/skills/test-harness

test-harness

Verified

Writing or running tests, proving a fix, a red or flaky run, a fix that didn't land, 'is it verified', 'run the gate': tests where green means something.

FileKindSize
SKILL.mdskill md18.6 KB
receipts.mdmarkdown7.1 KB
references/captures.mdmarkdown7.2 KB
references/coverage-map.mdmarkdown5.4 KB
references/external-creates.mdmarkdown1.6 KB
references/harnesses.mdmarkdown15.6 KB
references/rules-from-bugs.mdmarkdown29.3 KB
references/vendor-surfaces.mdmarkdown6.8 KB

references/harnesses.md

View on GitHub
# The harnesses, the script contract and the authenticated mile

How the suites are arranged, as built on Next.js (App Router), Supabase Postgres and Clerk with Vitest and Playwright. The folder and script names are a suggested layout; call yours what you like.

## Every harness

Before adding a harness, write down what it fakes. If everything else already fakes the same side, it mostly re-covers ground.

| Harness | Folder | Real | Fake | Runs |
|---|---|---|---|---|
| **Unit** | `tests/unit/` | pure modules: authorization functions, schemas, token hashing, limits, escaping, the env schema, the token and kit guards (source tests that refuse a pattern in the tree), the edge-rule numbers | database, identity, browser | `gate` |
| **Integration** | `tests/integration/` | the development Postgres: the applied schema, RLS deny-all, cross-tenant, invariants and races through the real SQL functions. **Not a migration-from-zero test** unless one is added and named | identity (a mock verifier behind the same seam as the identity provider) | `gate` |
| **Schema catalog** | `tests/integration/` | every public relation and function, enumerated from the catalog so future objects are covered: RLS on, zero policies, grants, invoker views, and no `security definer` function but the pinned signatures: those a second runtime's login calls, and a trigger that writes on a restricted writer's behalf (the posture itself is `supabase-production`) | no identity, no fixtures. One read-only catalog query against the project the app's own database URL names, by one of two routes: a fixed metadata RPC through the secret key — itself a function the catalog must clear — or the database CLI's query command with the target in a connection string (`npx supabase db query --db-url "$DEV_DB_URL" --file <query.sql> --output-format json`, the URL expanded from the operator file, never typed: `environments`, `references/variables.md`), which adds nothing to `public` and holds a credential for that one database. Never through the CLI's link (`--linked`): the CLI is never linked, because which of a link and a typed `--project-ref` wins is not something to rely on (`rules-from-bugs.md § The database`), so a developer's machine and hosted CI take the same route (`supabase-production`, `references/access-model.md`) | `gate` |
| **Browser, public** | `tests/e2e/` | Chromium, phone profile, the development database; the anonymous journey needs no account | member identity; fixtures need development credentials | `gate` |
| **Browser, authenticated** | `tests/e2e/*.auth.spec.ts` | the identity provider's real development instance and a real session: sign-in through the provider's own sign-in component, the member's own screens, an account per role (in one organization, in the organization shape), the anonymous public beside them | mail (a test address and a fixed code); any app email transport | **local** `gate` |
| **Visual** | `tests/visual/` | a bounded release set of captures, each row naming one width (360×800 / 390 / 1280 / 1920) and one theme; and, on request, a sweep that walks every lifecycle state as every role at every width and theme. **320 is an assertion width, never a capture width** (`ui-design`, `references/screens.md`). Images are written **beside** the runner's `outputDir`, never inside it | identity where a capture does not sign in | `npm run visual` — outside the gate on purpose: at the end of any phase that changes a screen or a fixture, and the release set before a release; the full sweep on request, never a release step (`captures.md`) |
| **Live agents** — only when the product embeds agents | `tests/live/` | the real model: each agent's instructions and request-scoped tools through the Claude API tool runner (`embedded-agents-messages-api`) | — | `npm run agents:live` — outside the gate on purpose, because it costs money; before any release that changes an agent, its prompt or its model (`release-gate`) |
| **Smoke** | `tests/smoke/` | a deployed URL, write-free: `/health`, then each edge rule tripped with `POST`s that write nothing | — | after every deploy, **last** — it refuses your own IP for a minute (`release-gate`, `references/netlify.md`) |

## When the product has two halves

Learned on a remote-control product: a relay service with a web client, and an editor extension (and desktop app) that hosts a CLI agent. The same question, with more columns: for each harness, say which of **backend, client, host and the vendor tool it drives** is real, which is a stub, and which is absent.

- **A defect that needs both real halves is invisible when every suite fakes the other half — and one that needs a restart is doubly so.** Every backend suite drove a real browser against a scripted stub host (instant, always warm, never refuses, never dies); every extension suite drove a real host against a stubbed client (no outbox, no tab memory, no reconnect); and every harness booted once into steady state and stayed there. A race in restoring a session shipped through all of them. **Build one harness where both halves are real and a process can be killed in the middle of a journey**: new session → send → second session → refresh → host restart → reconnect. On the day it first ran it found two more of the same family that no other suite could reach — a message sent while the host restarted was silently dropped by the client, and the host silently abandoned a send when a concurrent session start overtook it. It runs locally only (it needs the sibling checkout), so the local gate before a promotion is where it runs; **record both repositories' commits, and any uncommitted changes in the sibling tree, with its result**, because it exercises that working tree as it stands.
- **Two DOM suites, one on each side of a wire, cannot see the wire.** A journey that crosses it — a click on the phone that must arrive at the host and come back rendered — needs a browser harness that asserts the arrival. (The remote-control product's sign-in journey test also measured the sign-in link against its own button, which is how it caught the link shipping at 1.12:1 contrast.)
- **An old client must keep working against a new backend, and a test holds that contract.** The client is often whatever the user installed, while the backend is always as new as the last deploy. A compatibility harness drives the real client page against a stub host that speaks only the last published client's message set, from a **frozen fixture** so that history cannot drift. It asserts that the page boots clean, that an affordance the old host cannot feed **hides itself** rather than rendering a dead control, and — the forward-looking one — that the new side sends **nothing the old one would drop**. It also asserts the contrast case (a host that does send the new message gets the feature), so the gate cannot pass by simply breaking the feature. The release order that goes with it is in `release-gate` (`references/runbook.md § C`).
- **Replay real captures.** A fixture of a vendor tool's stream is recorded from the tool itself and replayed in the browser at phone and desk width, rather than written from its documentation.
- **A check that costs money or buys real resources is an operator check**, outside every gate and run on purpose after the change it covers — even when it is the only test of the promise the product is sold on, end to end. Live model suites are the same kind (`release-gate`, `references/runbook.md § A`).
- **The smoke that proves a deploy is often unauthenticated, so it cannot see the authenticated mile.** Pair it with an **authenticated canary** — a real session, link approval, device listing, connection and revocation, on dedicated canary credentials, never a real user's — run after any deploy that touches sign-in, sessions, ownership, entitlements or connection admission.
- **When a hosted CI job runs a subset of the local gate, write down what it leaves out and why** (credentials it does not have, a sibling checkout it cannot clone), and put the commands and results of the local run in the pull request, so a reviewer can tell an unrun gate from a failure.

## The browser configs and the shared server module

- **Two configs for the browser suites, not two projects in one.** `playwright.config.ts` is the public suite and `playwright.auth.config.ts` the authenticated one. Nothing in the second — the testing token, the test accounts, the workspace reset — runs when the first does, so `npm run e2e` still needs no identity.
- **Both, and the visual config, share one `webServer` module.** The build directory (`NEXT_DIST_DIR=.next-e2e`), the port (`E2E_PORT`, default 3101) and the app URL are declared there and nowhere else, and so is **every optional configuration a suite depends on, each pinned on or off**, so the developer's `.env` cannot decide what a suite asserts (`rules-from-bugs.md § The harness's environment`).
- **The module sets `reuseExistingServer: false`** (`rules-from-bugs.md § Servers, builds and worktrees`). The suites cannot drift apart, and a server started by hand copies the module rather than remembers it.
- **The browser suites build into their own directory and serve on their own port**, so they never fight a `next dev` you have open — Next refuses two dev servers in one repository.
- **All three configs pin `colorScheme`, and say why in the config** (`rules-from-bugs.md`, a harness default decides what every test asserts).
- **A suite whose screens show a secret turns off trace and video**, because they keep what the page showed, a one-time token in a dialog included. Screenshots stay off too, unless a capture must show the screen: then it masks the secret's element (Playwright's `mask` option) or revokes the secret in a `finally`, so the image holds a dead one, and the spec says which. Playwright's trace is a worker option: `test.use({ trace })` inside a `describe` is refused ("forces a new worker", 1.62.1, `2026-11-07`), so turn it off for the whole file or config, or give the secret screen its own file; where it must stay in a shared capture file, revoke the secret when the run ends, so a retained trace holds a dead one. (`2026-10-23`, a monitoring product's API-token journey.) (`2026-11-07`, a live Q&A product: a release capture reviewed by people held a valid write token until the next run, and its one secret row sat in a 32-row capture file that retains traces on failure.)

## The script contract

- **`typecheck`** — `tsc --noEmit --incremental false`; covers `tests/` too, so keep `tsconfig.json` including `**/*.ts`.
- **`test`** — unit + integration. A product whose integration suite needs a credential the unit run must not (such as development's connection string, `DEV_DB_URL`, above) splits it out as `test:integration`, still inside `gate` and listed here.
- **`e2e`** · **`e2e:auth`**.
- **`gate` runs the applicable contract at phase end and before deploying pushes.** By default, it includes all the suites above, in that order, for the product's shape and current stage. Compare `package.json` with this contract; the owner decides any smaller set, recorded in the plan with its reason and revisit trigger.
- **`visual`** — outside `gate`, run on purpose.
- **`smoke -- <url>`** — the URL always explicit.

There is no hosted CI unless it was wired as its own change (`release-gate`, `references/go-live.md § Guards and hosted CI`). If it is, it runs `gate` minus `e2e:auth`, and that gap is covered by the local gate, never by a silent skip. **The names here match `package.json`; a script that does not exist is not listed.**

For example, a product can add a unit test that fails when its `gate` script stops invoking `e2e` or `e2e:auth`. Check what the commands execute, including aliases, rather than accepting a name in a comment. (`2026-11-14`, an evals product: nine development deploys passed a gate with neither browser suite; every member-action test mocked the session module.)

## The authenticated mile — what it takes, what it resets

The suite that signs in as real accounts on the identity provider's development instance. The design holds on any provider; the Clerk mechanics are summarised here and kept in depth in the `clerk-production` skill.

**One value to set, and no way to set it half-way.** Two variables: a test admin address (on Clerk, an address carrying the `+clerk_test` subaddress) and the verification code the development instance accepts for every such address (on Clerk, while **Test mode** is on). No mailbox, no real person's account. **Every further account — a second admin, one per additional role — is derived from the first.**

**`globalSetup` runs before any browser opens**, and does everything a worker will need, because a worker inherits only the environment that existed when it was spawned:

1. fetches the provider's testing token (Clerk's bot protection refuses a headless browser without it);
2. creates the test users if the instance lacks them;
3. in the organization shape, **creates an organization for each and lands it past any screen the provider puts in front of the app** (Clerk's `choose-organization` task: with organizations on and membership required, a session without one is pending, and `auth()` treats it as signed out);
   - **One new-customer test covers what setup skips:** a fresh test address per run (on Clerk, `+clerk_test`), through the provider's own sign-up component (`<SignUp/>`) and the organization task to the member home and a first created resource; teardown deletes the user and organization. (`2026-11-07`, a live Q&A product: its adoption pass found no suite drove sign-up or the organization task.)
4. gives each secondary role its membership in the first admin's organization;
5. deletes everything those accounts own.

**The teardown deletes it again. A run starts from nothing and leaves nothing, and running it twice in a row, as two separate invocations, is the check that this is true** — Playwright's `--repeat-each` repeats tests, not `globalSetup` or teardown, so it is not a clean repeat. Never pick one test out of a serial group either: it depends on the records and ids the journeys before it left.

**In the individual shape** (one user per workspace, proven once, `2026-10-11`, on one product), `globalSetup` is shorter: it refuses to start unless the test address and the code and the provider's secret key are set; fetches the testing token; and finds the test user through the provider's backend API, creating it when missing. No organization, no task to land past, no second-role membership. **It never creates the workspace row**: the app's session seam creates it on first sight, the path a real user takes. The reset is not yet proven in this shape — nothing the first product's suite asserted depended on rows an earlier run left — so a suite whose assertions do deletes what the test account's workspace owns, in `globalSetup` and in the teardown, as above.

**Three rules for the sign-in itself:**

- **Sign in through the provider's own component** (Clerk's `<SignIn />`), not a programmatic sign-in. The product's sign-in surface *is* the vendor's component, and this is the suite that has to notice when it stops working.
- **Wait for the event, not the field.** A verification-code box can render while the code is still being sent; on Clerk, wait for the `prepare_first_factor` call before filling the code, never for the input to appear (`rules-from-bugs.md § Browsers and layout`).
- **Make a silent refusal name itself.** Record any Server Action response the provider marks as signed out (on Clerk, `x-clerk-auth-status: signed-out`) and throw it in `afterEach`. A development instance occasionally refuses an authenticated action with a `200` and nothing on screen; this does not prevent it, but the failure then says what happened (`clerk-production`).
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall