Skip to content

phuryn/product-engineering / ai-shipping/skills/test-harness

test-harness

Verified

Writing or running tests, proving a fix, a red or flaky run, a fix that didn't land, 'is it verified', 'run the gate': tests where green means something.

Test harness — so a green gate means something

Make each harness prove something real before trusting its green result. A suite can pass against the wrong server, database, fixture branch, or developer's .env and prove nothing. This skill arranges the harnesses, defines gate runs and reporting, and keeps green runs honest with rules learned from bugs.

On another stack, keep the principle and swap the mechanism. Each rule states its principle first and the mechanism second. The web-app mechanisms use Next.js (App Router), Supabase Postgres, Clerk, Vitest, and Playwright. The two-halves mechanisms were learned on a relay service with a web client and an editor extension with desktop installers that drives a CLI agent.

The product's own instructions come first — where they name its scripts, folders, or coverage file, follow them.

The one question: what does each harness fake?

A bug survives exactly where every harness fakes the same side. Coverage of surfaces is not coverage of journeys. Before adding a suite, write down what it makes real. If everything else already fakes the same side, it mostly re-covers ground.

HarnessMakes realFakesRuns
Unitpure modules: authorization functions, schemas, token hashing, limits, escaping, the env schema, source guards, the edge-rule numbersdatabase, identity, browsergate
Integrationthe development Postgres: the applied schema, deny-all, cross-tenant, invariants and races through the real SQL functionsidentity (a mock verifier behind the same seam as the real one)gate
Schema catalogevery public relation and function, enumerated from the catalog so future objects are coveredidentity, fixturesgate
Browser, publica real browser on a phone profile against the development database; the anonymous journey needs no accountmember identitygate
Browser, authenticatedthe identity provider's real development instance and real sessions, an account per rolemaillocal gate
Visuala bounded release set of captures; on request, a full sweepidentity where a capture does not sign inoutside the gate, on purpose
Live agents (only with embedded agents)the real model with each agent's tools—outside the gate: it costs money
Smokea deployed URL, write-free—after every deploy, last

When a product has two halves, ask what is faked per half. This includes a backend and client, or a host and the vendor tool it drives. Defects needing both real halves, or a restart mid-journey, are invisible when every suite fakes the other side. Build one harness with both halves real that can kill a process mid-journey. Add a compatibility suite that holds an old client working against the new backend.

Every row in full, including its target and run placement, is in references/harnesses.md. That reference also covers the two-halves harnesses, browser configs, script contract, and authenticated mile.

The gate

Run the full local gate at phase end and before deploying pushes. Development deploys need browser coverage too. By default, gate runs typecheck, test (unit + integration), e2e, then e2e:auth, in that order, as applicable to the product's shape and current stage. Compare the actual script with references/harnesses.md's script contract; a green script can omit a whole suite. The owner decides any narrower gate; record that choice in the plan, with its reason and revisit trigger.

visual, the live agent suite, and smoke -- <url> run outside it on purpose. There is no hosted CI unless wired as its own change (release-gate). If wired, it runs the gate minus the authenticated mile. The local gate covers that gap: never skip it silently, and name any suite that did not run.

Run touched tests at stage end; reserve full gates for phases. The coordinator runs new and touched tests itself, checks changed journeys' end-to-end specs, and reads the diff and corrected docs. The read of the diff runs the router rows it matches; an edited test, gate script or instruction file needs its reason. It takes a quick look at running screens at both widths and, when present, both themes. Run the full gate on the finished phase before its exploratory and visual checks, docs check and review series (independent-review § When). Rerun it on the cleared state if a fix followed its last full run, before the closing push. Time one full run and write it in the plan; phase estimates count required runs. A gate of a couple of minutes can run on every piece.

Tests and docs are part of the change, not a later step

  • Budget integration tests in round trips to the real database. Include fixture setup and cleanup at the target's measured latency. A test needing more than 30 seconds states why and sets its own limit; don't raise the whole suite's timeout. When the builder cannot reach the database or launch the browser, budget one integration-fix pass by the session agent after its first real run. Put that budget in both the brief and stage plan.
  • A bug fix lands with a regression test run against broken behaviour. It must fail on the broken behaviour and pass with the fix; run the failing half, don't assume it. Run the new test on the tree before the fix: the parent commit, or the fix reverted. If it passes there, the bug was not what the finding said: revert the fix and keep the assertion. Prove a review finding's fix the same way.
  • Any other behaviour change lands with the tests it owes (below and in the references). Put each new test in the harness whose Runs column says where it runs: in the gate, outside it, or by hand. Say which in the summary.
  • When a rule must hold for every input, test it as a property. An example picks inputs with its writer's bias; a generator does not. State what cannot fail: a round trip, an invariant, idempotency, or two implementations agreeing. Let fast-check, Hypothesis or proptest try to break it, with a bounded run count in the gate. Commit each counterexample as a regression test.
  • When a rewrite must not change behaviour, run old and new on the same inputs. Every difference is a bug or an approved change. Normalize what does not matter, such as whitespace, and list the approved differences. Inputs come from generators, fixtures or a corpus, never production data.
  • Each stage lands end-to-end tests of journeys it adds or changes. Write each flow's test once and commit it with the change; the gate reruns older journeys. A scratchpad script or an exploratory click-through leaves no durable regression check. A mocked session tests data rules, not the signed-in journey. Steps costing money or needing a real inbox stay in the live or operator suite; the gate covers the steps around them. The owner decides exceptions, recorded in the plan with a revisit trigger.
  • The builder runs its tests and fixes failures before the handoff. Give it development test access and serialize suites on shared data. Apply codex-agents for a Codex build's test-running mode and delegation for its boundaries. Where the tool or platform cannot run them, name the concrete blocker and the session agent's checks; budget the fix pass above. A test changed to make it pass is named in the handoff, with why: a loosened assertion or an added fallback can turn a real failure green. The coordinator reruns touched and new tests at stage end and the full gate at phase end.
  • Every change updates the doc lines it made false. If a doc disagrees with code you did not change, say so in the summary and let the owner decide which side moves. When instructions are silent, the skills' defaults apply. When those are silent too, decide and say so in the summary. When a library default conflicts with the product's instructions, the instructions win.

When a fix doesn't land

After two fixes that pass their checks and leave the reported symptom unchanged, stop fixing. Green checks with an unchanged symptom mean the model of the problem is wrong: the tests exercise a sequence the person never meets. Write down the explanations still standing. Get one observation whose outcome tells them apart: a screenshot, a log line from their run, or a question answerable in a minute. Ask after the second fix that didn't land, not the fifth.

  • First check that the person is running your build. A fix tested against yesterday's install fails twice for no reason in the code.
  • Reproduce the original symptom before the next fix, in a test or by hand, and keep that reproduction. It is the regression test the fix lands with.

The fix that lands comes with a test that reproduced the symptom before it, and the person confirms the symptom is gone in their own run.

Required for every feature that touches data

Every feature that touches data gets the required boundary and integrity tests. These checks establish:

  • A workspace A member cannot see or act with workspace B's ids.
  • Refuse a once-only action's second attempt; hold the quota under a race.
  • A script tag renders as text.
  • Idempotent work runs twice and acts once.
  • A token does only what it was scoped to.
  • Every page renders once: A page-level smoke in the browser suites opens each page as each role, with a foreign workspace's id, and signed out. Assert it renders, refuses, or sends to sign-in as its permission row says, never an error page. Per-route tests prove refusals; only this proves every page renders.
  • Every new database function runs end to end once, in the integration suite against the applied schema, with the rows it will meet. A migration applying and a catalog check passing prove nothing about a body plpgsql checks only when run (supabase-production, references/gotchas.md).
  • Test deletion of a fully populated workspace as one test. Put a row in every table a member can fill, then delete the workspace. The deletion completes and leaves none.

The saas-authorization skill explains what each protects and why invariants live in the database rather than code.

Reporting a run

  • Read every exit code from the log, never the notification. A background wrapper can report exit 0 while the command inside it failed.
  • The gate is the exit code, never the printed counts. A run can print every test passed and exit 1 on an unhandled rejection. Counts without the exit code establish nothing (references/rules-from-bugs.md § Reading a red run).
  • An inherited report is a claim, not evidence. Re-run the gate yourself before repeating a count.
  • Every preview and go message names each suite that did not run. Counts alone can read as full coverage. For example: "Unit and integration passed; e2e and e2e:auth did not run: browser launch blocked." Include the reason and what unblocks each missing run.
  • The coverage map is written by whoever ran the gate, and by nobody else. A builder records only what it actually ran, including blocked or partial suites. Read every line it adds and rewrite completion evidence from your own applicable run, never by editing its numbers. Use references/coverage-map.md for the map's format.
  • A green run that flakes is not a pass. Judge a suite on a series. Record both a red run and the green one after it. A flake goes in the backlog on its first appearance; its second makes the fix due (references/rules-from-bugs.md § Reading a red run). Never cure one with a sleep, a retry until green or a skip: each hides the cause. A suite that has never been green reports nothing and hides the next real red.
  • A failure whose duration equals the budget is a hang until shown otherwise. Read the trace, network, and whole log body before the failure list (references/rules-from-bugs.md § Reading a red run).

The rules most often missed

Each of these rules has its receipt in the references. They are the ones that cost the most when skipped.

  • A fallback in a test is a bug in disguise. "Try the new way, else the old way" passes exactly when the thing under test is absent. Assert the one true path.

  • A pin must not pass against a dead branch. A substring assertion is satisfied by if (false && …). Assert the whole conditional and watch it fail under that mutation.

  • A new precondition breaks the harnesses CI does not run. If a change adds something a person must do before the product starts, every harness launching a real instance must seed it. Run the suites outside CI before the release, not inside it.

  • Test sign-in the way a new user meets it, with no saved credential on the machine; a signed-in machine passes a broken sign-in.

  • A reused server can be a build from before your change. The browser suites never reuse a server already on the port. When an assertion disagrees with source you just edited, check what answers the port before changing code.

  • The developer's .env is a harness default. Pin every optional configuration a suite's meaning depends on in the shared server module. Otherwise, one assertion means different things on different machines.

  • A DOM assertion cannot see a layout. Claims about layout need geometry, and a person looks at the captures.

  • Nothing else touches the gate's data while it runs. Run browser suites one at a time: they share a server and test accounts, and each resets what those accounts own. The same holds for every other actor on development. That includes a parallel agent driving a test account and a background worker acting on the suites' fixtures (references/rules-from-bugs.md § The harness's environment).

    A deployed worker never stops. Before the first deploy running one against development's database, scope it away from test-account workspaces or give the suites their own database. That can be a test project or the local stack. Until then, let suites' setup fail loudly when anything outside them has claimed one of their fixtures.

  • Never test against production data or with production credentials. Use the development instance, development project, and dedicated test accounts. Only the smoke touches production, and it writes nothing.

  • A generated file can arrive rewritten by the tooling that wrote it. A backtick inside git commit -m "…" is command substitution: the shell runs the word and replaces it with its output, usually nothing. The text vanishes silently. Commit messages go through a heredoc (git commit -F -). A test written through a shell into an interpreter is rewritten the same way. Generate through the fewest layers, using the agent's file-write tool, and read the bytes produced (references/rules-from-bugs.md § Generated files).

  • A suite outside the gate rots silently. A screen's captures are part of its surface. Update visual specs in the same stage, and run the visual suite at phase end when screens changed.

Each rule's origin, the case that taught it, is in receipts.md beside this file; read it when a rule's edge is unclear.

How you can tell it worked

Your own run covered the tests the piece touches, or the full gate at the phase's end, and you read its exit code. Each new test sits in the harness whose Runs column says where it runs. The coverage map was updated from the full gate's run.

Files

FileHoldsRead when
references/harnesses.mdevery harness in full; harnesses for a product with two halves (both real, old-client compatibility, the wire between DOM suites, operator checks, the authenticated canary); the two browser configs and the shared server module; the script contract; the authenticated mileadding a suite or a script; a client and its backend; setting up or debugging the authenticated browser suite
references/rules-from-bugs.mdthe gate and harness rules, grouped by moment: reading a red run (the exit code, flakes, hangs, rotted checks); writing assertions; fixtures; the database; servers, builds and worktrees; browsers and layout; the harness's environment (preconditions, sign-in without a credential); same-origin checks; generated fileswriting or running any test; a red, flaky or hanging run; creating or removing a worktree
references/captures.mdthe capture harness and the exploratory pass: the release set as a budget, where captures live, sweepsat the end of a phase that changes a screen; before a release's visual pass
references/vendor-surfaces.mdvendor-rendered elements measured from the DOM; link previews asserted on the raw response; icons and generated imagestesting a third-party component, a share card, an icon or an OG image
references/coverage-map.mdthe coverage map's sections and who writes itafter a gate run; at a release; when a stage adds tests

Use the related skills for screen claims, coverage documents, and release mechanics. What a screen may claim and what to verify visually are outside this skill: widths, states, themes, roles, and the 320 px floor. Use the ui-design skill's references/screens.md. The coverage map as a document, including its required reviewer contents, is shipping-artifacts. The derive-tests skill derives one from docs; its check mode lists stale entries and writes nothing. Use release-gate for the release checklist. The identity provider's harness mechanics in depth are clerk-production; the database's catalog posture and error codes are supabase-production.

The folders and scripts named here are a suggested layout: Call yours what you like:

  • tests/unit/, tests/integration/, tests/e2e/ (*.auth.spec.ts for the authenticated suite), tests/visual/, tests/live/, tests/smoke/;
  • typecheck, test, e2e, e2e:auth, gate, visual, agents:live, smoke (references/harnesses.md § The script contract).

The coverage map is the product's record of what was verified, by which run (references/coverage-map.md). The variables doc lists every environment variable the code reads. The kit is the product's design system and component primitives (ui-design).

finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 319kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 319kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 319kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 196kUpdated Claude CodeCodexVerifiedInstall