phuryn/product-engineering / ai-shipping/skills/test-harness
test-harness
VerifiedWriting or running tests, proving a fix, a red or flaky run, a fix that didn't land, 'is it verified', 'run the gate': tests where green means something.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 18.6 KB |
| receipts.md | markdown | 7.1 KB |
| references/captures.md | markdown | 7.2 KB |
| references/coverage-map.md | markdown | 5.4 KB |
| references/external-creates.md | markdown | 1.6 KB |
| references/harnesses.md | markdown | 15.6 KB |
| references/rules-from-bugs.md | markdown | 29.3 KB |
| references/vendor-surfaces.md | markdown | 6.8 KB |
references/captures.md
View on GitHub# Captures and the exploratory pass
The visual harness and the person who looks at it. A capture is a picture of a screen taken by a spec; the exploratory pass is a person opening the running app and going looking. A release needs both, because each is blind to what the other sees.
What to look *for* — the widths, the states, the themes, the roles, the 320 px floor asserted with geometry, and what a screen may claim — is the `ui-design` skill's `references/screens.md`. This file is how the captures are kept so that the looking is worth something.
## Why both
- **A capture pass finds what no assertion can see, including a state nobody styled.** Look at screens before a release; when a capture finds a defect, check whether a test is agreeing with it. (`2026-09-16`, a test pinned the primary fill on the disabled control whose missing style the capture exposed.)
- **A capture pass is a checklist; a release also needs an exploratory one.** A fixed list cannot reach what it never names, and a capture is not a judgement — some defects require measurement. Before release, explore changed screens and affected shared surfaces, including third-party renders. Sweep the whole product at go-live or after broad UI changes. Use an account for each affected role in one organization (in the individual shape, one account per test user). An admin sweep never sees the second role's screens. Cover applicable lifecycle states, themes and widths, including the signed-out surface where applicable, and feed discoveries back into captures. Apply `release-gate` § The owner's preview for the promotion scope. (`2026-09-16`, the owner found defects outside the capture list and inside PNGs already inspected.)
## The release set is a budget, not a matrix
- **Screens × widths × themes × states × roles multiplies until nobody can read the gallery, and a gallery nobody reads is not evidence.** Keep the release set to a fixed number of rows, each naming one width and one theme and crossed with nothing, split into standing rows and rotating ones. **Guard the count in the spec**, so a new row has to name the row it displaces. The full sweep stays an instrument run on request, not a release step. (`2026-09-19`, a live Q&A product's two capture specs had reached 4,747 PNGs of about twenty screens.)
- **A row earns its place with new pixels, not a new assertion.** A page that renders many states at once needs one row; find duplicates by hashing the images, not by reading the scenario names.
- **Say out loud which cell you are not buying.** Write the uncovered cell, and why it was the cheapest to drop, into the coverage map's gaps as the first candidate for the next rotation (`coverage-map.md`).
- **A green capture run proves completion, not coverage.** Read the row names against what the release changed before calling the pass done. (`2026-09-19`, three rotating rows anchored on different cards of one page were the same photograph 287 bytes apart while a whole editor had no row; `2026-09-21`, every row green and none showing the control the release changed.)
## Reading a capture
- **A capture proves a screen was seen, not that it was compared against the system.** A capture catches a screen that moved, never one that never matched the kit: a single frame cannot say what its spacing was supposed to be, so a defect that is the same in every release reads as that screen's style. When a screen is added to the captures, or a capture is read, **look at it beside its siblings** — the product's other forms, lists, dialogs — because the comparison is the instrument. Fix a mismatch with the kit primitive that already owns the pattern, never with a number invented for the screen (`ui-design`). (`2026-09-22`, a form caption sat directly on its input in the same capture across several releases, until the field was put inside the kit's field-group primitive.)
- **A layout or performance finding is a hypothesis until it is measured on the running build, and the measurement settles it in either direction.** Reading the stylesheet can give correct arithmetic and a false conclusion — a flex item's automatic minimum is its content, so cells wider than their container may shrink rather than overflow — and "obviously it shrinks" is as much a guess as "obviously it overflows". **Measure the copy, not the container**, and compare it with what sits beside it, because matching box edges can be produced by the defect itself. For a performance claim, **measure both builds**: the baseline from `HEAD` in its own worktree with its own `npm ci`, one script against both, the numbers written down before and after. The measurement can retire the fix you proposed. (`2026-09-20`, a reported overflow measured 16 px clear at ten widths; `2026-09-25`, a proposed layout-shift fix targeted 0.002 of shift while 0.031 came from a poll.)
## Where captures live
- **Artifacts a person is meant to look at must not live inside a runner's output directory.** Playwright empties `outputDir` at the start of every run, even a run of one test. Put captures beside it: traces are the runner's scratch space, pictures are a person's evidence. (`2026-09-17`, rerunning one test deleted a 2,420-image sweep.)
- **A shell left standing inside that directory stops the runner clearing it.** Windows refuses to remove a process's working directory, and partial cleanup can mix two builds in one gallery. Reach in by path rather than `cd`, and check where shells stand before starting a tool that empties a directory. (`2026-09-17`, `EBUSY` exposed a mixed-build gallery.)
- **Read the gallery the run generated, not the folder.** A directory that accumulates across runs keeps an older image under a current row's name: list by time (newer than the run's start), and when the gallery is the deliverable, re-run the whole suite, never a filter. (`2026-09-19`, a live Q&A product: one capture directory was wiped while its sibling accumulated, and a filtered re-run left older images under current row names in an apparently complete folder.)
## Sweeps and the suite outside the gate
- **A sweep photographs only the states its own script can reach, and its action order can put one out of reach.** When a change adds a state, check that the sweep can still be in it, not merely that it runs. (`2026-09-17`, a fixture walked past the one state a new feature existed for.)
- **A suite outside the gate rots silently, and "don't touch the tests" is how you break the camera.** A screen's captures are part of its surface: update its visual specs in the same stage, and run the visual suite at the end of every phase that changes a screen. (`2026-09-17`, instructions to leave visual specs alone stranded them on removed controls and screens.)
- **Retiring a suite orphans the coverage that lived only in it.** Before a suite stops running, list the features whose only test at some layer was in it; move what matters into the gate and write the rest into the coverage map's gaps. (`2026-09-19`, a live Q&A product: a retired browser suite held the only browser test of the host's written answer, leaving none in the gate.)
- **Read the visual run's result from its log** (have the runner print its exit code as its last line, such as `VISUAL_EXIT=<code>`), with its image count, into the coverage map beneath the gate runs.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |