shift-editor/shift / .codex/skills/writing-e2e-tests
writing-e2e-tests
UnverifiedCanonical rules for writing, rewriting, or reviewing Shift desktop Playwright E2E tests and visual goldens under `apps/desktop/e2e/`. Use whenever you add or change a `.spec.ts`, a fixture, `EditorDriver`, a screenshot baseline, or Playwright project membership, and whenever you investigate a flaky or failing E2E test. Covers waits, oracles, goldens, fixtures, projects, and flake verification.
/writing-e2e-tests — How desktop E2E tests are written
An E2E test is worth its cost only when it fails for exactly one reason: the user-visible behavior it names is broken. It must not fail because the machine was slow, the theme changed, a sidebar moved, or a retry happened to pass.
Read /writing-tests first for the general rules (observable state, no mocks, migration ledgers). Read apps/desktop/e2e/README.md for the full fixture and project reference. This skill is the checklist that turns those into a test that stays green for the right reasons.
Choose the layer before writing anything
| The behavior is… | Owning test |
|---|---|
| Pure computation (layout math, latch rules, range selection) | Pure unit test — extract the function first |
| Observable through editor state after a click, drag, or key | TestEditor tool or command test |
| A real browser or Electron contract (DOM events, focus, native menus, dialogs, windows, processes, files) | Semantic Playwright test |
| Appearance: colours, strokes, handles, layout | One focused golden, after a semantic precondition |
| Hardware rendering, GPU residency, WebGPU presentation | gpu project test |
| Save, quit, crash, recovery, file activation, cross-OS paths | platform project test |
If most assertions in a draft E2E test read editor state, the behavior probably belongs in TestEditor. Keep the E2E test as a thin proof that the real UI reaches it.
Waiting: never sleep
waitForTimeout is banned. Wait for the condition that makes the next step valid:
| Before you… | Wait for |
|---|---|
| Read geometry, project coordinates, or send pointer input after setup changed geometry or camera | editor.waitForCanvasRender() |
Assert after raw page.mouse moves | editor.flushPointerMoves(), then waitForCanvasRender() or waitForIdle() |
| Act on an editor route | waitForEditorReady() / openCatalogGlyph() (works for authored and preview sessions) |
| Click a catalog cell by coordinates | clickFirstCatalogGlyph() — it waits for a laid-out, settled Grid |
| Act on a workspace | waitForWorkspaceReady() |
| Assert after quit or close | dirtyDocumentDecisions() / dirtyDocumentRequests() — main-process records, not elapsed time |
| Compare Grid frames after an edit | Published Grid state (data-grid-readiness, data-atlas-build-count, active location), armed after setup settles |
When elapsed time is itself the contract (debounce, momentum, idle timeouts), put the rule in a pure function driven by timestamps and unit-test it. If the E2E test must remain, dispatch events inside the page and wait on the page clock (performance.now()) that stamps them. glyph-view.spec.ts zoom momentum is the template.
Negative assertions ("nothing else happened") need a positive anchor: wait for the event that would have triggered the unwanted outcome, then assert the outcome is absent.
Oracles: assert what the editor decided, not what pixels look like
Never count or sample hard-coded colours from a canvas. Colour oracles break under themes, alpha, anti-aliasing, device scale, and renderer changes, and many pass when the feature is broken (an empty canvas has zero "wrong" pixels).
Use published state instead:
- geometry:
editor.outline(),point(),selectionIds(),selectionBounds(); - interaction:
toolState(),hoverId(),activeSnapGuides(); - rendering decisions:
visibleOutlines(nodeId)andhandleStates(node)on the glyph node definition; - UI state: roles, accessible names,
aria-pressed,aria-checked,aria-disabled,data-*state published for tests; - persistence:
savedGlyphNames()for.shiftfiles andexportedGlyphNames()for exported fonts — neverexistsSync, byte inequality, orsize > 0.
If the state you need is not published, publish it from the production code that renders it (same computation, not a parallel reimplementation), add a unit test for it, then assert it. That is how visibleOutlines and handleStates were added.
Assertions must be exact enough to fail on the wrong answer:
- assert the exact set or order, not
count() > Nornot.toEqual(before); - assert a read whole, as behaviour tests do:
await expect.poll(() => editor.documentState()).toMatchObject({ dirty: true }), not.poll(async () => (await editor.documentState())?.dirty)(seewriting-testsTest helpers;shift/no-drill-into-readslints it); - derive expectations from the fixture or the UI (for example, read the ordered category buttons and compute the expected range) instead of hard-coding one neighbour;
- run the mental-deletion check: if the feature were a no-op, would this assertion fail?
Goldens: one contract, one focused capture
All goldens go through fixtures/snapshots.ts; scripts/check-e2e-projects.mjs rejects toHaveScreenshot or toMatchSnapshot anywhere else.
expectCanvasSnapshot(editor, name)— the editor canvas stack, exact comparison, after the canvas has rendered.expectPanelSnapshot(locator, name)— one panel, menu, toolbar, or dialog.expectPageSnapshot(page, name)— full window; use sparingly, only for overall chrome.
Before every golden:
- Assert the semantic precondition that makes the image meaningful: the point count, selection, tool state, outline targets, handle states, zoom, active theme (
data-color-theme), ordevicePixelRatio. - Park the pointer off the captured element unless hover is the contract.
- Capture the smallest element that shows the contract. Do not use a full-page golden to protect a toolbar button.
Rules:
- One golden per distinct visual contract. Do not add near-duplicate images; if two captures are byte-identical, one of them is redundant.
- Canvas goldens use
maxDiffPixels: 0withthreshold: 0.02on every host: enough for antialiasing differences between hosts (≤ 0.009), far below a real colour-token change (~0.07). Never raise either to make a mismatch pass; the default threshold of 0.2 silently accepts token changes. - Interface goldens (
expectPanelSnapshot,expectPageSnapshot) are exact but compared on CI only, with baselines generated on the runner via theci: update visual snapshotslabel. Never commit a locally generated interface baseline. Prefer semantic assertions and keep interface goldens few. - Tag every test that captures a golden:
test(title, { tag: "@golden" }, …). The snapshot label regenerates only@goldentests, and the helpers fail an untagged golden. - Goldens never pass on retry. The helpers throw on a retry attempt, so explain the first failure.
- Use canvas-local positions (
dragCanvas(),canvasPagePoint()) so layout changes cannot redraw geometry. - Theme goldens select the theme through the product and assert
data-color-themefirst. HiDPI goldens usetest.use({ deviceScaleFactor: 2 })and assertdevicePixelRatiofirst. Add one golden per palette branch or rendering path, not one per theme.
Updating baselines: only for an intentional appearance change. Run the focused spec with --update-snapshots -g "<title>", open every changed PNG and describe what changed, then rerun without update mode and with --retries=0 --repeat-each=3. Canvas baselines may be generated locally. Interface baselines are generated only by the ci: update visual snapshots label; inspect the runner-generated images before merging.
Fixtures and isolation
- Use the shared
editorfixture; constructEditorDriverdirectly only for additional windows. - Relaunch through the
relaunchfixture, never rawelectron.launch. The process registry owns every launched tree, attaches diagnostics, and kills it even when the test times out. - Every launch environment comes from
shiftTestEnvironment(). Never spreadprocess.envinto a launch. - Script native dialogs with the fixture options (
dirtyDocumentChoice(s),saveShiftPaths,openFontPath). An unexpected renderer dialog fails the test; opt in withallowRendererDialogsonly when the dialog is the behavior. - Put shared setup in fixtures or
EditorDriver(openScratchGlyph(),commitInputValue(),toolButton()), not copied helpers in specs. Do not growEditorDriverwith one-off helpers. - Draw glyph content with
editor.draw(fixture)and the shapes from@/testing/fixtures(rect,square, …), name the points a test needs, open fonts byFONTS.<name>and useWEIGHTfrom@/testing/fonts: the same fixture data vitest uses. - When a test's setup is a font build (glyphs, axes, sources, source
metrics,instances, the startinglocation), settest.use({ font: FontFixture })onworkspaceTest,perfApp'stestorrecoveryTest; it is built before the test, asuseTestEditorbuilds it. A test that needs an id reads it through a selector (source({ name }),axis({ tag })) only where identity is the point. - Prefer
getByRole()/getByLabel()and locators fromfixtures/appLocators.ts. No parent traversal (locator("..")), no styling classes, notoHaveCSS()unless the style value is the product contract.
Structure
- One behavior per test. Split long flows when an early failure would hide unrelated coverage.
- Do not accumulate state across unrelated steps; start each test from a fixture.
- Setup the font fixture does not cover goes through the driver's surface actions (
addAxis,addSource,setSourceMetric,renameSource,addInstance,setLocation,editSources, …). They are the vitest surface's actions under the same names, written once intesting/SurfaceActions.ts, and each calls the public command the UI calls. The behavior under test must go through the user surface. - Specs and fixtures never call
page.evaluate,page.evaluateHandle,page.waitForFunctionorelectronApp.evaluate(shift/no-page-evaluate-in-specs). The one exempt file isfixtures/appHooks.ts:readsOf(page)calls the window's app readers (agent/AppReaders.ts, the read layer MCP agents also use) andhooksOf(page)the renderer's E2E hooks (testing/e2e/hooks.ts: the surface actions, canvas targets for aiming the pointer, session facts, perf instrumentation) as typed method calls. Fixtures that only measure the page or drive Electron main mark each evaluation with a reasoned disable;locator.evaluateon a provable locator (agetByRole(…)/locator(…)chain or a binding annotated: Locator) is fine.pnpm --filter @shift/desktop lint:checkenforces it, withshift/no-lookup-in-testsandshift/no-drill-into-reads; the remaining spec disables wait on the open-glyph refactor'sdrawnGlyph(#824). - Name tests after the behavior: "shift-selects glyph categories as a visible range", not "category test".
Projects
Membership is explicit in apps/desktop/playwright.config.ts. Add a new spec to the right list — visual, platform, gpu, or perf — and run node scripts/check-e2e-projects.mjs. A spec that needs native lifecycle behavior belongs in platform, which runs fully on Linux. Tag a platform test @os only when its behavior differs by OS (window controls, signals, native menus, file activation and paths); only @os tests run on macOS and Windows. A spec whose only GPU dependency is incidental belongs in visual.
Large-data performance regressions
- The
perfproject runs nightly or manually, not in the normal correctness suite. Protect a large-list UI change with a smallvisualinteraction test that reaches an offscreen row, preserves selection and keyboard focus, and bounds the number of mounted rows. - In a real-app performance test, time the whole user-visible operation through confirmed edit settlement and canvas readiness with the relevant panels mounted. Timing only a local mutation, Rust apply, or IPC handler misses renderer main-thread stalls. Keep hardware-sensitive time ceilings in
perf, notvisual; verify the test fails for the original regression rather than only when it times out. - Use
EditorDriverfor navigation and edit settling. Keep the performance fixture responsible for the hardware launch and generated data, not editor behavior.
Proving a test is not flaky
Before committing a new or changed E2E test:
pnpm test:e2e:visual e2e/<spec>.spec.ts -g "<title>" --repeat-each=10 --retries=0
For timing-sensitive flows, repeat under CPU load (for example, several yes > /dev/null & processes, killed afterwards), because CI runners are slower than development machines. For platform behavior, rely on the Windows and Linux CI jobs and say so in the pull request.
When a test fails or flakes:
- Read the trace and
electron-diagnosticsbefore changing anything. - Classify it: product regression, test oracle, or infrastructure. Do not call a single failure flaky without a retry pass or a rerun on the same commit.
- Fix the oracle or the wait. Never add a sleep, a retry, or tolerance, and never regenerate a baseline without understanding the diff.
The CI "E2E Report" job lists tests that passed only on retry. Treat every entry as a bug to fix, not noise.
Review checklist
- The layer is right; logic that
TestEditorcan observe is tested there. - No
waitForTimeout, no fixed delays, and no polling without a semantic condition. - No colour counting, pixel sampling, byte inequality, or "something changed" oracle.
- Assertions are exact and fail when the feature is a no-op.
- Every golden goes through
fixtures/snapshots.ts, captures one contract at the smallest element, and follows a semantic precondition. - Changed baselines were inspected and verified with
--retries=0. - Launches, relaunches, and dialogs go through fixtures.
- The spec belongs to the intended project, and the project check passes.
- The test passed
--repeat-each=10 --retries=0, and the pull request lists the exact E2E commands and any coverage not run.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |