MaarkssDN/Birthub-360 / .agents/skills/flake-pattern-reference
flake-pattern-reference
UnverifiedReference catalog of the eight flake patterns - async/timing, test ordering, shared parallel state, resource leaks, network, locator drift, environment variance, randomness - with detection heuristics, remediation per pattern, and the concrete code-level fixes: replacing fixed sleeps with framework auto-waits, isolating state in beforeEach fixtures, per-worker DB schemas via workerIndex, try/finally teardown, mocking network + clock at the boundary, stable role-based locators, TZ pinning, and RNG seeding. Use when triaging an unknown flake to identify the category before bisecting, or when a classified flake needs the specific code change to apply.
flake-pattern-reference
Terminology note: "flaky test" is a practitioner-emergent term popularized by the Google Testing Blog (google-causes, google-flaky); ISTQB does not maintain a canonical entry. This catalog reflects industry-engineering consensus, not ISTQB authority.
A flake is rarely random - it almost always falls into one of eight
recurring patterns. Identifying the pattern early shrinks the bisect
search space dramatically. This catalog is a reference, not a
workflow; the matching workflow is in
flaky-test-quarantine.
The Google Testing Blog observed a near-linear correlation between test size and flakiness rate across ~4.2M tests (google-causes) - larger tests touch more of the eight patterns at once.
Pattern 1: async / timing
The most common flake category in UI and integration tests.
| Signal | What's happening |
|---|---|
| Fails ~5 - 20% of runs; passes when the machine is faster | Test waits for an arbitrary setTimeout(N) instead of a deterministic event. |
| Fails on CI but never locally | CI runners have different cold-start timings than dev laptops. |
| Fails after a dependency upgrade with no test code change | Library's internal timing changed (e.g. Playwright auto-wait window). |
Remediation:
- Replace fixed sleeps with deterministic waits -
await expect(loc).toBeVisible(),page.waitForLoadState('networkidle'),page.waitForFunction(...), etc. - For animations, disable them in test setup
(
animations: 'disabled'in Playwright;Cypress.config('animationDistanceThreshold', 0)in Cypress). - For absolute clock dependencies, freeze time with
sinon.useFakeTimers()/vi.useFakeTimers()/ Playwright'spage.clock.install().
Pattern 2: test ordering
Tests pass alone, fail when run with siblings.
| Signal | What's happening |
|---|---|
npm test -- --testNamePattern='^X$' passes; full run fails | Test relies on state from a previously-run test. |
| Adding a new test breaks an unrelated existing one | Implicit ordering dependency exposed by the new test pushing the old test into a different position. |
Random-order test runners (Jest randomize) flag the suite | Suite is order-dependent. |
Remediation:
- Run the suite with explicit randomization in CI to surface
ordering deps early (
jest --randomize,pytest --random-order,mocha --sortreverse). - Move ANY shared setup into
beforeEach/afterEach, never rely onbeforeAllfor state that the test mutates. - Database tests: roll back transactions after each test instead of truncating between describe blocks.
Pattern 3: shared parallel state
Tests pass sequentially, fail when run in parallel workers.
| Signal | What's happening |
|---|---|
Fails ~50% of runs in CI matrix; passes locally with -j 1 | Two workers writing to the same DB row / file / port. |
| Fails more often as worker count goes up | Linear shared-state contention. |
| Error message mentions "duplicate key" / "address in use" / "file already exists" | Direct collision evidence. |
Remediation:
- Find the shared state two workers collide on - DB rows, files, ports, or global module state.
- Per-worker isolation: per-worker DB schemas (
PG_SCHEMA=test_${WORKER}), per-worker temp dirs (TMPDIR=/tmp/test-${WORKER}), per-worker port ranges. - For unique IDs: use UUIDs or a per-worker namespace prefix, not auto-increment integers shared across workers.
Pattern 4: resource leaks
Tests pass on a fresh machine, fail after the test process has run for hours.
| Signal | What's happening |
|---|---|
| Fails increasingly often as suite duration grows | Memory or file-descriptor leak in the test setup. |
EMFILE / EADDRINUSE errors mid-suite | File-descriptor or port exhaustion. |
| Long-running processes (Playwright browsers, Cypress runners) crash mid-suite | Process accumulating zombies. |
Remediation:
- Always
await browser.close()/await server.close()inafterAll, with atry/finallyso failed tests still clean up. - Set per-test timeouts and ensure the framework kills the process,
not just the test (
--testTimeout,test.setTimeout()). - Run
lsof | wc -landps aux | wc -lbefore / after the suite in CI to detect leaks; alert when growth exceeds a threshold.
Pattern 5: network / external service
Tests pass when the upstream is healthy, fail otherwise.
| Signal | What's happening |
|---|---|
| Fails on the same handful of tests that hit the same external URL | Real network call to a flaky third party. |
| Fails right after a deploy of a non-test service | Test is hitting prod / staging of a sibling service. |
ETIMEDOUT / ECONNRESET in error logs | Network-layer error, not test-logic error. |
Remediation:
- Mock at the boundary - never let test code reach a real network
endpoint. Use Mock Service Worker (MSW), nock, WireMock, or
Playwright's
page.route(). - For tests that must hit a real service (smoke / contract tests), isolate them in a separate suite that doesn't gate the main CI.
- DNS-level: pin to specific resolvers in CI to avoid resolver variance.
Pattern 6: locator drift
UI tests pass when the page looks one way, fail when it shifts.
| Signal | What's happening |
|---|---|
| Fails after an unrelated CSS change | Selector matched by position rather than identity. |
selector matched 2 elements errors | Ambiguous selector now matches more than one node. |
| Fails only at certain viewports | Layout shifts cause mobile / desktop selectors to differ. |
Remediation:
- Use role-based selectors first (
page.getByRole('button', { name: 'Submit' })), thendata-testid, onlytext=/ CSS as a last resort. - For Playwright: enable
strict: trueso any ambiguous selector fails immediately rather than silently picking the first match. - For viewport-specific UIs: snapshot at every breakpoint via the
breakpoint matrix in
playwright-snapshots; visual signal exposes layout-shift flakes faster than text checks.
Pattern 7: environment variance
Tests pass on Linux CI, fail on macOS dev machines (or vice versa).
| Signal | What's happening |
|---|---|
| Fails only on a specific CI runner / OS | OS-specific path separator, line ending, or filesystem case sensitivity. |
| Snapshot tests fail with sub-pixel diffs across OS | OS font / anti-aliasing differences (see playwright-snapshots). |
Fails in tz configurations not set to UTC | Timezone-sensitive assertion. |
Remediation:
- Pin CI to one OS / one timezone (
TZ=UTC) for deterministic runs. - Run snapshot updates only in CI, never from a developer laptop
(per
playwright-snapshots). - For path-sensitive code, normalize with
path.posix.join()/node:path.
Pattern 8: randomness
Tests use random data without a controlled seed.
| Signal | What's happening |
|---|---|
| Failures don't reproduce on retry | Test data was randomized; the failing combination is gone. |
| Test asserts a property that holds "almost always" | Property-based test exposing a real edge case (this is good - fix the production bug). |
| Faker-generated data triggers a layout overflow | Random string longer than the assertion expected. |
Remediation:
- Seed every random source:
Math.randomviaseedrandom, faker viafaker.seed(N), property-based testing viafc.assert(prop, { seed }). - For property-based failures, don't mark them as flake - copy the failing seed into a regression test.
- Persist the seed used in each CI run as a build artifact so a flake can be replayed.
Triage decision tree
Test fails ~50% of runs?
├── Yes → likely "shared parallel state" or "test ordering"
└── No → fails ~5-20% of runs?
├── Yes → likely "async/timing" or "network"
└── No → fails only on specific OS / runner?
├── Yes → "environment variance"
└── No → fails after long suite duration?
├── Yes → "resource leaks"
└── No → fails after unrelated UI change?
├── Yes → "locator drift"
└── No → does the test use random data?
├── Yes → "randomness"
└── No → run a structured bisect
For systematic bisection, run a structured bisect that varies one axis at a time per the patterns above.
Code-level fixes
Once the pattern is named - by inspection above or by experiment via
flake-axis-bisection - apply the smallest change the pattern calls for
(a targeted edit, not a rewrite), then re-measure at a real depth: re-run
at an N chosen from the failure rate you are willing to ship, not the
screening N; a clean 0/20 does not prove the flake is gone
(flake-axis-bisection). Quarantine via flaky-test-quarantine if the
flake blocks the trunk while the fix is in review.
The per-pattern code fixes, grounded in Playwright, Cypress, MSW, and Faker official docs:
- Patterns 1-4 (async/timing, test ordering, shared parallel state, resource leaks): references/timing-and-isolation-fixes.md.
- Patterns 5-6 (network mock-at-the-boundary with
page.route()/ MSW; role-based locators withdata-testidfallback): references/network-and-locator-fixes.md. - Patterns 7-8 (pin
TZ=UTC, freeze the clock, normalize paths; seed every RNG and persist the seed): references/environment-and-randomness-fixes.md.
Worked example
A checkout test, tests/checkout.spec.ts:42, fails about 15% of runs in
CI and always passes locally.
- Classify.
flake-axis-bisectionimplicates the network-latency axis, and reading the source shows the assertion is gated on a fixedpage.waitForTimeout(2000), not on the response. That is Pattern 1 (async / timing): the sleep is shorter than the slowest CI response. - Apply the fix. Replace the fixed sleep with a web-first assertion that retries until the condition holds:
// Before - the 2s sleep races a variable-latency XHR
await page.getByRole('button', { name: 'Place order' }).click();
await page.waitForTimeout(2000);
expect(await page.getByText('Order confirmed').isVisible()).toBe(true);
// After - retries until the confirmation renders or the timeout expires
await page.getByRole('button', { name: 'Place order' }).click();
await expect(page.getByText('Order confirmed')).toBeVisible();
- Re-measure and ship. The team's tolerance is 1%, so re-run at
N=300 (
flake-axis-bisectionStep 2). A clean 0/300 bounds the rate at roughly 1%; a clean 0/20 would have proved nothing. No quarantine was needed - the fix landed inside the PR the flake was blocking.
Quick-reference: pattern to fix
| Pattern | Key fix | Primary API |
|---|---|---|
| async / timing | Replace sleep with auto-wait assertion | await expect(loc).toBeVisible() |
| test ordering | Move setup to beforeEach; roll back DB per test | test.beforeEach / test.afterEach |
| shared parallel state | Per-worker schema / dir / port via workerIndex | testInfo.workerIndex |
| resource leaks | browser.close() in afterAll with try/finally | test.afterAll + try/finally |
| network | Mock at boundary; never reach real endpoints | page.route() / MSW |
| locator drift | Role-based locators; data-testid fallback | getByRole() |
| environment variance | Pin TZ=UTC; freeze clock; normalize paths | page.clock.install() |
| randomness | Seed every RNG; persist seed in CI log | faker.seed(N) |
Per-fix citations live in the three reference files above.
References
- google-flaky - Google Testing Blog overview.
- google-causes - Google's correlation analysis on ~4.2M tests; "test size correlates with flakiness rate."
flaky-test-quarantine- workflow that uses this catalog during triage.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 318k | Updated | Claude CodeCodex | Unverified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 318k | Updated | Claude CodeCodex | Unverified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 318k | Updated | Claude CodeCodex | Unverified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |