heymegabyte/agent-skills / 07-quality-and-verification
quality-and-verification
Unverified5-level verification pyramid: static→unit→Playwright E2E (homepage-first, 6bp)→AI visual→post-deploy. 8-check quality gate. Multi-agent testing (functional/security/a11y/performance). Playwright v1.56+ AI agents (Planner/Generator/Healer, v1.63 current). WCAG 2.2 AA via axe-core v4.13. Percy+Chromatic visual regression. ADA Title II 2027/2028 deadlines.
visual-regression.md
View on GitHub---
name: "Visual Regression Testing"
version: "2.0.0"
updated: "2026-04-23"
description: "Three-tier visual regression: Percy AI Visual Review Agent (CI, 3x review reduction, 40% false positive filtering) + Chromatic (component-level via Storybook+Playwright) + pixelmatch (local/lightweight). Baseline capture → compare → AI-classify diffs."
---
# Visual Regression Testing
## Tool Hierarchy
1. **Percy AI Visual Review Agent (2025+)** — CI integration, 3x review reduction, 40% false positive filtering, OCR for text-shift elimination. 5K screenshots/month free tier. Best for: full-page regression in CI.
2. **Chromatic** — Pixel-perfect detection built around Storybook, now supports Playwright. Best for: component-level regression, design system changes.
3. **pixelmatch** — Lightweight zero-dependency PNG diffing for local CI. ~150 lines, no native bindings. Best for: fast local checks, custom thresholds.
**Best combo:** Percy (CI integration) + Chromatic (component-level) together. pixelmatch for local dev loop.
## Baseline Workflow
1. Capture screenshots at 6 breakpoints (see `visual-inspection-loop.md`)
2. First run: save as baselines → `screenshots/baselines/{route}_{breakpoint}.png`
3. Subsequent runs: capture → compare against baselines → diff images
4. Review diffs → accept changes (update baselines) or fix regressions
## Implementation
```typescript
import { PNG } from 'pngjs';
import pixelmatch from 'pixelmatch';
import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'fs';
interface DiffResult {
route: string;
breakpoint: string;
diffPixels: number;
totalPixels: number;
diffPercent: number;
passed: boolean;
diffImagePath?: string;
}
const THRESHOLD = 0.1; // pixelmatch color threshold (0-1, lower = stricter)
const MAX_DIFF_PERCENT = 0.5; // fail if >0.5% pixels differ
function compareScreenshots(
baselinePath: string,
currentPath: string,
diffOutputPath: string,
): DiffResult {
const baseline = PNG.sync.read(readFileSync(baselinePath));
const current = PNG.sync.read(readFileSync(currentPath));
// Handle size mismatches — immediate fail
if (baseline.width !== current.width || baseline.height !== current.height) {
return {
route: '', breakpoint: '', diffPixels: -1, totalPixels: -1,
diffPercent: 100, passed: false,
};
}
const { width, height } = baseline;
const diff = new PNG({ width, height });
const diffPixels = pixelmatch(
baseline.data, current.data, diff.data,
width, height,
{ threshold: THRESHOLD, includeAA: false },
);
writeFileSync(diffOutputPath, PNG.sync.write(diff));
const totalPixels = width * height;
const diffPercent = (diffPixels / totalPixels) * 100;
return {
route: '', breakpoint: '', diffPixels, totalPixels,
diffPercent: Math.round(diffPercent * 100) / 100,
passed: diffPercent <= MAX_DIFF_PERCENT,
diffImagePath: diffOutputPath,
};
}
```
## Playwright Integration
```typescript
// e2e/visual-regression.spec.ts
import { test, expect } from '@playwright/test';
import { compareScreenshots, captureAllScreenshots, BREAKPOINTS } from '../utils/visual-regression';
const PROD_URL = process.env.PROD_URL!;
const BASELINE_DIR = 'screenshots/baselines';
const CURRENT_DIR = 'screenshots/current';
const DIFF_DIR = 'screenshots/diffs';
const UPDATE_BASELINES = process.env.UPDATE_BASELINES === 'true';
const ROUTES = ['/', '/about', '/contact', '/pricing'];
test.describe('Visual Regression', () => {
test('All pages match baselines at all breakpoints', async ({ page }) => {
const failures: string[] = [];
for (const route of ROUTES) {
await page.goto(`${PROD_URL}${route}`);
await page.waitForLoadState('networkidle');
for (const bp of BREAKPOINTS) {
await page.setViewportSize({ width: bp.width, height: bp.height });
await page.waitForTimeout(300);
const name = `${route.replace(/\//g, '_') || '_index'}_${bp.name}`;
const currentPath = `${CURRENT_DIR}/${name}.png`;
const baselinePath = `${BASELINE_DIR}/${name}.png`;
const diffPath = `${DIFF_DIR}/${name}_diff.png`;
await page.screenshot({ path: currentPath, fullPage: true });
if (UPDATE_BASELINES) {
// Copy current to baseline
const { copyFileSync } = await import('fs');
copyFileSync(currentPath, baselinePath);
continue;
}
if (!existsSync(baselinePath)) {
failures.push(`Missing baseline: ${name}`);
continue;
}
const result = compareScreenshots(baselinePath, currentPath, diffPath);
if (!result.passed) {
failures.push(`${name}: ${result.diffPercent}% diff (${result.diffPixels}px)`);
}
}
}
expect(failures, `Visual regressions:\n${failures.join('\n')}`).toEqual([]);
});
});
```
## Threshold Configuration
```typescript
// Per-route thresholds for pages with dynamic content
const ROUTE_THRESHOLDS: Record<string, number> = {
'/': 0.5, // Homepage — tight
'/blog': 2.0, // Blog — dynamic content, looser
'/dashboard': 3.0, // Dashboard — data changes
};
// Per-element ignore regions (mask dynamic areas)
const IGNORE_REGIONS: Record<string, Array<{ x: number; y: number; w: number; h: number }>> = {
'/': [{ x: 0, y: 0, w: 300, h: 50 }], // Mask timestamp header
};
```
## Visual QA Agent Integration
`visual-regression` feeds into the `visual-qa` agent (07/visual-inspection-loop.md):
1. Run pixelmatch diff → any failures?
2. Yes → send diff images + current screenshots to GPT Image 2 vision
3. AI classifies: intentional change | regression | flaky (animation/timing)
4. Intentional → auto-update baselines
5. Regression → generate fix suggestions
6. Flaky → increase threshold or add ignore region
## CI Integration (GitHub Actions)
```yaml
# .github/workflows/visual-regression.yml
name: Visual Regression
on: [pull_request]
jobs:
visual-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: '24' }
- run: bun install
- run: bunx playwright install --with-deps chromium
- run: bun run test:visual
env:
PROD_URL: ${{ vars.STAGING_URL }}
- uses: actions/upload-artifact@v4
if: failure()
with:
name: visual-diffs
path: screenshots/diffs/
retention-days: 7
```
## Commands
```bash
# Capture new baselines
UPDATE_BASELINES=true bun run test:visual
# Run visual regression tests
bun run test:visual
# Compare specific route
ROUTES="/" bun run test:visual
```
## Anti-Patterns
### Never
- Screenshot in headful mode (font rendering differs)
- Compare across OS (Linux vs macOS renders differ)
- Skip `networkidle` (loading spinners cause false diffs)
- Use exact match (`threshold:0` catches subpixel antialiasing)
### Always
- Run in Docker / CI for consistent rendering
- Mask dynamic regions (dates, avatars)
- Store baselines in git (LFS for large repos)
- Review diff images before accepting
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |