Skip to content

heymegabyte/agent-skills / 07-quality-and-verification

quality-and-verification

Unverified

5-level verification pyramid: static→unit→Playwright E2E (homepage-first, 6bp)→AI visual→post-deploy. 8-check quality gate. Multi-agent testing (functional/security/a11y/performance). Playwright v1.56+ AI agents (Planner/Generator/Healer, v1.63 current). WCAG 2.2 AA via axe-core v4.13. Percy+Chromatic visual regression. ADA Title II 2027/2028 deadlines.

FileKindSize
SKILL.mdskill md5.5 KB
accessibility-gate.mdmarkdown5.8 KB
adversarial-testing.mdmarkdown5.5 KB
agentic-security.mdmarkdown3.4 KB
audio-video-sync.mdmarkdown1.7 KB
build-breaking-rules.mdmarkdown17.0 KB
chrome-and-browser-workflows.mdmarkdown5.5 KB
completeness-verification.mdmarkdown5.3 KB
computer-use-automation.mdmarkdown6.0 KB
contract-testing.mdmarkdown4.3 KB
e2e-accumulation.mdmarkdown1.6 KB
eval-driven-development.mdmarkdown6.3 KB
evidence-collection.mdmarkdown3.7 KB
llm-evals.mdmarkdown15.7 KB
performance-optimization.mdmarkdown4.8 KB
picovoice-eagle-biometric.mdmarkdown1.6 KB
security-hardening.mdmarkdown8.7 KB
semgrep-codebase-rules.mdmarkdown4.2 KB
slop-detection.mdmarkdown4.3 KB
spec-driven-development.mdmarkdown5.0 KB
stagehand-ai-fallback.mdmarkdown1.6 KB
stagehand-ai-testing.mdmarkdown7.7 KB
tdd-verification.mdmarkdown6.7 KB
testing-matrices.mdmarkdown5.3 KB
ui-completeness-sweep.mdmarkdown6.3 KB
visual-inspection-loop.mdmarkdown14.0 KB
visual-regression.mdmarkdown7.0 KB
wcag-2-2-2026.mdmarkdown3.2 KB

visual-regression.md

View on GitHub
---
name: "Visual Regression Testing"
version: "2.0.0"
updated: "2026-04-23"
description: "Three-tier visual regression: Percy AI Visual Review Agent (CI, 3x review reduction, 40% false positive filtering) + Chromatic (component-level via Storybook+Playwright) + pixelmatch (local/lightweight). Baseline capture → compare → AI-classify diffs."
---

# Visual Regression Testing

## Tool Hierarchy

1. **Percy AI Visual Review Agent (2025+)** — CI integration, 3x review reduction, 40% false positive filtering, OCR for text-shift elimination. 5K screenshots/month free tier. Best for: full-page regression in CI.
2. **Chromatic** — Pixel-perfect detection built around Storybook, now supports Playwright. Best for: component-level regression, design system changes.
3. **pixelmatch** — Lightweight zero-dependency PNG diffing for local CI. ~150 lines, no native bindings. Best for: fast local checks, custom thresholds.

**Best combo:** Percy (CI integration) + Chromatic (component-level) together. pixelmatch for local dev loop.

## Baseline Workflow

1. Capture screenshots at 6 breakpoints (see `visual-inspection-loop.md`)
2. First run: save as baselines → `screenshots/baselines/{route}_{breakpoint}.png`
3. Subsequent runs: capture → compare against baselines → diff images
4. Review diffs → accept changes (update baselines) or fix regressions

## Implementation

```typescript
import { PNG } from 'pngjs';
import pixelmatch from 'pixelmatch';
import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'fs';

interface DiffResult {
  route: string;
  breakpoint: string;
  diffPixels: number;
  totalPixels: number;
  diffPercent: number;
  passed: boolean;
  diffImagePath?: string;
}

const THRESHOLD = 0.1; // pixelmatch color threshold (0-1, lower = stricter)
const MAX_DIFF_PERCENT = 0.5; // fail if >0.5% pixels differ

function compareScreenshots(
  baselinePath: string,
  currentPath: string,
  diffOutputPath: string,
): DiffResult {
  const baseline = PNG.sync.read(readFileSync(baselinePath));
  const current = PNG.sync.read(readFileSync(currentPath));

  // Handle size mismatches — immediate fail
  if (baseline.width !== current.width || baseline.height !== current.height) {
    return {
      route: '', breakpoint: '', diffPixels: -1, totalPixels: -1,
      diffPercent: 100, passed: false,
    };
  }

  const { width, height } = baseline;
  const diff = new PNG({ width, height });

  const diffPixels = pixelmatch(
    baseline.data, current.data, diff.data,
    width, height,
    { threshold: THRESHOLD, includeAA: false },
  );

  writeFileSync(diffOutputPath, PNG.sync.write(diff));

  const totalPixels = width * height;
  const diffPercent = (diffPixels / totalPixels) * 100;

  return {
    route: '', breakpoint: '', diffPixels, totalPixels,
    diffPercent: Math.round(diffPercent * 100) / 100,
    passed: diffPercent <= MAX_DIFF_PERCENT,
    diffImagePath: diffOutputPath,
  };
}
```

## Playwright Integration

```typescript
// e2e/visual-regression.spec.ts
import { test, expect } from '@playwright/test';
import { compareScreenshots, captureAllScreenshots, BREAKPOINTS } from '../utils/visual-regression';

const PROD_URL = process.env.PROD_URL!;
const BASELINE_DIR = 'screenshots/baselines';
const CURRENT_DIR = 'screenshots/current';
const DIFF_DIR = 'screenshots/diffs';
const UPDATE_BASELINES = process.env.UPDATE_BASELINES === 'true';

const ROUTES = ['/', '/about', '/contact', '/pricing'];

test.describe('Visual Regression', () => {
  test('All pages match baselines at all breakpoints', async ({ page }) => {
    const failures: string[] = [];

    for (const route of ROUTES) {
      await page.goto(`${PROD_URL}${route}`);
      await page.waitForLoadState('networkidle');

      for (const bp of BREAKPOINTS) {
        await page.setViewportSize({ width: bp.width, height: bp.height });
        await page.waitForTimeout(300);

        const name = `${route.replace(/\//g, '_') || '_index'}_${bp.name}`;
        const currentPath = `${CURRENT_DIR}/${name}.png`;
        const baselinePath = `${BASELINE_DIR}/${name}.png`;
        const diffPath = `${DIFF_DIR}/${name}_diff.png`;

        await page.screenshot({ path: currentPath, fullPage: true });

        if (UPDATE_BASELINES) {
          // Copy current to baseline
          const { copyFileSync } = await import('fs');
          copyFileSync(currentPath, baselinePath);
          continue;
        }

        if (!existsSync(baselinePath)) {
          failures.push(`Missing baseline: ${name}`);
          continue;
        }

        const result = compareScreenshots(baselinePath, currentPath, diffPath);
        if (!result.passed) {
          failures.push(`${name}: ${result.diffPercent}% diff (${result.diffPixels}px)`);
        }
      }
    }

    expect(failures, `Visual regressions:\n${failures.join('\n')}`).toEqual([]);
  });
});
```

## Threshold Configuration

```typescript
// Per-route thresholds for pages with dynamic content
const ROUTE_THRESHOLDS: Record<string, number> = {
  '/': 0.5,          // Homepage — tight
  '/blog': 2.0,      // Blog — dynamic content, looser
  '/dashboard': 3.0, // Dashboard — data changes
};

// Per-element ignore regions (mask dynamic areas)
const IGNORE_REGIONS: Record<string, Array<{ x: number; y: number; w: number; h: number }>> = {
  '/': [{ x: 0, y: 0, w: 300, h: 50 }], // Mask timestamp header
};
```

## Visual QA Agent Integration

`visual-regression` feeds into the `visual-qa` agent (07/visual-inspection-loop.md):

1. Run pixelmatch diff → any failures?
2. Yes → send diff images + current screenshots to GPT Image 2 vision
3. AI classifies: intentional change | regression | flaky (animation/timing)
4. Intentional → auto-update baselines
5. Regression → generate fix suggestions
6. Flaky → increase threshold or add ignore region

## CI Integration (GitHub Actions)

```yaml
# .github/workflows/visual-regression.yml
name: Visual Regression
on: [pull_request]
jobs:
  visual-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '24' }
      - run: bun install
      - run: bunx playwright install --with-deps chromium
      - run: bun run test:visual
        env:
          PROD_URL: ${{ vars.STAGING_URL }}
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: visual-diffs
          path: screenshots/diffs/
          retention-days: 7
```

## Commands

```bash
# Capture new baselines
UPDATE_BASELINES=true bun run test:visual

# Run visual regression tests
bun run test:visual

# Compare specific route
ROUTES="/" bun run test:visual
```

## Anti-Patterns

### Never

- Screenshot in headful mode (font rendering differs)
- Compare across OS (Linux vs macOS renders differ)
- Skip `networkidle` (loading spinners cause false diffs)
- Use exact match (`threshold:0` catches subpixel antialiasing)

### Always

- Run in Docker / CI for consistent rendering
- Mask dynamic regions (dates, avatars)
- Store baselines in git (LFS for large repos)
- Review diff images before accepting
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall