Skip to content

JPeetz/agent-skills / skills/qa-testing/performance-testing

performance-testing

Unverified

Use this skill when planning, executing, or analyzing performance tests — load testing with k6, stress testing, soak/endurance testing, spike testing, Lighthouse CI for Core Web Vitals, performance budgets as CI gates, bottleneck identification, and capacity planning. Covers k6 scripting, thresholds, scenarios, browser-level metrics, and Lighthouse audit configuration.

Performance Testing

Performance work is measurement, not guesswork. Every test must answer a specific question ("can we hold 2× Black Friday traffic?", "did this release regress LCP?"), have a pass/fail criterion defined before it runs, and produce numbers you can compare against a baseline.

1. Choose the right test type

TypeQuestion answeredLoad shapeDurationPass criterion
SmokeDoes the system work under minimal load?1–5 VUs1–2 minNo errors, sane latency
LoadDo we meet SLOs at expected peak?Ramp to expected peak, hold15–60 minp95/p99 within SLO, error rate < 1%
StressWhere does it break, and how?Step or ramp beyond peak until failure30–60 minFind breaking point; graceful degradation
Soak / enduranceWhat breaks over time?60–80% of peak, held2–24 hNo memory growth, no latency drift, no connection leaks
SpikeCan we survive sudden bursts?Instant jump 1→10×, hold, drop5–15 minRecovery time, error burst bounded
ScalabilityDoes throughput scale linearly with resources?Repeat load test at N, 2N, 4N capacityVariesThroughput/cost curve, identify knee
Browser / RUMWhat does the user actually experience?Real or synthetic browsersContinuousCore Web Vitals "good" at p75

Rules of thumb:

  • Always start with a smoke test. A failing smoke test invalidates everything downstream.
  • Never run stress or spike tests against production without an approved window, a kill switch, and someone watching dashboards.
  • Test one variable at a time. Changing script, data, and infrastructure together makes results uninterpretable.
  • Model think time and realistic user mixes; hammering an endpoint with zero pause measures a throughput ceiling nobody experiences.

2. Define goals before writing scripts

Write these down in the test plan (or a comment block at the top of the script):

Goal:      Validate checkout API at 3× current peak (900 rps) for 30 min
SLO:       p95 < 400 ms, p99 < 900 ms, error rate < 0.5%
Baseline:  release 2026-11-01, 300 rps, p95 210 ms
Env:       staging, same instance class as prod, seeded with 1M orders
Data:      10k unique users, cart with 1–5 items, 5% payment failures injected
Abort if:  error rate > 10% for 60 s, or DB CPU > 95%

If you cannot state the SLO numerically, you are not ready to test.

3. k6 fundamentals

3.1 Minimal script

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  vus: 10,
  duration: '30s',
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<500', 'p(99)<1000'],
  },
};

export default function () {
  const res = http.get('https://staging.example.com/api/products');
  check(res, {
    'status is 200': (r) => r.status === 200,
    'body is JSON': (r) => r.headers['Content-Type'].includes('application/json'),
    'has items': (r) => r.json('items').length > 0,
  });
  sleep(1);
}

3.2 Options that matter

  • stages — ramp profile for ramping-vus / ramping-arrival-rate.
  • thresholds — SLO assertions; a breached threshold exits non-zero (your CI gate).
  • scenarios — multiple independent workloads in one run (browse + checkout + search).
  • discardResponseBodies: true — big memory win when you only need status/latency.
  • summaryTrendStats — control which percentiles appear in the summary.
  • tags / group() — slice metrics by endpoint or user journey.
  • noConnectionReuse: false — keep default keep-alive; disabling it changes results massively.
  • userAgent, insecureSkipTLSVerify — only when the environment requires it.

3.3 Thresholds (the CI gate)

export const options = {
  thresholds: {
    // Global
    http_req_failed:   ['rate<0.01'],
    http_req_duration: ['p(95)<400', 'p(99)<900'],
    // Per-tag
    'http_req_duration{endpoint:checkout}': ['p(95)<800'],
    'http_req_duration{endpoint:search}':   ['p(95)<250'],
    // Custom metrics
    'http_reqs': ['rate>500'],
    'order_success_rate': ['rate>0.995'],
    // Abort early to save time and money
    checks: [{ threshold: 'rate>0.95', abortOnFail: true, delayAbortEval: '10s' }],
  },
};

abortOnFail prevents a 2-hour soak from burning time after a catastrophic break.

3.4 Scenarios and executors

ExecutorUse forKey params
constant-vusSteady load, fixed concurrencyvus, duration
ramping-vusLoad with rampsstages
constant-arrival-rateFixed request rate independent of latencyrate, timeUnit, preAllocatedVUs, maxVUs
ramping-arrival-rateArrival-rate ramps (better for capacity curves)startRate, stages
per-vu-iterationsFixed iterations per VUvus, iterations
shared-iterationsFixed total iterations across VUsvus, iterations
externally-controlledDrive VUs/rate at runtime via RESTmaxVUs, duration

Arrival-rate executors are usually correct for capacity testing. Constant VUs conflate concurrency with throughput: as latency degrades, a constant-vus test silently sends fewer requests, hiding the problem.

export const options = {
  scenarios: {
    browse: {
      executor: 'ramping-arrival-rate',
      startRate: 20, timeUnit: '1s',
      preAllocatedVUs: 100, maxVUs: 800,
      stages: [
        { target: 100, duration: '2m' },  // ramp
        { target: 100, duration: '10m' }, // hold
        { target: 0,   duration: '1m' },  // ramp down
      ],
      exec: 'browse',
      tags: { journey: 'browse' },
    },
    checkout: {
      executor: 'constant-arrival-rate',
      rate: 10, timeUnit: '1s',
      duration: '13m',
      preAllocatedVUs: 50, maxVUs: 200,
      exec: 'checkout',
      startTime: '2m',
      tags: { journey: 'checkout' },
    },
  },
};

export function browse() { /* ... */ }
export function checkout() { /* ... */ }

3.5 Custom metrics

import { Trend, Rate, Counter, Gauge } from 'k6/metrics';

const cartValue   = new Trend('cart_value_usd', true);  // true = unit ms-style percentiles ok
const loginOk     = new Rate('login_success_rate');
const orders      = new Counter('orders_created');
const queueDepth  = new Gauge('queue_depth');

export default function () {
  const t0 = Date.now();
  const res = http.post(`${__ENV.BASE_URL}/api/login`, payload, params);
  loginOk.add(res.status === 200);
  if (res.status === 200) orders.add(1);
  cartValue.add(res.json('cart.total') || 0);
}

Custom metrics can be thresholded exactly like built-ins — this is how you gate on business outcomes (orders/min, login success) rather than only HTTP mechanics.

3.6 Data, auth, and realism

import { SharedArray } from 'k6/data';
import papaparse from 'https://jslib.k6.io/papaparse/5.1.1/index.js';
import { randomItem } from 'https://jslib.k6.io/k6-utils/1.4.0/index.js';

const users = new SharedArray('users', () =>
  papaparse.parse(open('./data/users.csv'), { header: true }).data);

export function setup() {
  const res = http.post(`${__ENV.BASE_URL}/api/auth/token`, JSON.stringify({
    client_id: __ENV.CLIENT_ID, client_secret: __ENV.CLIENT_SECRET,
  }), { headers: { 'Content-Type': 'application/json' } });
  return { token: res.json('access_token') };
}

export default function (data) {
  const user = randomItem(users);
  const res = http.get(`${__ENV.BASE_URL}/api/orders`, {
    headers: { Authorization: `Bearer ${data.token}` },
    tags: { endpoint: 'orders' },
  });
  check(res, { 'ok': (r) => r.status === 200 });
}
  • SharedArray keeps one copy of large datasets across VUs. A plain array is copied per VU and will exhaust memory at a few hundred VUs.
  • Use setup() for expensive one-time auth; never log in every iteration unless you are explicitly testing the auth path.
  • Seed data deterministically (__VU, __ITER or a seeded RNG) so failures are reproducible.
  • Use environment variables (__ENV) for URLs, credentials, and profile switches. Never commit secrets.

3.7 Checks vs thresholds

check() records pass/fail but does not fail the test. Only thresholds fail a run. Use checks for diagnostics, and mirror anything important into a threshold.

4. Running k6

# Local
k6 run --vus 50 --duration 2m script.js

# Named environment + summary export
k6 run -e BASE_URL=https://staging.example.com \
       --summary-export=results/summary.json script.js

# Stream raw metrics to Prometheus / InfluxDB / Grafana Cloud
K6_PROMETHEUS_RW_SERVER_URL=http://localhost:9090/api/v1/write \
  k6 run -o experimental-prometheus-rw script.js

# Machine-readable output for regression diffing
k6 run --out json=results/raw.json script.js

# Distributed / cloud (optional)
k6 cloud run script.js

Keep a baseline artifact (summary.json) per release and diff p95/p99 and throughput. Absolute numbers across machines are meaningless; deltas against the same rig are not.

5. Lighthouse CI and Core Web Vitals

5.1 Core Web Vitals thresholds (p75, mobile)

MetricGoodNeeds improvementPoor
LCP (Largest Contentful Paint)≤ 2.5 s2.5–4.0 s> 4.0 s
INP (Interaction to Next Paint)≤ 200 ms200–500 ms> 500 ms
CLS (Cumulative Layout Shift)≤ 0.10.1–0.25> 0.25
FCP (supporting)≤ 1.8 s1.8–3.0 s> 3.0 s
TTFB (supporting)≤ 800 ms0.8–1.8 s> 1.8 s

Notes:

  • INP replaced FID in March 2024. Do not gate on FID.
  • Lab tools (Lighthouse) cannot measure INP faithfully; they approximate with TBT (Total Blocking Time, target < 200 ms). Gate lab INP/TBT and validate INP with RUM.
  • Field data (CrUX / RUM) is truth; lab data is the fast proxy. Use both.

5.2 lighthouserc.js

module.exports = {
  ci: {
    collect: {
      url: ['https://staging.example.com/', 'https://staging.example.com/pricing'],
      numberOfRuns: 3,                  // median smooths noise; never trust 1 run
      settings: {
        preset: 'desktop',              // or omit for mobile (mobile is the default gate)
        throttlingMethod: 'simulate',
        onlyCategories: ['performance', 'accessibility', 'best-practices', 'seo'],
      },
      startServerCommand: 'npm run start',
      startServerReadyPattern: 'ready on',
    },
    assert: {
      assertions: {
        'categories:performance': ['error', { minScore: 0.9, aggregationMethod: 'median-run' }],
        'categories:accessibility': ['error', { minScore: 0.95 }],
        'largest-contentful-paint': ['error', { maxNumericValue: 2500 }],
        'cumulative-layout-shift': ['error', { maxNumericValue: 0.1 }],
        'total-blocking-time': ['error', { maxNumericValue: 200 }],
        'first-contentful-paint': ['warn', { maxNumericValue: 1800 }],
        'server-response-time': ['warn', { maxNumericValue: 800 }],
        'uses-responsive-images': 'warn',
        'unused-javascript': ['warn', { maxNumericValue: 40000 }],
        'budgetsFile': 'budgets.json',
      },
    },
    upload: {
      target: 'temporary-public-storage',
      // or: target: 'lhci', serverBaseUrl: 'https://lhci.example.com'
    },
  },
};

Run locally: npx @lhci/cli autorun. Failures exit non-zero — that is the CI gate.

5.3 Performance budgets file

[
  {
    "path": "/*",
    "resourceSizes": [
      { "resourceType": "script", "budget": 170 },
      { "resourceType": "stylesheet", "budget": 50 },
      { "resourceType": "image", "budget": 300 },
      { "resourceType": "font", "budget": 100 },
      { "resourceType": "total", "budget": 700 }
    ],
    "resourceCounts": [
      { "resourceType": "third-party", "budget": 5 },
      { "resourceType": "font", "budget": 2 }
    ],
    "timingBudget": [
      { "metric": "interactive", "budget": 3500 },
      { "metric": "first-contentful-paint", "budget": 1800 }
    ]
  }
]

Budgets are in KiB. Set them from today's measured baseline minus a stretch target, not from an aspirational number nobody meets — a permanently red gate gets ignored.

5.4 GitHub Actions

name: performance
on: [pull_request]
jobs:
  lighthouse:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: npm }
      - run: npm ci && npm run build
      - name: Lighthouse CI
        run: npx @lhci/cli autorun
        env:
          LHCI_GITHUB_APP_TOKEN: ${{ secrets.LHCI_GITHUB_APP_TOKEN }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: lighthouse-report
          path: .lighthouseci/
  k6:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: grafana/setup-k6-action@v1
      - run: k6 run --summary-export=summary.json tests/load/smoke.js
        env:
          BASE_URL: ${{ secrets.STAGING_URL }}

Run smoke-level k6 on PRs (1–2 min) and full load/soak on a schedule against staging. Nobody wants a 30-minute gate on every push.

5.5 Measuring Web Vitals in the browser

// RUM: web-vitals library
import { onLCP, onINP, onCLS, onFCP, onTTFB } from 'web-vitals';
const send = (m) => navigator.sendBeacon('/rum', JSON.stringify({
  name: m.name, value: m.value, id: m.id, rating: m.rating, path: location.pathname,
}));
onLCP(send); onINP(send); onCLS(send); onFCP(send); onTTFB(send);

k6 browser module for synthetic user journeys with real rendering:

import { browser } from 'k6/browser';
import { check } from 'k6';

export const options = {
  scenarios: {
    ui: { executor: 'constant-vus', vus: 5, duration: '2m',
          options: { browser: { type: 'chromium' } } },
  },
  thresholds: { browser_web_vital_lcp: ['p(95)<2500'] },
};

export default async function () {
  const page = await browser.newPage();
  try {
    await page.goto('https://staging.example.com/');
    await page.getByRole('button', { name: 'Add to cart' }).click();
    check(page, { 'cart badge visible': async (p) => !!(await p.locator('.cart-count').textContent()) });
  } finally {
    await page.close();
  }
}

Browser tests are expensive (each VU ≈ one Chromium). Use a handful of VUs for journey validation, not for load.

6. Bottleneck identification

Work top-down: is time spent in the client, the network, the app, the DB, or a dependency? Never optimize without evidence from one of these sources.

Frontend

  • LCP element is slow → check server TTFB, render-blocking CSS/JS, unoptimized hero image (use srcset, AVIF/WebP, fetchpriority="high", preload).
  • High TBT/INP → long tasks from hydration or third-party scripts; code-split, defer, and break up long tasks (scheduler.yield, setTimeout chunking).
  • CLS → missing image dimensions, injected banners/ads, late-loading fonts (font-display, size-adjust metrics).

Backend

  • p50 fine, p99 bad → queueing, lock contention, GC pauses, cold starts, noisy neighbors.
  • CPU saturated with low throughput → N+1 queries, synchronous serialization, blocking I/O in an async path.
  • Thread/worker pool exhaustion → check pool sizes, connection limits, and timeouts first.
  • Look for the knee: throughput rises linearly then flattens while latency explodes. That inflection, not the failure point, is your real capacity.

Database

  • Enable slow query log; inspect EXPLAIN (ANALYZE, BUFFERS) for seq scans on large tables.
  • Missing/incorrect indexes show as high rows-examined-to-rows-returned ratios.
  • Connection pool saturation: requests waiting on pg_stat_activity or an exhausted pool manifests as latency with idle CPU.
  • Lock waits and long transactions turn a read-heavy test into a serial queue.

Network / infrastructure

  • TTFB high but server app time low → DNS, TLS handshake, proxy/LB, CDN miss.
  • Throughput capped exactly at a round number → load balancer, bandwidth limit, or connection limit, not application logic.
  • Errors clustered at ramp changes → autoscaling lag, readiness probe failures, rate limiting.

Diagnostic discipline: correlate k6 timeline (ramp start, hold start) with Grafana/APM graphs at the same timestamps. The causal step change is the bottleneck.

7. Capacity planning

  1. Measure unit cost — requests/sec per instance at the knee, with p95 ≤ SLO.
  2. Project demand — peak RPS now, growth rate, and known events (launch, campaign, Black Friday). Use p95 of the last 30 days × growth × event multiplier.
  3. Compute headroom — required_instances = peak_rps / per_instance_rps × (1 + headroom). Standard headroom is 30–50%; N+1 redundancy is on top of that, not inside it.
  4. Verify at scale — run a load test at the projected instance count, not just the current one. Scaling is rarely linear (shared DB, cache, locks).
  5. Define autoscaling triggers from measured signals (RPS per pod, p95 latency, queue depth), not CPU alone, and account for scale-up latency in the spike test.
  6. Record it — a capacity table (instance class, RPS at SLO, headroom, cost per 1k req) that product and finance can read.

Re-derive the table after every significant architecture change. Stale capacity numbers cause outages more often than bad code.

8. Anti-patterns and fixes

Anti-patternWhy it misleadsFix
Testing only with 1–10 VUsHides queueing and connection pool limitsRamp past expected peak until the knee
No think timeUnrealistic throughput ceiling, no cache realismsleep() 0.5–5 s per journey step
Identical data for every VUCache hits everywhere; DB never stressedUnique IDs from SharedArray, randomized
Ignoring http_req_failed500s return fast and look "performant"Threshold error rate; check body content
Judging by average latencyAverages hide the p99 that users feelGate on p95/p99, report distribution
One Lighthouse run±10 score noise from a single runnumberOfRuns: 3, median aggregation
Desktop-only LighthouseMobile is the slower, real-world caseGate mobile; test desktop separately
Soak test on a laptopThermal throttling and shared CPU pollute resultsDedicated load generator, stable network
Load generator and target on one hostYou measure the generator's limitSeparate hosts; watch generator CPU
Threshold set after seeing resultsGuarantees a green build, teaches nothingDefine SLO first, then tune the system
Testing against a shared staging DBOther teams' writes skew resultsIsolated, seeded environment
No baselineCannot distinguish regression from noiseStore summary.json per release, diff it

9. Reporting

Every performance report should contain: test type and load shape, environment and data, SLO vs measured (p50/p95/p99, error rate, throughput), baseline delta, bottleneck evidence (graph or query plan), verdict (pass/fail against thresholds), and a prioritized recommendation list with expected impact. A report without a verdict is a dashboard, not a test result.

10. Bundled files

  • scripts/k6-helpers.js — reusable stages, thresholds, auth setup, and data helpers.
  • references/sources.md — authoritative documentation links (k6, Lighthouse, Web Vitals).
  • evals/evals.json — trigger and behavior evals for this skill.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 464kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 464kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 464kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 192kUpdated Claude CodeCodexVerifiedInstall