Skip to content

luthersystems/elps / .claude/skills/benchmark

benchmark

Verified

/benchmark — Performance Benchmarking Skill

Runs before/after benchmark comparisons using benchstat to measure the performance impact of changes.

Trigger

Use when asked to benchmark changes, check for performance regressions, or optimize code.

Workflow

1. Find Benchmarks

go test -list 'Benchmark' ./... 2>/dev/null | grep -E '^Benchmark'

This lists all available benchmark functions across the codebase.

For an interpreter change, start with the whole-program benchmarks: BenchmarkWorkload (lisp/lisplib/workload_bench_test.go: json, sortedmap, string, recursion, template) and BenchmarkCorpus (elpstest/). A gain claimed for a change should show there; micro-benchmarks explain it.

2. Measure base and head interleaved

Build one test binary per arm and alternate them, so machine noise lands on both arms alike:

git worktree add /tmp/elps-base origin/main
(cd /tmp/elps-base && go test -c -o /tmp/base.test ./lisp/lisplib/)
go test -c -o /tmp/head.test ./lisp/lisplib/
cd lisp/lisplib
for i in $(seq 10); do
  /tmp/base.test -test.run='^$' -test.bench='^BenchmarkWorkload' -test.benchmem >> /tmp/bench-before.txt
  /tmp/head.test -test.run='^$' -test.bench='^BenchmarkWorkload' -test.benchmem >> /tmp/bench-after.txt
done

Swap the package and -test.bench pattern for a package-specific benchmark. On a shared or loaded machine, allocs/op and B/op are deterministic and are the primary evidence; treat sec/op as indicative, and run make bench-burnin first.

3. Compare with benchstat

benchstat base=/tmp/bench-before.txt pr=/tmp/bench-after.txt

4. Interpret Results

Report the results with focus on:

  • Time: sec/op — lower is better
  • Memory: B/op — lower is better
  • Allocations: allocs/op — lower is better
  • Statistical significance: benchstat shows ~ for no significant change, +/- for changes with confidence intervals

Regression threshold: CI's gate fails a statistically significant bad-direction move of 15% or more on timing and 5% or more on allocations (see "CI Integration" below). Report any significant regression, and investigate those at or above the gate before pushing.

5. Report

Format results as a clear summary:

## Benchmark Results

| Benchmark | Before | After | Change |
|-----------|--------|-------|--------|
| BenchmarkEval | 1.23 µs/op | 1.15 µs/op | -6.5% |
| BenchmarkParse | 456 ns/op | 460 ns/op | ~ (no change) |

No regressions detected. Memory allocations reduced by 12%.

CI Integration

The repo has a benchmark CI workflow (.github/workflows/benchmark.yml) that automatically runs benchstat comparisons on PRs. It posts results as a PR comment.

The comparison is adjudicated by cmd/benchgate (a Go tool built on golang.org/x/perf/benchfmt + benchmath; substrate runs the same binary), which fails the PR on a significant bad-direction move at or above the threshold for that metric class (15% for timing, 5% for allocations — set in benchmark.yml).

When the gate fires on a regression you mean to accept

Do not raise either threshold, and do not skip the gate. Those are per-metric-class noise floors; moving one to accept a single benchmark blinds every other benchmark in the repo to the same magnitude of move. Add a waiver to scripts/benchstat-waivers.txt instead:

pkg | benchmark | metric | ceiling | expires | issue | reason

It covers exactly one package, one benchmark and one metric column; it records a ceiling, so the row fails again if the regression grows; it must name a tracking issue and it expires. A waived row is still measured, still printed in the job log, and still shown in the PR comment marked WAIVED — the waiver changes the verdict, never the visibility. The format is documented at the top of that file, and scripts/ci-gates-test.sh covers the mechanism.

Run the gate locally against a saved comparison before pushing:

benchstat base=/tmp/bench-before.txt pr=/tmp/bench-after.txt > /tmp/benchstat.txt
make bench-gate BENCHSTAT_OUT=/tmp/benchstat.txt              # as CI will judge it
# or drive the binary directly (this is what `make bench-gate` runs):
go run ./cmd/benchgate -waivers-default scripts/benchstat-waivers.txt /tmp/benchstat.txt
BENCH_WAIVERS= go run ./cmd/benchgate /tmp/benchstat.txt      # with waivers off

benchgate can also adjudicate the two raw go test -bench arms directly, with no benchstat binary in the loop (make bench-gate-arms BENCH_BASE=… BENCH_HEAD=…).

CI measures at GOMAXPROCS=1

benchmark.yml pins GOMAXPROCS: "1" (issue #767). At 2 Ps the concurrent GC's mark workers ran on the second P, and allocation-heavy rows measured how the shared runner scheduled that thread: ±7-20% on identical code, enough to cross the 15% gate on PRs that never touched them. One P keeps GC cost in sec/op and makes it deterministic. So CI rows carry no -N suffix, and to reproduce a CI verdict locally measure the same way:

GOMAXPROCS=1 go test -run='^$' -bench=. -skip='Parallel$' -benchmem -benchtime=100ms -count=10 ./lisp/... | tee /tmp/bench.txt

The mechanism (concurrent GC on the second P) is inferred from one controlled experiment plus the PR's own run, not observed with perf counters.

RunParallel benchmarks must be named *Parallel (ci-gates-test enforces it). The gated job skips them; the benchmark-parallel job measures them at GOMAXPROCS=4 and reports benchgate's verdict without failing on it, because on identical code they spread ~30% per arm.

benchgate strips the -N suffix only as told by -gomaxprocs / BENCH_GOMAXPROCS (bench-compare.sh passes the job's GOMAXPROCS): at 1 a benchmark's own trailing -2 is kept, at N>1 exactly -N is stripped, and unset keeps the old strip-any-trailing-number behaviour.

Before you measure: is the machine fit? (make bench-burnin)

make bench-burnin      # ~half a second; exit 0 fit, exit 3 re-measure elsewhere

A fixed, code-independent loop run seven times, requiring the samples to agree to within ±10%. A machine that cannot reproduce a fixed loop cannot resolve a 10% gate on anything else either, and half an hour of benchmarking on one produces numbers that read like findings. Run it first — on a laptop with a browser open as much as on CI.

UNMEASURABLE and exit 3

The gate has a fourth verdict (issue #542). A timing row whose own confidence interval is at or above the fitness ceiling (-variance-ceiling, default ±30%) is UNMEASURABLE: its delta is printed and adjudicated in neither direction. If such a row moved at or above the gate, the run exits 3 (RUNNER-UNFIT) — it found no regression and certified nothing, so the answer is to re-run, not to read the diff. An unmeasurable row below its gate is a warning only and changes no exit code.

Exit 1 still wins over exit 3: a regression measured on a row that could be measured is a finding regardless of what else in the table was unmeasurable. And the ceiling can only ever withhold a finding — it never turns a passing run into a failing one.

The shape it exists for: an arm measuring itself at ±71% against the other arm's ±3%, on code that did not change, adjudicated as +83% REGRESSION. Preserved as cmd/benchgate/testdata/elps/benchstat-runner-unfit-542.txt.

Checklist

  • Base and head measured interleaved, with the same flags
  • Whole-program benchmarks (BenchmarkWorkload) checked for an interpreter change
  • benchstat comparison run
  • Results reported with clear formatting
  • Significant regressions reported; any at or above the CI gate (15% timing / 5% allocs) investigated or waived
  • Temp files cleaned up when done
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 196kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 193kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 193kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 193kUpdated Claude CodeCodexVerifiedInstall