luthersystems/elps / .claude/skills/benchmark
benchmark
Verified/benchmark — Performance Benchmarking Skill
Runs before/after benchmark comparisons using benchstat to measure the performance impact of changes.
Trigger
Use when asked to benchmark changes, check for performance regressions, or optimize code.
Workflow
1. Find Benchmarks
go test -list 'Benchmark' ./... 2>/dev/null | grep -E '^Benchmark'
This lists all available benchmark functions across the codebase.
For an interpreter change, start with the whole-program benchmarks:
BenchmarkWorkload (lisp/lisplib/workload_bench_test.go: json, sortedmap,
string, recursion, template) and BenchmarkCorpus (elpstest/). A gain
claimed for a change should show there; micro-benchmarks explain it.
2. Measure base and head interleaved
Build one test binary per arm and alternate them, so machine noise lands on both arms alike:
git worktree add /tmp/elps-base origin/main
(cd /tmp/elps-base && go test -c -o /tmp/base.test ./lisp/lisplib/)
go test -c -o /tmp/head.test ./lisp/lisplib/
cd lisp/lisplib
for i in $(seq 10); do
/tmp/base.test -test.run='^$' -test.bench='^BenchmarkWorkload' -test.benchmem >> /tmp/bench-before.txt
/tmp/head.test -test.run='^$' -test.bench='^BenchmarkWorkload' -test.benchmem >> /tmp/bench-after.txt
done
Swap the package and -test.bench pattern for a package-specific benchmark.
On a shared or loaded machine, allocs/op and B/op are deterministic and
are the primary evidence; treat sec/op as indicative, and run
make bench-burnin first.
3. Compare with benchstat
benchstat base=/tmp/bench-before.txt pr=/tmp/bench-after.txt
4. Interpret Results
Report the results with focus on:
- Time:
sec/op— lower is better - Memory:
B/op— lower is better - Allocations:
allocs/op— lower is better - Statistical significance: benchstat shows
~for no significant change,+/-for changes with confidence intervals
Regression threshold: CI's gate fails a statistically significant bad-direction move of 15% or more on timing and 5% or more on allocations (see "CI Integration" below). Report any significant regression, and investigate those at or above the gate before pushing.
5. Report
Format results as a clear summary:
## Benchmark Results
| Benchmark | Before | After | Change |
|-----------|--------|-------|--------|
| BenchmarkEval | 1.23 µs/op | 1.15 µs/op | -6.5% |
| BenchmarkParse | 456 ns/op | 460 ns/op | ~ (no change) |
No regressions detected. Memory allocations reduced by 12%.
CI Integration
The repo has a benchmark CI workflow (.github/workflows/benchmark.yml) that automatically runs benchstat comparisons on PRs. It posts results as a PR comment.
The comparison is adjudicated by cmd/benchgate (a Go tool built on
golang.org/x/perf/benchfmt + benchmath; substrate runs the same binary),
which fails the PR on a significant bad-direction move at or above the threshold
for that metric class (15% for timing, 5% for allocations — set in
benchmark.yml).
When the gate fires on a regression you mean to accept
Do not raise either threshold, and do not skip the gate. Those are
per-metric-class noise floors; moving one to accept a single benchmark blinds
every other benchmark in the repo to the same magnitude of move. Add a waiver
to scripts/benchstat-waivers.txt instead:
pkg | benchmark | metric | ceiling | expires | issue | reason
It covers exactly one package, one benchmark and one metric column; it records a
ceiling, so the row fails again if the regression grows; it must name a tracking
issue and it expires. A waived row is still measured, still printed in the job
log, and still shown in the PR comment marked WAIVED — the waiver changes the
verdict, never the visibility. The format is documented at the top of that file,
and scripts/ci-gates-test.sh covers the mechanism.
Run the gate locally against a saved comparison before pushing:
benchstat base=/tmp/bench-before.txt pr=/tmp/bench-after.txt > /tmp/benchstat.txt
make bench-gate BENCHSTAT_OUT=/tmp/benchstat.txt # as CI will judge it
# or drive the binary directly (this is what `make bench-gate` runs):
go run ./cmd/benchgate -waivers-default scripts/benchstat-waivers.txt /tmp/benchstat.txt
BENCH_WAIVERS= go run ./cmd/benchgate /tmp/benchstat.txt # with waivers off
benchgate can also adjudicate the two raw go test -bench arms directly, with no
benchstat binary in the loop (make bench-gate-arms BENCH_BASE=… BENCH_HEAD=…).
CI measures at GOMAXPROCS=1
benchmark.yml pins GOMAXPROCS: "1" (issue #767). At 2 Ps the concurrent
GC's mark workers ran on the second P, and allocation-heavy rows measured how
the shared runner scheduled that thread: ±7-20% on identical code, enough to
cross the 15% gate on PRs that never touched them. One P keeps GC cost in
sec/op and makes it deterministic. So CI rows carry no -N suffix, and to
reproduce a CI verdict locally measure the same way:
GOMAXPROCS=1 go test -run='^$' -bench=. -skip='Parallel$' -benchmem -benchtime=100ms -count=10 ./lisp/... | tee /tmp/bench.txt
The mechanism (concurrent GC on the second P) is inferred from one controlled experiment plus the PR's own run, not observed with perf counters.
RunParallel benchmarks must be named *Parallel (ci-gates-test enforces it).
The gated job skips them; the benchmark-parallel job measures them at
GOMAXPROCS=4 and reports benchgate's verdict without failing on it, because
on identical code they spread ~30% per arm.
benchgate strips the -N suffix only as told by -gomaxprocs /
BENCH_GOMAXPROCS (bench-compare.sh passes the job's GOMAXPROCS): at 1 a
benchmark's own trailing -2 is kept, at N>1 exactly -N is stripped, and
unset keeps the old strip-any-trailing-number behaviour.
Before you measure: is the machine fit? (make bench-burnin)
make bench-burnin # ~half a second; exit 0 fit, exit 3 re-measure elsewhere
A fixed, code-independent loop run seven times, requiring the samples to agree to within ±10%. A machine that cannot reproduce a fixed loop cannot resolve a 10% gate on anything else either, and half an hour of benchmarking on one produces numbers that read like findings. Run it first — on a laptop with a browser open as much as on CI.
UNMEASURABLE and exit 3
The gate has a fourth verdict (issue #542). A timing row whose own
confidence interval is at or above the fitness ceiling (-variance-ceiling,
default ±30%) is UNMEASURABLE: its delta is printed and adjudicated in
neither direction. If such a row moved at or above the gate, the run exits 3
(RUNNER-UNFIT) — it found no regression and certified nothing, so the answer
is to re-run, not to read the diff. An unmeasurable row below its gate is a
warning only and changes no exit code.
Exit 1 still wins over exit 3: a regression measured on a row that could be measured is a finding regardless of what else in the table was unmeasurable. And the ceiling can only ever withhold a finding — it never turns a passing run into a failing one.
The shape it exists for: an arm measuring itself at ±71% against the other arm's
±3%, on code that did not change, adjudicated as +83% REGRESSION. Preserved as
cmd/benchgate/testdata/elps/benchstat-runner-unfit-542.txt.
Checklist
- Base and head measured interleaved, with the same flags
- Whole-program benchmarks (
BenchmarkWorkload) checked for an interpreter change -
benchstatcomparison run - Results reported with clear formatting
- Significant regressions reported; any at or above the CI gate (15% timing / 5% allocs) investigated or waived
- Temp files cleaned up when done
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 193k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 193k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 193k | Updated | Claude CodeCodex | Verified | Install |