wado-lang/wado / .claude/skills/benchmark
benchmark
FlaggedMeasure Wado's performance or the size of the Wasm it generates, and refresh the benchmark/ and wasm-size/ README files. Use for any benchmarking or size-measurement request, whichever benchmark it names.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 7.2 KB |
SKILL.md
View on GitHub---
name: benchmark
description: Measure Wado's performance or the size of the Wasm it generates, and refresh the benchmark/ and wasm-size/ README files. Use for any benchmarking or size-measurement request, whichever benchmark it names.
---
# Benchmark
Run the benchmarks and update `benchmark/README.md` (and `wasm-size/README.md`).
## Prerequisites
```sh
mise run on-task-started
```
- `vendor/wasmtime` submodule must exist (the SessionStart hook handles it;
otherwise `git submodule update --init --recommend-shallow vendor/wasmtime`).
- http-routing needs `oha` (`cargo install oha`); `bun` is mise-managed.
- gale-gen's and sqlite-parse's ANTLR4 references need `java` and `javac`, which
`benchmark/mise.toml` provides; the jar is fetched to `~/.cache/gale`. Those
rows are skipped if the tool is absent.
- wasm-size needs `rustup target add wasm32-wasip1` and Moonbit
(`curl -fsSL https://cli.moonbitlang.com/install/unix.sh | bash`, then
`moon update` in each `wasm-size/*` dir).
## Procedure
1. Record the PGO profiles again first:
`scripts/dev-bin.sh wado test --pgo benchmark package-gale-highlight-html`,
and commit the profiles it rewrites.
2. Run `mise run benchmark-all` **three times**, each to its own log in
`scratchpad/`, then pick per row with
`mise exec -- node benchmark/pick.ts scratchpad/run{1,2,3}.log` (throttling
only ever slows things down). Use that tool rather than reading the logs by
eye: it keys rows by (task, implementation, phase) and selects on ms/iter, so
a rate that rounds to a tie across runs cannot pair with the wrong ms/iter.
Which benchmarks run, and in what order, is `benchmark/mise.toml`'s `all`
task.
3. Run http-routing separately (needs `oha` + pinned cores):
`SLICE=10 ROUNDS=3 SHAPES="1 4" mise run benchmark-http-routing`. It keeps
the per-(server, request) max internally, so one invocation suffices, and it
measures one worker shape per entry in `SHAPES`. Add `HEADROOM_CHECK=1` when
`CONNECTIONS_PER_WORKER`, `OHA_CORE_COUNT` or `SHAPES` changed, to confirm
`oha` was not the ceiling.
4. Refresh the README Environment line versions: `mise exec -- node --version`,
`mise -C benchmark exec -- bun --version`, `rustc --version`,
`cc --version | head -1`. `mise -C benchmark ls` names wasi-sdk, and wasmtime's
version is the `vendor/wasmtime` workspace version.
5. Update the tables, following README.md's existing layout. http-routing is
req/s (higher is better), one table per worker shape. Keep measured figures
inside tables — prose around them is not re-measured and drifts. An input
size in the prose comes from the program's own header line, such as
`gale-gen (Rust): 42786 bytes`, so compare each one with the log.
6. wasm-size: `mise run report-wasm-size`, then update `wasm-size/README.md`.
`mise -C wasm-size ls` names Zig, and `moon version` names Moonbit.
7. Compare every Wado row with the README's previous figure, and attribute each
one that moved (see [PGO](#pgo)). Report those to the user.
## PGO
A Wado row in the procedure is a PGO build: `wado run` and `wado compile` apply
the profile in the benchmark's `pgo/` by default. `benchmark-all` runs each one
again as its `(no-pgo)` row, which `wado_rows` in `benchmark/wado.sh` prints, and
the tables list both. http-routing has no such row, and the README says so.
wasm-size has no profile, so a size change is the compiler's or the program's.
The README's PGO section compares the two builds by wall clock. It records no
fuel: `benchmark/fuel.json` is the one record of that.
Compare them with
`mise exec -- node benchmark/ab.ts --base <no-pgo logs> --head <PGO logs>`. Its
verdict gives the table's `noise`. It keys rows by implementation, so split a
`benchmark-all` log into one log per build first.
A profile can move a row through what the program allocates, because a change of
allocation moves where collections land. `wado run --gc-log` shows that, and
`microgpt` needs a new heap sweep when it happens.
Two figures from different sessions can differ by noise alone, so attribute a
Wado row that moved by an A/B in one session: the compiler that measured the
README against this one, with `WADO_BIN` and `ab.ts` (see the `wado-performance`
skill). If the verdict is `noise`, the row did not move. Otherwise read the
verdict on its `(no-pgo)` row. If that moved too, the compiler moved it apart
from PGO. If not, PGO moved it: either the row's `pgo/` file, or how the compiler
uses a profile, such as the thresholds in `pgo.rs` or the inliner's hot price.
`git log` on each shows what changed, and when both did, isolate them before
naming one. http-routing has no `(no-pgo)` row, so its A/B, which
`benchmark/http_routing/README.md` states, ends at the first verdict.
`mise run update-baselines` records the profiles again, so a branch's
profiles match its code once it goes up for review.
## Sweeping a compiler knob
`WADO_BENCH_FLAGS` is appended to every `wado compile` / `wado run` the harness
issues, so an arm costs a benchmark run rather than a release rebuild:
```sh
set -e # a failed arm would leave pick.ts choosing among the rest
for t in 13 20 32; do
WADO_BENCH_FLAGS="--optimize-inline-threshold $t" mise run benchmark-all > scratchpad/thr$t.log 2>&1
done
mise exec -- node benchmark/pick.ts scratchpad/thr{13,20,32}.log
```
Read the sweep with `pick.ts` the same way as a best-of-three: it keys rows by
(task, implementation, phase), so the "best" column names the winning arm per
row. Only a knob both `wado compile` and `wado run` accept can be swept this
way, since the harness passes the flags to both. A knob only the engine reads,
such as `--function-alignment`, goes in `WADO_RUN_FLAGS`, which the harness
appends to `wado run` and `wado serve` alone. The knob a sweep settles on is
a default in `optimize.rs`, not a flag the README's numbers were taken under —
re-run the suite unflagged before updating the tables.
Comparing the settled default against `origin/main` is a different measurement,
and `WADO_BIN` plus `ab.ts` is how: see the `wado-performance` skill.
## Reading output
Each program prints a throughput line — `<rate> <unit>/s (<ms> ms/iter, <n> iter)` — per phase (zlib prints two phases, `Compress:`/`Decompress:`).
Read the rate and the ms/iter straight off; the iteration count auto-calibrates
to ~1s, so there is no total to report. The unit is the benchmark's own
(numbers/s, px/s, conversions/s, MB/s, req/s); `vs best` = fastest rate / this
rate.
Report a change between two measurements with improvement as positive: `+9%`
means faster, `-13%` slower. This is the sign `ab.ts` prints.
Each benchmark prints the implementations it compared, so read them off the run
rather than from a list here. Two are conditional: the ANTLR4 rows for
`sqlite-parse` and `gale-gen` are skipped when `java` is absent, and the run
says `SKIP:` when it drops one.
## Workload sizing
Benchmarks auto-calibrate their iteration count to run ~1s, so no manual
denomination is needed. A workload whose single iteration approaches that
reports one iteration and stops averaging — shrink its problem size, across
every language implementation of that benchmark, until it calibrates again.
## Notes
- Tool versions come from mise, not the system.
- Cloud VMs are noisy; best-of-three absorbs the drift.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-changelog-updatemorluto/reaPrepare or rewrite REA release changelogs and GitHub release notes from pinned Git history, with verified contributor thanks and Release Please synchronization. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| academy-guideanthropics/skillsStop and check this skill before finishing any reply to a question about how to use Claude or a Claude product — it recommends matching courses, tutorials, and use cases from Claude Academy (academy.claude.com), Anthropic's learning hub. Trigger on: "how do I", "how can I", "getting started with", "what can Claude do", "teach me", "learn to use"; questions about artifacts, projects, skills, plugins, connectors, MCP; requests about rolling Claude out to a team, class, or organization; and any ask for training materials, onboarding content, or learning resources. Use it when the user is learning how to use a feature or product — not when they are mid-task and just want the task done. This skill composes with other skills: after consulting product documentation to answer how a Claude feature works, also check here for a matching course or tutorial — a docs-grounded answer and an Academy recommendation belong together. Only recommend on a strong match; never invent Academy content. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| claude-apianthropics/skillsReference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration. TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens). SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file). | Stars: 198k | Updated | Claude CodeCodex | Unverified | Install |
| doc-coauthoringanthropics/skillsGuide users through a structured workflow for co-authoring documentation. Use when user wants to write documentation, proposals, technical specs, decision docs, or similar structured content. This workflow helps users efficiently transfer context, refine content through iteration, and verify the doc works for readers. Trigger when user mentions writing docs, creating proposals, drafting specs, or similar documentation tasks. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |