wado-lang/wado / .claude/skills/benchmark
benchmark
FlaggedMeasure Wado's performance or the size of the Wasm it generates, and refresh the benchmark/ and wasm-size/ README files. Use for any benchmarking or size-measurement request, whichever benchmark it names.
Benchmark
Run the benchmarks and update benchmark/README.md (and wasm-size/README.md).
Prerequisites
mise run on-task-started
vendor/wasmtimesubmodule must exist (the SessionStart hook handles it; otherwisegit submodule update --init --recommend-shallow vendor/wasmtime).- http-routing needs
oha(cargo install oha);bunis mise-managed. - gale-gen's and sqlite-parse's ANTLR4 references need
javaandjavac, whichbenchmark/mise.tomlprovides; the jar is fetched to~/.cache/gale. Those rows are skipped if the tool is absent. - wasm-size needs
rustup target add wasm32-wasip1and Moonbit (curl -fsSL https://cli.moonbitlang.com/install/unix.sh | bash, thenmoon updatein eachwasm-size/*dir).
Procedure
- Run
mise run benchmark-allthree times, each to its own log inscratchpad/, then pick per row withmise exec -- node benchmark/pick.ts scratchpad/run{1,2,3}.log(throttling only ever slows things down). Use that tool rather than reading the logs by eye: it keys rows by (task, implementation, phase) and selects on ms/iter, so a rate that rounds to a tie across runs cannot pair with the wrong ms/iter. Which benchmarks run, and in what order, isbenchmark/mise.toml'salltask. - Run http-routing separately (needs
oha+ pinned cores):SLICE=10 ROUNDS=3 SHAPES="1 4" mise run benchmark-http-routing. It keeps the per-(server, request) max internally, so one invocation suffices, and it measures one worker shape per entry inSHAPES. AddHEADROOM_CHECK=1whenCONNECTIONS_PER_WORKER,OHA_CORE_COUNTorSHAPESchanged, to confirmohawas not the ceiling. - Refresh the README Environment line versions:
mise exec -- node --version,mise -C benchmark exec -- bun --version,rustc --version,cc --version | head -1.mise -C benchmark lsnames wasi-sdk, and wasmtime's version is thevendor/wasmtimeworkspace version. - Update the tables, following README.md's existing layout. http-routing is
req/s (higher is better), one table per worker shape. Keep measured figures
inside tables — prose around them is not re-measured and drifts. An input
size in the prose comes from the program's own header line, such as
gale-gen (Rust): 42786 bytes, so compare each one with the log. - wasm-size:
mise run report-wasm-size, then updatewasm-size/README.md.mise -C wasm-size lsnames Zig, andmoon versionnames Moonbit. - Compare every Wado row with the README's previous figure, and attribute each one that moved (see PGO). Report those to the user.
PGO
A Wado row in the procedure is a PGO build: wado run and wado compile apply
the profile in the benchmark's pgo/ by default. benchmark-all runs each one
again as its (no-pgo) row, which wado_rows in benchmark/wado.sh prints, and
the tables list both. http-routing has no such row, and the README says so.
wasm-size has no profile, so a size change is the compiler's or the program's.
The README's PGO section compares the two builds by wall clock. It records no
fuel: benchmark/fuel.json is the one record of that.
Compare them with
mise exec -- node benchmark/ab.ts --base <no-pgo logs> --head <PGO logs>. Its
verdict gives the table's noise. It keys rows by implementation, so split a
benchmark-all log into one log per build first.
A profile can move a row through what the program allocates, because a change of
allocation moves where collections land. wado run --gc-log shows that, and
microgpt needs a new heap sweep when it happens.
Two figures from different sessions can differ by noise alone, so attribute a
Wado row that moved by an A/B in one session: the compiler that measured the
README against this one, with WADO_BIN and ab.ts (see the wado-performance
skill). If the verdict is noise, the row did not move. Otherwise read the
verdict on its (no-pgo) row. If that moved too, the compiler moved it apart
from PGO. If not, PGO moved it: either the row's pgo/ file, or how the compiler
uses a profile, such as the thresholds in pgo.rs or the inliner's hot price.
git log on each shows what changed, and when both did, isolate them before
naming one. http-routing has no (no-pgo) row, so its A/B, which
benchmark/http_routing/README.md states, ends at the first verdict.
mise run update-baselines records the profiles again, so a branch's
profiles match its code once it goes up for review.
Sweeping a compiler knob
WADO_BENCH_FLAGS is appended to every wado compile / wado run the harness
issues, so an arm costs a benchmark run rather than a release rebuild:
set -e # a failed arm would leave pick.ts choosing among the rest
for t in 13 20 32; do
WADO_BENCH_FLAGS="--optimize-inline-threshold $t" mise run benchmark-all > scratchpad/thr$t.log 2>&1
done
mise exec -- node benchmark/pick.ts scratchpad/thr{13,20,32}.log
Read the sweep with pick.ts the same way as a best-of-three: it keys rows by
(task, implementation, phase), so the "best" column names the winning arm per
row. Only a knob both wado compile and wado run accept can be swept this
way, since the harness passes the flags to both. A knob only the engine reads,
such as --function-alignment, goes in WADO_RUN_FLAGS, which the harness
appends to wado run and wado serve alone. The knob a sweep settles on is
a default in optimize.rs, not a flag the README's numbers were taken under —
re-run the suite unflagged before updating the tables.
Comparing the settled default against origin/main is a different measurement,
and WADO_BIN plus ab.ts is how: see the wado-performance skill.
Reading output
Each program prints a throughput line — <rate> <unit>/s (<ms> ms/iter, <n> iter) — per phase (zlib prints two phases, Compress:/Decompress:).
Read the rate and the ms/iter straight off; the iteration count auto-calibrates
to ~1s, so there is no total to report. The unit is the benchmark's own
(numbers/s, px/s, conversions/s, MB/s, req/s); vs best = fastest rate / this
rate.
Report a change between two measurements with improvement as positive: +9%
means faster, -13% slower. This is the sign ab.ts prints.
Each benchmark prints the implementations it compared, so read them off the run
rather than from a list here. Two are conditional: the ANTLR4 rows for
sqlite-parse and gale-gen are skipped when java is absent, and the run
says SKIP: when it drops one.
Workload sizing
Benchmarks auto-calibrate their iteration count to run ~1s, so no manual denomination is needed. A workload whose single iteration approaches that reports one iteration and stops averaging — shrink its problem size, across every language implementation of that benchmark, until it calibrates again.
Notes
- Tool versions come from mise, not the system.
- Cloud VMs are noisy; best-of-three absorbs the drift.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-changelog-updatemorluto/reaPrepare or rewrite REA release changelogs and GitHub release notes from pinned Git history, with verified contributor thanks and Release Please synchronization. | Stars: 467k | Updated | Claude CodeCodex | Verified | Install |
| academy-guideanthropics/skillsStop and check this skill before finishing any reply to a question about how to use Claude or a Claude product — it recommends matching courses, tutorials, and use cases from Claude Academy (academy.claude.com), Anthropic's learning hub. Trigger on: "how do I", "how can I", "getting started with", "what can Claude do", "teach me", "learn to use"; questions about artifacts, projects, skills, plugins, connectors, MCP; requests about rolling Claude out to a team, class, or organization; and any ask for training materials, onboarding content, or learning resources. Use it when the user is learning how to use a feature or product — not when they are mid-task and just want the task done. This skill composes with other skills: after consulting product documentation to answer how a Claude feature works, also check here for a matching course or tutorial — a docs-grounded answer and an Academy recommendation belong together. Only recommend on a strong match; never invent Academy content. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| claude-apianthropics/skillsReference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration. TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens). SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file). | Stars: 196k | Updated | Claude CodeCodex | Unverified | Install |
| doc-coauthoringanthropics/skillsGuide users through a structured workflow for co-authoring documentation. Use when user wants to write documentation, proposals, technical specs, decision docs, or similar structured content. This workflow helps users efficiently transfer context, refine content through iteration, and verify the doc works for readers. Trigger when user mentions writing docs, creating proposals, drafting specs, or similar documentation tasks. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |