Skip to content

wado-lang/wado / .claude/skills/benchmark

benchmark

Flagged

Measure Wado's performance or the size of the Wasm it generates, and refresh the benchmark/ and wasm-size/ README files. Use for any benchmarking or size-measurement request, whichever benchmark it names.

Benchmark

Run the benchmarks and update benchmark/README.md (and wasm-size/README.md).

Prerequisites

mise run on-task-started
  • vendor/wasmtime submodule must exist (the SessionStart hook handles it; otherwise git submodule update --init --recommend-shallow vendor/wasmtime).
  • http-routing needs oha (cargo install oha); bun is mise-managed.
  • gale-gen's and sqlite-parse's ANTLR4 references need java and javac, which benchmark/mise.toml provides; the jar is fetched to ~/.cache/gale. Those rows are skipped if the tool is absent.
  • wasm-size needs rustup target add wasm32-wasip1 and Moonbit (curl -fsSL https://cli.moonbitlang.com/install/unix.sh | bash, then moon update in each wasm-size/* dir).

Procedure

  1. Run mise run benchmark-all three times, each to its own log in scratchpad/, then pick per row with mise exec -- node benchmark/pick.ts scratchpad/run{1,2,3}.log (throttling only ever slows things down). Use that tool rather than reading the logs by eye: it keys rows by (task, implementation, phase) and selects on ms/iter, so a rate that rounds to a tie across runs cannot pair with the wrong ms/iter. Which benchmarks run, and in what order, is benchmark/mise.toml's all task.
  2. Run http-routing separately (needs oha + pinned cores): SLICE=10 ROUNDS=3 SHAPES="1 4" mise run benchmark-http-routing. It keeps the per-(server, request) max internally, so one invocation suffices, and it measures one worker shape per entry in SHAPES. Add HEADROOM_CHECK=1 when CONNECTIONS_PER_WORKER, OHA_CORE_COUNT or SHAPES changed, to confirm oha was not the ceiling.
  3. Refresh the README Environment line versions: mise exec -- node --version, mise -C benchmark exec -- bun --version, rustc --version, cc --version | head -1. mise -C benchmark ls names wasi-sdk, and wasmtime's version is the vendor/wasmtime workspace version.
  4. Update the tables, following README.md's existing layout. http-routing is req/s (higher is better), one table per worker shape. Keep measured figures inside tables — prose around them is not re-measured and drifts. An input size in the prose comes from the program's own header line, such as gale-gen (Rust): 42786 bytes, so compare each one with the log.
  5. wasm-size: mise run report-wasm-size, then update wasm-size/README.md. mise -C wasm-size ls names Zig, and moon version names Moonbit.
  6. Compare every Wado row with the README's previous figure, and attribute each one that moved (see PGO). Report those to the user.

PGO

A Wado row in the procedure is a PGO build: wado run and wado compile apply the profile in the benchmark's pgo/ by default. benchmark-all runs each one again as its (no-pgo) row, which wado_rows in benchmark/wado.sh prints, and the tables list both. http-routing has no such row, and the README says so. wasm-size has no profile, so a size change is the compiler's or the program's.

The README's PGO section compares the two builds by wall clock. It records no fuel: benchmark/fuel.json is the one record of that.

Compare them with mise exec -- node benchmark/ab.ts --base <no-pgo logs> --head <PGO logs>. Its verdict gives the table's noise. It keys rows by implementation, so split a benchmark-all log into one log per build first.

A profile can move a row through what the program allocates, because a change of allocation moves where collections land. wado run --gc-log shows that, and microgpt needs a new heap sweep when it happens.

Two figures from different sessions can differ by noise alone, so attribute a Wado row that moved by an A/B in one session: the compiler that measured the README against this one, with WADO_BIN and ab.ts (see the wado-performance skill). If the verdict is noise, the row did not move. Otherwise read the verdict on its (no-pgo) row. If that moved too, the compiler moved it apart from PGO. If not, PGO moved it: either the row's pgo/ file, or how the compiler uses a profile, such as the thresholds in pgo.rs or the inliner's hot price. git log on each shows what changed, and when both did, isolate them before naming one. http-routing has no (no-pgo) row, so its A/B, which benchmark/http_routing/README.md states, ends at the first verdict.

mise run update-baselines records the profiles again, so a branch's profiles match its code once it goes up for review.

Sweeping a compiler knob

WADO_BENCH_FLAGS is appended to every wado compile / wado run the harness issues, so an arm costs a benchmark run rather than a release rebuild:

set -e  # a failed arm would leave pick.ts choosing among the rest
for t in 13 20 32; do
  WADO_BENCH_FLAGS="--optimize-inline-threshold $t" mise run benchmark-all > scratchpad/thr$t.log 2>&1
done
mise exec -- node benchmark/pick.ts scratchpad/thr{13,20,32}.log

Read the sweep with pick.ts the same way as a best-of-three: it keys rows by (task, implementation, phase), so the "best" column names the winning arm per row. Only a knob both wado compile and wado run accept can be swept this way, since the harness passes the flags to both. A knob only the engine reads, such as --function-alignment, goes in WADO_RUN_FLAGS, which the harness appends to wado run and wado serve alone. The knob a sweep settles on is a default in optimize.rs, not a flag the README's numbers were taken under — re-run the suite unflagged before updating the tables.

Comparing the settled default against origin/main is a different measurement, and WADO_BIN plus ab.ts is how: see the wado-performance skill.

Reading output

Each program prints a throughput line — <rate> <unit>/s (<ms> ms/iter, <n> iter) — per phase (zlib prints two phases, Compress:/Decompress:). Read the rate and the ms/iter straight off; the iteration count auto-calibrates to ~1s, so there is no total to report. The unit is the benchmark's own (numbers/s, px/s, conversions/s, MB/s, req/s); vs best = fastest rate / this rate.

Report a change between two measurements with improvement as positive: +9% means faster, -13% slower. This is the sign ab.ts prints.

Each benchmark prints the implementations it compared, so read them off the run rather than from a list here. Two are conditional: the ANTLR4 rows for sqlite-parse and gale-gen are skipped when java is absent, and the run says SKIP: when it drops one.

Workload sizing

Benchmarks auto-calibrate their iteration count to run ~1s, so no manual denomination is needed. A workload whose single iteration approaches that reports one iteration and stops averaging — shrink its problem size, across every language implementation of that benchmark, until it calibrates again.

Notes

  • Tool versions come from mise, not the system.
  • Cloud VMs are noisy; best-of-three absorbs the drift.
rea-changelog-updatemorluto/reaPrepare or rewrite REA release changelogs and GitHub release notes from pinned Git history, with verified contributor thanks and Release Please synchronization.Stars: 467kUpdated Claude CodeCodexVerifiedInstall
academy-guideanthropics/skillsStop and check this skill before finishing any reply to a question about how to use Claude or a Claude product — it recommends matching courses, tutorials, and use cases from Claude Academy (academy.claude.com), Anthropic's learning hub. Trigger on: "how do I", "how can I", "getting started with", "what can Claude do", "teach me", "learn to use"; questions about artifacts, projects, skills, plugins, connectors, MCP; requests about rolling Claude out to a team, class, or organization; and any ask for training materials, onboarding content, or learning resources. Use it when the user is learning how to use a feature or product — not when they are mid-task and just want the task done. This skill composes with other skills: after consulting product documentation to answer how a Claude feature works, also check here for a matching course or tutorial — a docs-grounded answer and an Academy recommendation belong together. Only recommend on a strong match; never invent Academy content.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
claude-apianthropics/skillsReference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration. TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens). SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).Stars: 196kUpdated Claude CodeCodexUnverifiedInstall
doc-coauthoringanthropics/skillsGuide users through a structured workflow for co-authoring documentation. Use when user wants to write documentation, proposals, technical specs, decision docs, or similar structured content. This workflow helps users efficiently transfer context, refine content through iteration, and verify the doc works for readers. Trigger when user mentions writing docs, creating proposals, drafting specs, or similar documentation tasks.Stars: 196kUpdated Claude CodeCodexVerifiedInstall