serejaris/personal-corp-os / skills/benchmark
benchmark
VerifiedUse when choosing between engines or models for an agent pipeline and the answer must come from measurement on your own data, not from marketing pages: speech recognition for meeting recordings, the model that turns a transcript into notes, the critic model that checks it. Runs every variant in an isolated container, measures time, cost per hour of input (USD and RUB, with price source and date) and quality (WER/CER and course terms for ASR; code checks plus a judge of a different model for LLM steps), writes results.json/csv and one static analytics page. Triggers on "бенчмарк", "сравни движки", "сколько стоит час записи", "какую модель взять для пайплайна", "benchmark the pipeline", "compare ASR engines".
Benchmark
Measure a pipeline on your own input before you pick an engine. One task file lists the variants; the scripts run them one by one in an isolated container and write numbers you can defend: time, cost per hour of input with a dated price source, quality against a reference.
Shape of a pipeline this skill knows:
input (recording) → ASR engine → transcript → author model (notes, chapters) → code checks → judge model
Each arrow is a slot. A variant fills one slot and keeps the rest fixed.
When to use
- «ElevenLabs, Whisper or a Russian engine: what does an hour of our meetings cost and who makes fewer mistakes»;
- «move the notes step from Claude to GLM: does quality hold»;
- before a pipeline goes unattended: pick the critic that finds the most real problems.
Not for: load testing, latency SLOs of a live service, model evals on public datasets.
Rules
- Isolation. Run in a dedicated container (LXC/VM), not on a laptop and not next to production: the timing is clean and the keys live only there. Egress: provider APIs; private network closed except the services the task needs (for example a self-hosted ASR).
- Keys only from env. The container gets a root-only env file; scripts read variable names from the task and never print values. Error bodies are redacted (some providers echo the key back).
- Same harness for every model. LLM steps run through
claude -p(Claude Code headless) against Anthropic-compatible endpoints; only the model changes. Tools, prompt, permissions stay the same. - Judge is never the author's model.
llm.pyrefuses such a variant. - No answer leaks. The author must not see the published result of the same input (a «form reference» that is the answer itself). Give it a sibling of the same kind.
- Prices carry source and date.
prices.json: every item hassourceandchecked; FX from the central bank of the day. A number without a source does not go into the report. - What was not run is a row too. Missing key, no balance, provider closed to new clients: write the reason and the list price, do not drop the row.
- Private input stays private. Transcripts and recordings go to a private repo; the analytics page shows only aggregate numbers.
Steps
- Task file (
task.example.json): input audio, language, terms list, reference transcript, ASR variants, LLM variants with judges,not_runrows with reasons. - Container. Create it by your infra rules (the section «Container» below). Install
ffmpeg, Python venv withhttpx jiwer faster-whisper, Node and@anthropic-ai/claude-code. Put keys into/etc/bench.env(root 600) through a pipe from your secret store. - ASR:
python3 scripts/asr.py --task task.json --out runs/<name>(engines run sequentially; ElevenLabs credits are read before and after). - LLM: write an adapter for your pipeline (contract in
scripts/llm.py), thenpython3 scripts/llm.py --task task.json --out runs/<name> --stage authoron the container and--stage judgewhere the judge's key lives (a subscription login on another machine is fine: the judge only reads files). - Score:
python3 scripts/score.py --task task.json --out runs/<name> --prices prices.json. - Page:
python3 scripts/report.py --results runs/<name>/results.json --out report.html --notes notes.md. - Method note next to the results: input, reference origin, what was not run and why, known biases.
Engines and providers
| Slot | Engine key | Tested | Needs |
|---|---|---|---|
| ASR | elevenlabs (Scribe, optional keyterms) | yes | ELEVENLABS_API_KEY |
| ASR | openai_compat (/v1/audio/transcriptions: self-hosted GigaAM, speaches, cloud) | yes | server URL, optional key |
| ASR | faster_whisper (CPU, int8) | yes | model download once |
| ASR | openrouter_audio (chat with input_audio, chunked) | request path only | OPENROUTER_API_KEY with balance |
| LLM | zai (GLM via api.z.ai/api/anthropic) | yes | ZAI_API_KEY |
| LLM | anthropic-local (the login on this machine) | yes | Claude Code logged in |
| LLM | openrouter, deepseek, anthropic-api, anthropic-oauth | not yet | key with balance |
Yandex SpeechKit and SaluteSpeech have no adapter yet: add one to ENGINES in asr.py (input: audio path, output: text, raw response, usage) when you have a key.
Metrics
ASR (against the reference, after lower case, ё→е, punctuation and speaker tags removed):
wer,cer: word and character error rate;wer_termnorm: WER after known term distortions are fixed in both texts;term_accuracy: share of term mentions written canonically, canon / (canon + known distortions), no reference needed;term_recall: canonical term mentions vs. the reference;asr_pairwise_wer: engines against each other, independent of the reference;rtf: wall time / audio time.
LLM: code checks before and after the fix round, fix rounds, judges' verdicts with critical / major / minor counts, tokens, turns, wall time.
Cost per hour of input: list price (api_price), tokens × list price (tokens), provider-reported cost (reported), or the CPU-seconds share of your server's month (server_share) for self-hosted engines.
Bias checklist (write it into the method note)
- Where the reference came from. A reference made by engine X favours X in WER; say so and read
term_accuracyand pairwise WER next to it. - Numbers: engines write «65» or «шестьдесят пять»; WER counts both as errors.
- Self-hosted time depends on the host's load at run time; note CPU threads and neighbours.
- One recording is one sample. Treat differences under a couple of WER points as noise.
Container
The skill does not create infrastructure. Follow your infra rules; minimum:
- a separate container for benchmarks, not a production or user container;
- no inbound exposure; no SSH if the host can exec into it;
- outbound: internet for APIs; private networks closed except named services;
- keys in a root-only env file, services started with
EnvironmentFile=, run as an unprivileged user; - CPU weight below production neighbours (for example
cpuunitslower than default).
Files
scripts/asr.py,scripts/llm.py,scripts/claude_step.py,scripts/score.py,scripts/report.py,scripts/common.pytask.example.json: a task with every engine typeprices.example.json: price table with sources, checked 2026-09-30examples/adapter_example.py: a minimal LLM adapter (author → checks → one fix → judge); run end to end with GLM-5.3-Flash as author and GLM-5.3 as judge
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |