sybil-solutions/inference-optimisation-skill / skills/inference-optimisation
inference-optimisation
UnverifiedHill-climb the serving speed of a local LLM on any hardware (NVIDIA, AMD, Intel, Apple, unified-memory laptops, multi-GPU) and any quantization (GGUF, EXL2/EXL3, AWQ/GPTQ, FP8, MLX, BF16): single-user and concurrent decode tok/s, speculative decoding, prefill tok/s, KV cache size, startup time and TTFT, under a memory cap and with full-context stability. Computes the theoretical ceiling first, runs noise-controlled A/B experiments, keeps a ledger, and qualifies the winner before publishing it. Use when asked to optimise, tune, speed up, benchmark or hill-climb a model server (vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, TensorRT-LLM...), to find the maximum possible tok/s on a machine, or to publish a tuned recipe. Triggers: hill climb, tok/s, decode speed, prefill speed, MTP, EAGLE, draft model, speculative decoding, KV cache, TTFT, startup time, theoretical maximum.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.8 KB |
| examples/ceiling-ornith-b390.json | config | 469 B |
| examples/config-existing-server.json | config | 173 B |
| examples/config-sglang-docker.json | config | 1.3 KB |
| references/ceilings.md | markdown | 4.3 KB |
| references/gotchas.md | markdown | 3.8 KB |
| references/levers.md | markdown | 6.6 KB |
| references/methodology.md | markdown | 4.5 KB |
| references/publishing.md | markdown | 3.5 KB |
| scripts/bench.pyexecutable | script | 12.5 KB |
| scripts/ceiling.pyexecutable | script | 2.6 KB |
| scripts/mem.shexecutable | script | 1.6 KB |
| scripts/queue.shexecutable | script | 786 B |
references/publishing.md
View on GitHub# Qualifying and publishing a tuned recipe
A tuned config is only useful if someone else can rebuild it and get the same result.
## 1. Land the code
- Commit kernel and engine-plugin changes to a branch of a public repo. A published build must fetch a pinned commit, never a local patch or a dirty tree.
- Put data files the config needs (draft vocabularies, configs) in the recipe source. Copy them into the image.
## 2. Pin and build cleanly
- **Pin everything:**
- the base image by digest;
- the engine and kernel source by commit;
- Python packages by version (fail the build if `pip freeze` differs);
- weights by repository revision.
- **Build from a clean checkout of the exact recipe commit.** Keep the build log as evidence.
- **Build every library with the same flags you benchmarked.** Never load two builds of the same op library in one process.
## 3. Write the launch exactly as benchmarked
- Copy the environment and arguments from the best config, then diff them mechanically. Change nothing "while you're there".
- Make advertised numbers true. If a registry derives KV capacity as `ctx × seqs (+ margin)`, set the engine's pool to that value. Note any difference from what you soak-tested.
## 4. Acceptance gates (all required, on the clean build)
1. **Load:** the server becomes healthy, and `/v1/models` lists the model.
2. **Chat:** a short answer with a stop finish.
3. **Reasoning:** thinking separated from content, with the correct answer.
4. **Tools:** a full round trip. The model calls the tool, receives the result, and quotes a code from it.
5. **Context:** a needle in a document at about 85% of the window, answered exactly.
6. **Speed:** single-user decode over the first 30 s after the first token, with no output cap, counted with the server tokenizer, above a floor.
- Also check **vision** if the model has it, plus a **concurrent full-length soak** if the recipe advertises N users at full context.
- Gate the same config more than once if acceptance ramps up (draft methods with warm-up). Never publish a "validated" result from a single wave.
- Record peak memory: driver residency plus anon on unified memory.
## 5. Prove the published build equals the best
- Run your harness back to back on the clean build and your hill-climb best image, against interleaved controls. They should match within noise.
- If they don't, find the difference before publishing.
## 6. Publish
- **Branch from the current main.** Squash-merged repos don't contain your old branch history.
- **Keep any branch that holds a pinned build commit alive.** Deleting it breaks reproducibility, unless the commit is also reachable from main or a PR ref.
- **Evidence to commit:**
- the gate run JSON;
- the rendered launch and its hash;
- build and server logs;
- memory samples;
- the vision response;
- the check and test logs;
- the experiment log (`PLAN.md`, `THEORY.md`, ledger).
- **Grep the bundle** for secrets, credential-file paths, tokens, personal emails and home-directory usernames before pushing.
- **State in the README** which numbers come from the gates and which from the hill-climb harness, and how each was measured.
## 7. Results card or poster (optional)
- Use medians, not best runs, and show honest `was X → now Y (+Z%)` deltas.
- Label cold vs warm and single runs vs medians.
- If you have a house template, change only the values. Render headless:
```bash
chromium --headless=new --force-device-scale-factor=2 --window-size=1024,574 --screenshot=out.png file://$PWD/card.html
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |