sybil-solutions/inference-optimisation-skill / skills/inference-optimisation
inference-optimisation
UnverifiedHill-climb the serving speed of a local LLM on any hardware (NVIDIA, AMD, Intel, Apple, unified-memory laptops, multi-GPU) and any quantization (GGUF, EXL2/EXL3, AWQ/GPTQ, FP8, MLX, BF16): single-user and concurrent decode tok/s, speculative decoding, prefill tok/s, KV cache size, startup time and TTFT, under a memory cap and with full-context stability. Computes the theoretical ceiling first, runs noise-controlled A/B experiments, keeps a ledger, and qualifies the winner before publishing it. Use when asked to optimise, tune, speed up, benchmark or hill-climb a model server (vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, TensorRT-LLM...), to find the maximum possible tok/s on a machine, or to publish a tuned recipe. Triggers: hill climb, tok/s, decode speed, prefill speed, MTP, EAGLE, draft model, speculative decoding, KV cache, TTFT, startup time, theoretical maximum.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.8 KB |
| examples/ceiling-ornith-b390.json | config | 469 B |
| examples/config-existing-server.json | config | 173 B |
| examples/config-sglang-docker.json | config | 1.3 KB |
| references/ceilings.md | markdown | 4.3 KB |
| references/gotchas.md | markdown | 3.8 KB |
| references/levers.md | markdown | 6.6 KB |
| references/methodology.md | markdown | 4.5 KB |
| references/publishing.md | markdown | 3.5 KB |
| scripts/bench.pyexecutable | script | 12.5 KB |
| scripts/ceiling.pyexecutable | script | 2.6 KB |
| scripts/mem.shexecutable | script | 1.6 KB |
| scripts/queue.shexecutable | script | 786 B |
examples/config-sglang-docker.json
View on GitHub{
"name": "ctl-a",
"base_url": "http://127.0.0.1:8000",
"model": "my-model",
"launch": ["docker", "run", "-d", "--name", "llm", "--device", "/dev/dri", "--shm-size", "2g",
"-p", "127.0.0.1:8000:8000",
"-v", "/path/to/weights:/models:ro",
"-v", "/path/to/persistent-cache:/root/.cache",
"my-engine-image@sha256:...",
"python3", "-m", "sglang.launch_server", "--model-path", "/models", "--served-model-name", "my-model",
"--host", "0.0.0.0", "--port", "8000", "--context-length", "262144",
"--max-running-requests", "4", "--chunked-prefill-size", "4096",
"--speculative-algorithm", "NEXTN", "--speculative-num-steps", "1",
"--speculative-eagle-topk", "1", "--speculative-num-draft-tokens", "2"],
"stop": ["bash", "-c", "curl -s -X POST localhost:8000/abort_request -d '{\"abort_all\":true}' -H 'Content-Type: application/json'; sleep 5; docker stop -t 60 llm; docker rm -f llm"],
"health_timeout_s": 900,
"concurrency": 4,
"prefill_lengths": [4096, 32768],
"extra_body": {"chat_template_kwargs": {"enable_thinking": false}},
"temperature": 0.6,
"memory": {"cap_gib": 46, "probe": ["sudo", "-n", "bash", "scripts/mem.sh", "llm", "unified"], "interval_s": 3},
"require_ac": true
}
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |