Skip to content

sybil-solutions/inference-optimisation-skill / skills/inference-optimisation

inference-optimisation

Unverified

Hill-climb the serving speed of a local LLM on any hardware (NVIDIA, AMD, Intel, Apple, unified-memory laptops, multi-GPU) and any quantization (GGUF, EXL2/EXL3, AWQ/GPTQ, FP8, MLX, BF16): single-user and concurrent decode tok/s, speculative decoding, prefill tok/s, KV cache size, startup time and TTFT, under a memory cap and with full-context stability. Computes the theoretical ceiling first, runs noise-controlled A/B experiments, keeps a ledger, and qualifies the winner before publishing it. Use when asked to optimise, tune, speed up, benchmark or hill-climb a model server (vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, TensorRT-LLM...), to find the maximum possible tok/s on a machine, or to publish a tuned recipe. Triggers: hill climb, tok/s, decode speed, prefill speed, MTP, EAGLE, draft model, speculative decoding, KV cache, TTFT, startup time, theoretical maximum.

FileKindSize
SKILL.mdskill md9.8 KB
examples/ceiling-ornith-b390.jsonconfig469 B
examples/config-existing-server.jsonconfig173 B
examples/config-sglang-docker.jsonconfig1.3 KB
references/ceilings.mdmarkdown4.3 KB
references/gotchas.mdmarkdown3.8 KB
references/levers.mdmarkdown6.6 KB
references/methodology.mdmarkdown4.5 KB
references/publishing.mdmarkdown3.5 KB
scripts/bench.pyexecutablescript12.5 KB
scripts/ceiling.pyexecutablescript2.6 KB
scripts/mem.shexecutablescript1.6 KB
scripts/queue.shexecutablescript786 B

examples/config-sglang-docker.json

View on GitHub
{
  "name": "ctl-a",
  "base_url": "http://127.0.0.1:8000",
  "model": "my-model",
  "launch": ["docker", "run", "-d", "--name", "llm", "--device", "/dev/dri", "--shm-size", "2g",
             "-p", "127.0.0.1:8000:8000",
             "-v", "/path/to/weights:/models:ro",
             "-v", "/path/to/persistent-cache:/root/.cache",
             "my-engine-image@sha256:...",
             "python3", "-m", "sglang.launch_server", "--model-path", "/models", "--served-model-name", "my-model",
             "--host", "0.0.0.0", "--port", "8000", "--context-length", "262144",
             "--max-running-requests", "4", "--chunked-prefill-size", "4096",
             "--speculative-algorithm", "NEXTN", "--speculative-num-steps", "1",
             "--speculative-eagle-topk", "1", "--speculative-num-draft-tokens", "2"],
  "stop": ["bash", "-c", "curl -s -X POST localhost:8000/abort_request -d '{\"abort_all\":true}' -H 'Content-Type: application/json'; sleep 5; docker stop -t 60 llm; docker rm -f llm"],
  "health_timeout_s": 900,
  "concurrency": 4,
  "prefill_lengths": [4096, 32768],
  "extra_body": {"chat_template_kwargs": {"enable_thinking": false}},
  "temperature": 0.6,
  "memory": {"cap_gib": 46, "probe": ["sudo", "-n", "bash", "scripts/mem.sh", "llm", "unified"], "interval_s": 3},
  "require_ac": true
}
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 197kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall