sybil-solutions/inference-optimisation-skill / skills/inference-optimisation
inference-optimisation
UnverifiedHill-climb the serving speed of a local LLM on any hardware (NVIDIA, AMD, Intel, Apple, unified-memory laptops, multi-GPU) and any quantization (GGUF, EXL2/EXL3, AWQ/GPTQ, FP8, MLX, BF16): single-user and concurrent decode tok/s, speculative decoding, prefill tok/s, KV cache size, startup time and TTFT, under a memory cap and with full-context stability. Computes the theoretical ceiling first, runs noise-controlled A/B experiments, keeps a ledger, and qualifies the winner before publishing it. Use when asked to optimise, tune, speed up, benchmark or hill-climb a model server (vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, TensorRT-LLM...), to find the maximum possible tok/s on a machine, or to publish a tuned recipe. Triggers: hill climb, tok/s, decode speed, prefill speed, MTP, EAGLE, draft model, speculative decoding, KV cache, TTFT, startup time, theoretical maximum.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.8 KB |
| examples/ceiling-ornith-b390.json | config | 469 B |
| examples/config-existing-server.json | config | 173 B |
| examples/config-sglang-docker.json | config | 1.3 KB |
| references/ceilings.md | markdown | 4.3 KB |
| references/gotchas.md | markdown | 3.8 KB |
| references/levers.md | markdown | 6.6 KB |
| references/methodology.md | markdown | 4.5 KB |
| references/publishing.md | markdown | 3.5 KB |
| scripts/bench.pyexecutable | script | 12.5 KB |
| scripts/ceiling.pyexecutable | script | 2.6 KB |
| scripts/mem.shexecutable | script | 1.6 KB |
| scripts/queue.shexecutable | script | 786 B |
scripts/ceiling.py
View on GitHub#!/usr/bin/env python3
"""Theoretical ceilings for decode / prefill / KV from a small JSON of measured hardware and model sizes.
python3 ceiling.py model.json
Example model.json (Ornith-1.5-35B-A3B EXL3 4bpw on an Arc B390 iGPU; sizes in GB from the checkpoint):
{
"bandwidth_gbs": 115, "peak_tflops": 36, "mfu": 0.5,
"dense_gb": 0.758, "experts_gb": 16.26, "top_k": 8, "n_experts": 256, "lm_head_gb": 0.382,
"active_params_b": 2.43, "attn_layers": 10, "attn_heads": 16, "head_dim": 256,
"kv_bytes_per_token": 10240, "draft_layer_gb": 0.03, "draft_head_gb": 0.05,
"spec": [{"drafts": 0, "accept": 1.0}, {"drafts": 1, "accept": 1.74}, {"drafts": 2, "accept": 2.5}],
"contexts": [4096, 32768, 131072, 262144], "concurrency": [1, 4]
}
Dense models: set experts_gb to 0. draft_head_gb is the draft LM head as read per draft step (prune it!).
"""
import json, sys
def decode_ceiling(m, drafts, accept, ctx=0, n=1):
union = 1 - (1 - m["top_k"] / m["n_experts"]) ** ((drafts + 1) * n) if m.get("experts_gb") else 0
verify = m["dense_gb"] + m["lm_head_gb"] + m.get("experts_gb", 0) * union
verify += n * m.get("kv_bytes_per_token", 0) * ctx / 1e9 * (drafts + 1 if m.get("kv_per_verify_row") else 1)
draft = drafts * n * (m.get("draft_layer_gb", 0) + m.get("draft_head_gb", m["lm_head_gb"]))
return m["bandwidth_gbs"] / ((verify + draft) / (accept * n)) * n # aggregate tok/s
def prefill_ceiling(m, L):
linear = 2 * m["active_params_b"] * 1e9
attn = 4 * m.get("attn_layers", 0) * m.get("attn_heads", 0) * m.get("head_dim", 0) * (L / 2)
return m["mfu"] * m["peak_tflops"] * 1e12 / (linear + attn + m.get("extra_flops_per_token", 0))
def main(path):
m = json.load(open(path))
print(f"bandwidth {m['bandwidth_gbs']} GB/s, peak {m['peak_tflops']} TFLOPS, MFU {m['mfu']}")
for n in m.get("concurrency", [1]):
print(f"\nDecode ceiling, concurrency {n} (aggregate tok/s):")
print(" drafts accept" + "".join(f"{c // 1024:>8}K" for c in m.get("contexts", [0])))
for s in m.get("spec", [{"drafts": 0, "accept": 1.0}]):
row = "".join(f"{decode_ceiling(m, s['drafts'], s['accept'], c, n):9.0f}" for c in m.get("contexts", [0]))
print(f" {s['drafts']:>6} {s['accept']:>6}{row}")
print("\nPrefill ceiling (tok/s, practical):")
for L in m.get("contexts", [4096]):
print(f" {L // 1024:>4}K {prefill_ceiling(m, L):8.0f}")
if m.get("kv_bytes_per_token"):
print("\nKV: {:.2f} GB per 256K-token stream".format(m["kv_bytes_per_token"] * 262144 / 1e9))
if __name__ == "__main__":
main(sys.argv[1]) if len(sys.argv) > 1 else sys.exit(__doc__)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |