sybil-solutions/inference-optimisation-skill / skills/inference-optimisation
inference-optimisation
UnverifiedHill-climb the serving speed of a local LLM on any hardware (NVIDIA, AMD, Intel, Apple, unified-memory laptops, multi-GPU) and any quantization (GGUF, EXL2/EXL3, AWQ/GPTQ, FP8, MLX, BF16): single-user and concurrent decode tok/s, speculative decoding, prefill tok/s, KV cache size, startup time and TTFT, under a memory cap and with full-context stability. Computes the theoretical ceiling first, runs noise-controlled A/B experiments, keeps a ledger, and qualifies the winner before publishing it. Use when asked to optimise, tune, speed up, benchmark or hill-climb a model server (vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, TensorRT-LLM...), to find the maximum possible tok/s on a machine, or to publish a tuned recipe. Triggers: hill climb, tok/s, decode speed, prefill speed, MTP, EAGLE, draft model, speculative decoding, KV cache, TTFT, startup time, theoretical maximum.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.8 KB |
| examples/ceiling-ornith-b390.json | config | 469 B |
| examples/config-existing-server.json | config | 173 B |
| examples/config-sglang-docker.json | config | 1.3 KB |
| references/ceilings.md | markdown | 4.3 KB |
| references/gotchas.md | markdown | 3.8 KB |
| references/levers.md | markdown | 6.6 KB |
| references/methodology.md | markdown | 4.5 KB |
| references/publishing.md | markdown | 3.5 KB |
| scripts/bench.pyexecutable | script | 12.5 KB |
| scripts/ceiling.pyexecutable | script | 2.6 KB |
| scripts/mem.shexecutable | script | 1.6 KB |
| scripts/queue.shexecutable | script | 786 B |
scripts/mem.sh
View on GitHub#!/usr/bin/env bash
# Print the real memory footprint (bytes) of a containerised model server, for bench.py's memory watchdog.
# mem.sh CONTAINER [unified]
# Discrete NVIDIA GPU: sums nvidia-smi used memory of the container's processes + container anon RSS.
# Unified memory (Intel/AMD iGPU, Linux DRM drivers): sums drm-resident-* (deduplicated by drm-client-id)
# over the container's processes + cgroup anon. Reading other users' fdinfo needs root: run the watchdog
# with passwordless sudo for this script, or as root. Never embed a password here.
set -euo pipefail
c=${1:?container name}; mode=${2:-auto}
pids=$(docker top "$c" -eo pid 2>/dev/null | tail -n +2 | tr '\n' ' ')
[ -z "$pids" ] && { echo 0; exit 0; }
cid=$(docker inspect -f '{{.Id}}' "$c")
cg=$(ls -d /sys/fs/cgroup/system.slice/docker-"$cid".scope /sys/fs/cgroup/docker/"$cid" 2>/dev/null | head -1 || true)
anon=$(awk '/^anon /{print $2}' "$cg/memory.stat" 2>/dev/null || echo 0)
gpu=0
if [ "$mode" != unified ] && command -v nvidia-smi >/dev/null; then
for p in $pids; do
m=$(nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader,nounits | awk -F', ' -v p="$p" '$1==p{print $2}')
gpu=$((gpu + ${m:-0} * 1048576))
done
else
declare -A seen
for p in $pids; do
for f in /proc/"$p"/fdinfo/*; do
id=$(awk '/drm-client-id/{print $2}' "$f" 2>/dev/null) || continue
[ -z "$id" ] && continue; [ -n "${seen[$id]:-}" ] && continue; seen[$id]=1
kb=$(awk '/drm-resident-/{s+=$2} END{print s+0}' "$f"); gpu=$((gpu + kb * 1024))
done
done
fi
echo $((gpu + ${anon:-0}))
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 193k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 193k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 193k | Updated | Claude CodeCodex | Verified | Install |