Skip to content

sybil-solutions/inference-optimisation-skill / skills/inference-optimisation

inference-optimisation

Unverified

Hill-climb the serving speed of a local LLM on any hardware (NVIDIA, AMD, Intel, Apple, unified-memory laptops, multi-GPU) and any quantization (GGUF, EXL2/EXL3, AWQ/GPTQ, FP8, MLX, BF16): single-user and concurrent decode tok/s, speculative decoding, prefill tok/s, KV cache size, startup time and TTFT, under a memory cap and with full-context stability. Computes the theoretical ceiling first, runs noise-controlled A/B experiments, keeps a ledger, and qualifies the winner before publishing it. Use when asked to optimise, tune, speed up, benchmark or hill-climb a model server (vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, TensorRT-LLM...), to find the maximum possible tok/s on a machine, or to publish a tuned recipe. Triggers: hill climb, tok/s, decode speed, prefill speed, MTP, EAGLE, draft model, speculative decoding, KV cache, TTFT, startup time, theoretical maximum.

FileKindSize
SKILL.mdskill md9.8 KB
examples/ceiling-ornith-b390.jsonconfig469 B
examples/config-existing-server.jsonconfig173 B
examples/config-sglang-docker.jsonconfig1.3 KB
references/ceilings.mdmarkdown4.3 KB
references/gotchas.mdmarkdown3.8 KB
references/levers.mdmarkdown6.6 KB
references/methodology.mdmarkdown4.5 KB
references/publishing.mdmarkdown3.5 KB
scripts/bench.pyexecutablescript12.5 KB
scripts/ceiling.pyexecutablescript2.6 KB
scripts/mem.shexecutablescript1.6 KB
scripts/queue.shexecutablescript786 B

scripts/mem.sh

View on GitHub
#!/usr/bin/env bash
# Print the real memory footprint (bytes) of a containerised model server, for bench.py's memory watchdog.
#   mem.sh CONTAINER [unified]
# Discrete NVIDIA GPU: sums nvidia-smi used memory of the container's processes + container anon RSS.
# Unified memory (Intel/AMD iGPU, Linux DRM drivers): sums drm-resident-* (deduplicated by drm-client-id)
# over the container's processes + cgroup anon. Reading other users' fdinfo needs root: run the watchdog
# with passwordless sudo for this script, or as root. Never embed a password here.
set -euo pipefail
c=${1:?container name}; mode=${2:-auto}
pids=$(docker top "$c" -eo pid 2>/dev/null | tail -n +2 | tr '\n' ' ')
[ -z "$pids" ] && { echo 0; exit 0; }
cid=$(docker inspect -f '{{.Id}}' "$c")
cg=$(ls -d /sys/fs/cgroup/system.slice/docker-"$cid".scope /sys/fs/cgroup/docker/"$cid" 2>/dev/null | head -1 || true)
anon=$(awk '/^anon /{print $2}' "$cg/memory.stat" 2>/dev/null || echo 0)
gpu=0
if [ "$mode" != unified ] && command -v nvidia-smi >/dev/null; then
  for p in $pids; do
    m=$(nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader,nounits | awk -F', ' -v p="$p" '$1==p{print $2}')
    gpu=$((gpu + ${m:-0} * 1048576))
  done
else
  declare -A seen
  for p in $pids; do
    for f in /proc/"$p"/fdinfo/*; do
      id=$(awk '/drm-client-id/{print $2}' "$f" 2>/dev/null) || continue
      [ -z "$id" ] && continue; [ -n "${seen[$id]:-}" ] && continue; seen[$id]=1
      kb=$(awk '/drm-resident-/{s+=$2} END{print s+0}' "$f"); gpu=$((gpu + kb * 1024))
    done
  done
fi
echo $((gpu + ${anon:-0}))
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 196kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 193kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 193kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 193kUpdated Claude CodeCodexVerifiedInstall