Skip to content

rlaope/oh-my-hermes / skills/omh-inference-serving

omh-inference-serving

Verified

[omh] Self-hosted LLM serving on GPUs: choose the serving engine and quantization from decision tables, prepare deployment as an idempotent runbook with observed-only verification, and measure the endpoint with the standard TTFT/TPOT/goodput protocol. Use when the user says: inference-serving, inference serving, serve this model, serve the model, model serving, serving endpoint, vllm, llama.cpp.

FileKindSize
SKILL.mdskill md8.9 KB
references/serving-bench.mdmarkdown2.2 KB
references/serving-runbooks.mdmarkdown5.8 KB

references/serving-bench.md

View on GitHub
# Serving Benchmark Protocol

The measurement contract for a serving endpoint. A number without its load
shape, dataset, and metadata is an anecdote, not a benchmark.

## Metric vocabulary

- **TTFT** - time to first token; the interactivity metric.
- **TPOT** - time per output token after the first; the streaming-rate metric.
- **ITL** - inter-token latency distribution; jitter the user feels.
- **E2EL** - end-to-end request latency.
- Report each as mean, median, and P99 - a mean alone hides the tail.
- **Goodput** - the fraction of requests meeting an explicit SLO, stated like
  `ttft:500 tpot:50` (milliseconds). Throughput without an SLO rewards
  batching that ruins latency.

## Load shapes

Choose the shape before running anything, and name it in the results: infinite
burst (capacity ceiling), Poisson arrival at a target rate (steady state),
burstiness below 1 (spiky traffic), linear ramp between two rates (finding the
knee), and a concurrency cap (client-side backpressure). One shape per run;
mixed shapes measure nothing.

## Prefix-cache protocol

Three acceptable designs: (A) offline A/B - the same fixed-prompt workload
with prefix caching on, then off, with the repeat count controlling expected
hit rate; (B) a real shared-prefix corpus; (C) online with a synthetic
prefix-repetition dataset, whose four knobs (prefix length, suffix length,
number of prefixes, output length) are all recorded. A cache result without
its hit-rate assumption is not comparable to anything.

## Hygiene

- Save results as files with metadata (`version`, `tp`, model, quantization,
  load shape) so two runs can be compared without archaeology.
- The chat-completions backend pairs with the chat endpoint; mixing the
  completions endpoint into a chat benchmark invalidates TTFT.
- If the benchmark started the server, the benchmark stops the server.
- Verify targets before tuning: TTFT under ~500ms on short prompts and GPU
  utilization above ~80% are the usual first bars; a run that misses them
  goes to the symptom->flag table before any deeper tuning.

## Boundary

A benchmark plan is prepared_not_observed; only saved result files from
observed runs are measurement evidence, and one run is a sample, never a
capacity guarantee, review, CI, or merge evidence.
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall