Skip to content

rlaope/oh-my-hermes / skills/omh-inference-serving

omh-inference-serving

Verified

[omh] Self-hosted LLM serving on GPUs: choose the serving engine and quantization from decision tables, prepare deployment as an idempotent runbook with observed-only verification, and measure the endpoint with the standard TTFT/TPOT/goodput protocol. Use when the user says: inference-serving, inference serving, serve this model, serve the model, model serving, serving endpoint, vllm, llama.cpp.

Inference Serving

This is a Hermes-native inference-serving workflow skill.

Why This Exists

inference-serving exists so serving an LLM runs as one decided, gated, measured process instead of scattered flag folklore: the engine choice is a table, the deployment is an idempotent runbook whose only completion evidence is the observed verification, and the benchmark speaks the standard metric vocabulary.

Do Not Use When

  • A new model generation needs recognition, calibration, routing, and pricing onboarding; use omh-model-optimization.
  • The user wants their own machine's model routing or providers configured; use omh-model-setup.
  • The question is whether a coding runtime/executor can run at all; use omh-executor-runtime-readiness.
  • The goal is application or system performance rather than the serving endpoint itself; use ulw-perf.
  • The ask is producing the smaller weights -- GPTQ or AWQ calibration, pruning, or a distilled student -- and proving they still do the task; use omh-model-compression.

Examples

Good example:

  • Prompt: Serve Qwen on our two A100s for the team and tell me if prefix caching is worth turning on.
  • Expected behavior: Engine verdict (vLLM, TP dividing the head counts), quantization check, the k8s or docker runbook with its gates and verification, then the prefix-cache A/B protocol with hit-rate assumptions recorded - numbers only from observed runs.
  • Why: Serving plus a measured tuning question is exactly the decide-deploy-measure process this workflow owns.

Bad example:

  • Prompt: Just tell me the endpoint is fast enough, we already know it works.
  • Expected behavior: Refuse the unmeasured claim; run the benchmark protocol against the stated SLO or report the capacity question as unanswered.
  • Why: A fast-enough claim without a load shape and observed results is the folklore this skill replaces.

Completion Checklist

  • The engine/quantization verdict names the situation-table row it came from and the rejected options.
  • Every runbook step's status is prepared or observed, never assumed, and the port invariant was honored.
  • Benchmark numbers carry metrics, load shape, dataset, SLO, and saved metadata, or are not reported.
  • Anything the workflow started for measurement was stopped, and credentials never appear in artifacts.

Recovery Notes

  • If the hardware truth is unknown, probe it first (GPU inventory, VRAM) instead of assuming the engine.
  • If deployment verification fails, walk the failure ladder (toolkit, shared memory, permissions, token) before editing manifests.
  • If a benchmark misses the verify targets, go to the symptom->flag table and re-measure one change at a time.

Workflow Lane

  • Current lane: Research and company ops - research, signals, ops, and briefings.
  • If intent belongs to another lane, hand back to omh-routing or name the adjacent workflow.
  • Shared product, routing, compatibility, and evidence rules: skill_view(name='omh-routing', file_path='references/skill-common-rail.md').

When to Use

Use when a model needs to be served - engine and quantization chosen, docker or Kubernetes deployment prepared as a gated runbook, or the endpoint measured with the TTFT/TPOT/ITL/goodput protocol - and the user wants the process, not an ad-hoc command guess.

Strong routing signals: `inference-serving`, `inference serving`, `serve this model`, `serve the model`, `model serving`, `serving endpoint`, `vllm`, `llama.cpp`, `llama cpp`, `serve with vllm`, `deploy vllm`, `vllm deployment`, `serving benchmark`, `benchmark the endpoint`, `prefix caching benchmark`, `gguf quantization`, `which quantization`, `모델 서빙`, `모델 서빙해줘`, `모델 배포해서 서빙`, `서빙 벤치마크`, `vllm 배포`, `vllm 서빙`, `추론 서버 띄워줘`, `모델 띄워줘`

Catalog Metadata

Category: operations Phase: inference-serving Hermes role: operator Quality tier: observed-command-gated Reasoning demand: light

Quality bar:

  • Decide before deploying: engine from the situation table (vLLM for multi-user NVIDIA APIs, llama.cpp for CPU/Apple Silicon/edge, TensorRT-LLM only with ops budget), quantization to match (AWQ/GPTQ/FP8 vs the GGUF ladder with Q4_K_M default), tensor parallel head-divisible.
  • Deploy as the gated runbook: docker's three load-bearing flags (--ipc=host, HF cache mount, HF_TOKEN) or the Kubernetes five-step (secret gate, existing-deployment gate, apply, rollout+readiness verify, summary+smoke); the port invariant touches four places or it did not change the port.
  • Troubleshoot from the symptom table first - slow TTFT to prefix caching/chunked prefill, OOM to gpu-memory-utilization/max-model-len/quantization - before inventing flags.
  • Measure with the protocol: TTFT/TPOT/ITL/E2EL as mean/median/P99, goodput against an explicit SLO, one load shape per run, results saved with metadata; the full contract is omh-inference-serving/references/serving-bench.md.
  • Report observed-only: each runbook step is prepared until its command's exit status and output are seen.

Handoff policy:

Keep engine/quantization decisions, runbook preparation, and benchmark design in Hermes; the commands run through the operator's terminal with observed evidence, and repository changes (deploy manifests, benchmark harnesses) are coding work for the selected executor lane. A runbook or benchmark plan is prepared_not_observed until its commands' results are seen.

Required inputs:

  • the model id(s) and where the weights live (HF id, local path, gated or not)
  • the hardware truth: GPUs and VRAM, or CPU/Apple Silicon, and single- vs multi-user load
  • the delivery surface: docker, Kubernetes, or bare process, and the port/ingress constraints
  • for benchmarks: the SLO (TTFT/TPOT bounds) and the load shape the number must represent

Expected outputs:

  • engine and quantization verdict from the decision tables, with the rejected options named
  • deployment runbook with its gates (secret, existing-deployment), verification commands, and the four-places port invariant
  • benchmark plan naming metrics, load shape, dataset, and metadata to save
  • observed-only status: what ran, what was verified, what stays prepared

Artifact expectations:

  • serving decision and runbook per omh-inference-serving/references/serving-runbooks.md
  • benchmark protocol per omh-inference-serving/references/serving-bench.md
  • result files with metadata only after observed runs

Safety rules:

  • Never claim the server is up without the observed rollout/readiness or smoke-request evidence.
  • Never write credentials into runbooks or results; tokens are referenced (HF_TOKEN, a named secret), never inlined.
  • A healthy probe is not a benchmark; a benchmark number without its load shape and metadata is not reported.
  • If the workflow started a server for a benchmark, the workflow stops it.

Runtime Evidence

Record observed delegation results; otherwise return not_available or not_observed. Prepared OMH routing is not execution, review, CI, merge-readiness, or merge evidence.

  • Treat wrapper memory/context summaries as advisory local context, not proof of opaque Hermes memory reads or changes. Preserve workflow intent and stop conditions; verify before claiming completion. Reply in the user's own words and the host's own voice: its SOUL.md persona owns reply language, tone, speech level, and sentence endings, progress updates included (where it sets no language, use the one the user wrote in), and OMH shapes structure and content only. When the requested output is delivered, put any open points inside the result and end with no question or offer; only when the person asked to choose or decide and that choice is still theirs, end with one specific question about it, not a list, a generic offer, or a footer; the shared rail's Reply Language And Host Voice and Turn Ending sections hold the full rule.

Use Hermes-native subagent/delegation features when available: native subagents -> Hermes delegation when available, otherwise sequential lanes.

Shared product, compatibility, topology, memory, harness, and execution rules: skill_view(name='omh-routing', file_path='references/skill-common-rail.md'). Load it when applicable; otherwise name an unavailable capability.

writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall