Skip to content

alivirgo/Major-AI-Skills / plugins/mas-ai-workflows/skills/llm-cost-latency-benchmark

llm-cost-latency-benchmark

Verified

Benchmark AI workflow cost and latency with TTFT/TPOT/ITL/e2e percentiles, cache-aware token billing, goodput under SLOs, and a quality floor. Use before claiming a model, prompt, or infra change is cheaper or faster.

LLM Cost & Latency Benchmark AI Skill Guide (Claude)

Overview & Engine Architecture

Cost and latency are not one number. A claim is a tuple: workload × clock boundary × cache state × concurrency/arrival process × token mix × pricing snapshot × quality floor. Mixing those variables produces vendor-slide fiction. A response that is faster but fails acceptance is not an optimization win—it is a different product.

Claude operates as a Principal LLM Performance Economist, specializing in client-observed TTFT vs server TTFT, TPOT vs ITL, cache-cold vs cache-hot regimes, billable token categories (uncached input, cache read, cache write, output including reasoning), workflow wall-clock (retrieval + tools + retries + queue), percentile reporting, and SLO-qualified goodput.

┌─────────────────────────────────────────────────────────────┐
│              Cost × Latency Benchmark Stack                 │
│                                                             │
│  Workload contract                                          │
│  ├── Case population (prod-shaped) + quality floor          │
│  ├── Input/output length distributions (or fixed I/O)       │
│  ├── Concurrency + arrival process (closed vs open loop)    │
│  └── Cache policy: cold | warm | production-mix             │
│                                                             │
│  Timing planes (never conflate)                             │
│  ├── Client TTFT: send → first nonempty streamed token      │
│  ├── Prefill / queue / network (diagnostic breakdown)       │
│  ├── TPOT / ITL after first token                           │
│  └── Workflow e2e: user intent → final accepted artifact    │
│                                                             │
│  Cost planes                                                │
│  ├── Provider usage: input, cached_read, cache_write, out   │
│  ├── Non-token: tools, search, containers, retries, fails   │
│  └── Pricing manifest: URL + retrieved_at + model tier      │
│                                                             │
│  Decision                                                   │
│  ├── Percentiles + sample size + goodput@SLO                │
│  └── Compare only configs that meet the quality floor       │
└─────────────────────────────────────────────────────────────┘

When to use / when not to

Use when

  • Choosing models, regions, streaming settings, prompt-cache layouts, or concurrency limits.
  • Proving a “cheaper/faster” PR for prompts, packers, rerankers, or serving stacks.
  • Setting SLOs (interactive chat, voice, batch) and capacity planning.
  • Debugging why p50 looks fine but p99 destroys UX (queueing, cold start, long prefill).

Do not use when

  • Quality is undefined—run @prompt-regression-gate / @ai-evaluation-dataset first and set a quality floor.
  • Someone wants a single TPS headline without workload/cache/concurrency disclosure.
  • Extrapolating 10 synthetic short prompts to production guarantees.
  • Hard-coding remembered $/MTok as “current” without a dated pricing source.

Deep metric dictionary (formulas you must name)

Let request i be sent at client time s_i, first nonempty usable streamed token arrive at f_i, last token at e_i. Let O_i = provider-reported output/completion tokens (includes reasoning/formatting tokens when billed that way). Let I_i = total input/prompt tokens.

MetricFormula / definitionNotes that break comparisons
Client TTFTf_i − s_iIncludes network + queue + prefill. State client region.
Server TTFTreceipt → first token sentDifferent clock; do not swap with client TTFT.
E2E request latencye_i − s_iModel-call only unless labeled workflow e2e.
TPOT(e_i − f_i) / (O_i − 1)Undefined if O_i < 2. Excludes first token by convention (NVIDIA/vLLM-style).
ITLgaps t_{j+1}−t_jDistribution over intervals; p95 of all gaps ≠ mean of per-request TPOT.
Decode TPS (per user)(O_i − 1) / (e_i − f_i) = 1/TPOTNot full-request rate.
Full-request TPSO_i / (e_i − s_i)Includes TTFT; lower; different claim.
Aggregate TPSsum(O) / T_runFleet/GPU scope must be named.
Goodputcompliant completions / second or fraction meeting SLOName SLO (e.g. TTFT p99≤X ∧ TPOT p99≤Y); count errors as noncompliant.

First-token identity (reasoning models): declare whether f_i is (a) first SSE/data chunk, (b) first reasoning token, or (c) first visible answer token. Voice/chat UX usually cares about (c). Chunk ≠ token (common harness bug called out on r/LocalLLaMA tooling).

OpenTelemetry alignment (when instrumenting): prefer gen_ai.client.operation.time_to_first_chunk / gen_ai.usage.input_tokens / output_tokens, and cache detail attributes when available. Report billed units when providers distinguish billed vs consumed.


Cost accounting (think in billable categories)

Token classes (do not collapse)

Providers increasingly price separately:

  1. Uncached input — ordinary prompt tokens
  2. Cache read (cached_tokens) — discounted reuse of a prefix
  3. Cache write (cache_write_tokens) — may be >1× uncached input on some model families (e.g. OpenAI docs: writes at 1.25×, reads at 0.1× on newer families—verify live pricing)
  4. Output / completion — includes reasoning tokens and often invisible formatting/tool-channel tokens; billed as output even when not shown to the user

Arithmetic sketch (replace rates from a dated pricing pull):

cost ≈
  u_in  * rate_uncached_in
+ c_read * rate_cached_in
+ c_write * rate_cache_write
+ out   * rate_out
+ Σ tool_calls * tool_rate
+ Σ non_token_fees

Rules:

  • Prefer provider usage over local len/4 guesses (tools, images, schemas break heuristics).
  • If usage is missing → mark unknown, never invent.
  • Include failed and retried attempts in cost totals (they still billed or partially billed).
  • Stream with stream_options.include_usage (or provider equivalent) so finals are recorded.
  • Reasoning: budget max_output_tokens for hidden tokens + visible answer; otherwise empty completions skew latency/cost.

Pricing manifest (mandatory artifact)

{
  "retrieved_at": "2026-10-05T18:00:00Z",
  "source_url": "https://…/pricing",
  "model": "…",
  "currency": "USD",
  "rates_per_1m": {
    "input_uncached": null,
    "input_cached_read": null,
    "input_cache_write": null,
    "output": null
  },
  "notes": "Paste exact rows; do not rely on memory."
}

Unit economics to report

KPIDefinition
$/accepted taskTotal $ for all attempts / tasks that passed quality floor
$/1k input tokens (effective)Blended after cache hit mix
Cache hit ratecached_read / input (verify via usage fields, not hope)
Retry tax$ and latency of retries / successes
Cost at iso-qualityOnly among configs meeting the floor

Cheapest raw tokens with 2× retries and quality fails is usually more expensive per accepted task.


Workload contract (comparability gate)

Before trusting any A vs B claim, both sides must match—or differences must be labeled as the intentional experimental factor:

  1. Model + snapshot/version + tokenizer
  2. Prompt/tool schema hashes
  3. Input/output length policy (fixed I/O vs production distribution)
  4. Streaming on/off (TTFT/ITL meaningless without streaming)
  5. Cache regime: cold (force unique prefixes) vs warm (stable system prefix) vs measured mix
  6. Concurrency + arrival: closed-loop (send next after done) ≠ open-loop/Poisson ≠ “fire all at t=0”
  7. Client vantage (region/network)
  8. Warm-up policy (discard N cold starts; separate serverless cold-start study)
  9. Quality floor + same eval dataset hash
  10. Pricing date for cost claims

If any required field is missing → verdict cannot-normalize, not “approximately faster.”

Cache is a different experiment

Prefix/KV caching can dominate TTFT. Practitioners repeatedly show:

  • Accidental cache hits from reused prompts make TTFT look unrealistically low.
  • Calculated prefill savings ≠ measured TTFT (network/queue floor remains).
  • Production agents with growing tool histories live in prefill-bound regimes; chat short-prompt TPS rankings mislead.

Always report cold and warm separately, with measured cached_tokens / hit rate.

Concurrency and tails

  • p50 TTFT at concurrency=1 ≈ network + single prefill—almost useless for capacity.
  • Test near expected p95 concurrent load.
  • Under load, TTFT is often queue wait; log queue separately when the server exposes it.
  • For interactive/voice, p95/p99 matter more than mean; hedging (dual endpoint race) is an architectural response to tail risk, not a benchmark cheat—cost it explicitly (~2× input on hedged fraction).

Sample size honesty (IETF-style guidance): p99 needs large N (order ~10³ samples for rough relative accuracy); tiny benches must not claim p99 production SLOs.


Workflow vs model-call latency

For product decisions, measure workflow wall-clock:

user_submit → retrieve/tools/guards → model (TTFT…E2E) → validate/repair → accept

Attribute spans:

SpanTypical owners
Retrieval / rerank@rag-retrieval-audit
Tool round-tripsagent graph
Validation retries@llm-json-contract-check
Model TTFT/TPOTprovider/serving
Human-perceivedUX (streaming indicators reset abandonment clocks)

Optimizing only decode TPS while RAG prefill and tools dominate e2e is cargo-cult performance work.


Procedure

1. Preflight

  • Define population (prod log sample vs synthetic fixed I/O).
  • Set quality floor (pass rate / P0 gate).
  • Set SLOs (e.g. client TTFT p95 ≤ 800ms; TPOT p95 ≤ 40ms; or batch: maximize $/tok within overnight window).
  • Budget ($) and max duration.
  • Fetch live pricing → pricing manifest.
  • Choose cache regime(s) and concurrency sweep plan.

2. Instrument each attempt

Record per request (sanitized):

{
  "case_id": "c-17",
  "attempt": 1,
  "ok_quality": true,
  "error_type": null,
  "s_client": 0.0,
  "f_first_usable_token": 0.42,
  "e_last_token": 1.88,
  "ttft_ms": 420,
  "tpot_ms": 28.5,
  "usage": {
    "input": 3500,
    "cached_read": 3200,
    "cache_write": 0,
    "output": 180,
    "reasoning_output_subset": 40
  },
  "cost_usd": null,
  "cache_regime": "warm",
  "concurrency_bin": 16,
  "model": "…",
  "pricing_retrieved_at": "…"
}

3. Analyze

  • Percentiles p50/p95/p99 for TTFT, TPOT, workflow e2e—by cache regime and concurrency.
  • Cost totals and $/accepted.
  • Goodput @ declared SLO.
  • Error/retry rates.
  • Reject cross-regime averages that hide cold starts.

4. Decide

Only compare configs that clear the quality floor. Prefer:

  1. Higher goodput at equal or lower $/accepted, or
  2. Lower latency percentiles without quality loss, or
  3. Explicit Pareto: “+12% cost for −40% p99 TTFT” with product sign-off.

Deliver raw measurements + config + pricing refs + arithmetic. Label extrapolation limits.


Production example: percentile + cost rollup

"""Cost/latency rollup with TTFT, TPOT, and dated pricing."""
from __future__ import annotations

import json
import math
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Sequence


def percentile(sorted_vals: Sequence[float], p: float) -> float:
    """Nearest-rank percentile; p in [0, 100]. Empty → nan."""
    if not sorted_vals:
        return float("nan")
    if p <= 0:
        return sorted_vals[0]
    if p >= 100:
        return sorted_vals[-1]
    k = max(1, int(math.ceil(p / 100.0 * len(sorted_vals))))
    return sorted_vals[k - 1]


@dataclass(frozen=True)
class Rates:
    """USD per 1M tokens. None means unknown—do not invent."""
    input_uncached: float | None
    input_cached_read: float | None
    input_cache_write: float | None
    output: float | None


def cost_usd(usage: dict[str, int], rates: Rates) -> float | None:
    need = {
        "input_uncached": rates.input_uncached,
        "input_cached_read": rates.input_cached_read,
        "input_cache_write": rates.input_cache_write,
        "output": rates.output,
    }
    if any(v is None for v in need.values()):
        return None
    inp = usage.get("input", 0)
    cread = usage.get("cached_read", 0)
    cwrite = usage.get("cache_write", 0)
    # Uncached portion = input - cached_read - cache_write (clamp at 0)
    u = max(0, inp - cread - cwrite)
    out = usage.get("output", 0)
    return (
        u * rates.input_uncached
        + cread * rates.input_cached_read
        + cwrite * rates.input_cache_write
        + out * rates.output
    ) / 1_000_000.0


def tpot_ms(ttft_ms: float, e2e_ms: float, output_tokens: int) -> float:
    if output_tokens < 2:
        return float("nan")
    return (e2e_ms - ttft_ms) / (output_tokens - 1)


def summarize(rows: list[dict[str, Any]], rates: Rates, slo_ttft_ms: float) -> dict[str, Any]:
    ttfts = sorted(r["ttft_ms"] for r in rows if r.get("ttft_ms") is not None)
    tpots = sorted(
        tpot_ms(r["ttft_ms"], r["e2e_ms"], r["usage"]["output"])
        for r in rows
        if r.get("ttft_ms") is not None and r.get("e2e_ms") is not None
    )
    tpots = [x for x in tpots if x == x]

    costs = []
    unknown_cost = 0
    for r in rows:
        c = cost_usd(r["usage"], rates)
        if c is None:
            unknown_cost += 1
        else:
            costs.append(c)

    accepted = [r for r in rows if r.get("ok_quality")]
    good = [r for r in accepted if r["ttft_ms"] <= slo_ttft_ms]

    return {
        "n": len(rows),
        "ttft_p50": percentile(ttfts, 50),
        "ttft_p95": percentile(ttfts, 95),
        "ttft_p99": percentile(ttfts, 99),
        "tpot_p50": percentile(tpots, 50),
        "tpot_p95": percentile(tpots, 95),
        "cost_total_usd": sum(costs) if costs else None,
        "cost_unknown_rows": unknown_cost,
        "cost_per_accepted_usd": (sum(costs) / len(accepted)) if costs and accepted else None,
        "goodput_fraction_ttft_slo": (len(good) / len(rows)) if rows else float("nan"),
        "quality_pass_rate": (len(accepted) / len(rows)) if rows else float("nan"),
    }


# Load JSONL attempts, attach pricing from manifest, print summarize().

Minimal experiment matrix (start here)

CellPurpose
Fixed 0.5k/0.5k I/O, c=1, cache-coldPure model responsiveness
Same, cache-warmPrefix-cache benefit
Prod-length distribution, c≈p95 loadCapacity reality
Full workflow e2eProduct truth
+ quality floor filterEconomic truth

Technical troubleshooting matrix

SignatureDeep causeWhat to do
Amazing TTFT in bench, bad in prodAccidental prefix cache / short promptsForce unique prefixes; match prod lengths; report hit rate
p50 fine, p99 awfulQueueing, throttling, noisy neighborsCorrelate spikes with RPS vs wall clock; raise replicas or shed load; consider hedge
High TPS, unhappy usersOptimizing aggregate decode, ignoring TTFT SLOTrack goodput; plot latency vs offered load
“Cheaper model” costs moreMore output/reasoning/retriesOptimize $/accepted and output tokens, not $/MTok sticker
Empty/fast failuresReasoning burned max_tokensRaise output budget; classify as errors not wins
Serverless look slowCold start (pull/CUDA/weights)Separate cold-start study; warm pool for interactive
TTFT ≈ queueSaturationCapacity problem, not “slow model”
Cross-provider bakeoff chaosDifferent clocks/tokenizers/cacheComparability gate; cannot-normalize if mismatched
Client harness overheadSync client dominatingAsync load gen; note NVIDIA warnings on client overhead
Cost unknownStreaming without usageEnable usage on stream; else mark unknown

Best practices

  1. Quality floor first—then latency—then cost among survivors.
  2. Report percentiles, never mean-only for SLOs.
  3. Separate cold/warm cache and model-only vs workflow e2e.
  4. Pin and date pricing; re-pull when claiming savings.
  5. Include retries, failures, tool fees in cost.
  6. Name every TPS/TTFT definition in the report title.
  7. For p99 SLOs, collect enough samples or refuse the claim.
  8. Align product class: interactive (TTFT/ITL), RAG/agents (prefill + tools), batch (throughput/$).
  9. Prefer provider usage + OTel GenAI attributes for long-term dashboards.
  10. When changing prompts for cache locality, re-run @prompt-regression-gate—cache wins that break quality are not wins.

Limitations

  • Third-party latency leaderboards rarely match your prompt length, tools, and cache mix.
  • This skill does not replace provider SLAs or capacity postmortems.
  • Tokenization differs across models—iso-character prompts are not iso-token.
  • Stop and ask if quality floor, workload shape, or pricing source is missing.

Related skills

  • @prompt-regression-gate — quality floor / paired go-no-go
  • @ai-evaluation-dataset — representative case population
  • @rag-retrieval-audit — retrieval latency vs generation
  • @llm-json-contract-check — repair retries inflate cost/latency
  • @llm-cost-latency-benchmark — this skill
  • @vllm / provider skills — serving knobs (batching, prefix cache)

Agent Operational Directive

MANDATORY: Treat every cost/latency claim as a workload+clock+cache+concurrency+pricing+quality tuple. Measure client TTFT and TPOT/ITL with named formulas. Bill all token classes and failures honestly; date the pricing source. Never call a quality regression an optimization. Prefer goodput and $/accepted over vanity TPS. If comparability fields are missing, say cannot-normalize.


Source anchors (research)

  • Anyscale / vLLM metric definitions (TTFT, TPOT, ITL, goodput)
  • NVIDIA / Artificial Analysis / GPUSmith: clock boundaries, TPS ambiguity, cache disclosure
  • OpenAI: prompt caching billable categories; reasoning tokens billed as output; dated pricing page
  • OpenTelemetry GenAI: time_to_first_chunk, token usage (+ cache attributes), billed vs consumed
  • IETF draft LLM benchmarking methodology: percentiles, sample sizes, SLO capacity search
  • r/LLMDevs & r/LocalLLaMA: p99 tails, hedging, prefill-bound agents, cold starts, false TTFT from first chunk / cache reuse
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall