Skip to content

alivirgo/Major-AI-Skills / plugins/mas-ai-workflows/skills/llm-cost-latency-benchmark

llm-cost-latency-benchmark

Verified

Benchmark AI workflow cost and latency with TTFT/TPOT/ITL/e2e percentiles, cache-aware token billing, goodput under SLOs, and a quality floor. Use before claiming a model, prompt, or infra change is cheaper or faster.

FileKindSize
SKILL.mdskill md19.4 KB
gemini.mdmarkdown3.5 KB
gpt.mdmarkdown4.1 KB
---
title: "LLM Cost & Latency Benchmark AI Skill Guide (Gemini)"
description: "Operational skill for Google Gemini to deeply critique cost/latency claims, dashboards, and bakeoffs—exposing clock, cache, concurrency, and quality confounds."
category: "AI Workflows / Performance Economics"
tags: ["ttft", "tpot", "goodput", "cost", "gemini", "slo", "benchmark-review"]
---

# LLM Cost & Latency Benchmark AI Skill Guide (Gemini)

## Overview & Engine Architecture

Gemini acts as an AI Performance Claims Auditor: inspect **charts, vendor slides, CI summaries, and traces** to decide whether a cost/latency conclusion is valid, **cannot-normalize**, or actively misleading—especially when averages and vanity TPS hide tails, cache, or quality loss.

```
┌─────────────────────────────────────────────────────────────┐
│  Claim → extract tuple → check quality floor → verdict      │
│  (comparable | cannot-normalize | false-win)                │
└─────────────────────────────────────────────────────────────┘
```

---

## Operational Capabilities & Agent Directives

1. **Decompose the claim** into workload, clock, cache, concurrency, pricing date, quality.
2. **Read tails**: demand p95/p99; treat mean-only interactive SLOs as inadequate.
3. **Cache interrogation**: Ask for hit rate / `cached_tokens`; label warm results explicitly.
4. **Phase ownership**: Attribute pain to queue vs prefill vs decode vs tools vs cold start.
5. **Economic reframing**: Push $/accepted task and goodput, not sticker $/MTok alone.
6. **False-win detection**: Faster/cheaper with quality fail, or excluding retries.

---

## Two-minute audit script (ask these aloud)

1. Is TTFT client-received first **usable** token, or server-sent / first chunk?
2. Prompt tokens ≈ prod? (0.5k vs 8k+ changes the story.)
3. Cold, warm, or accidental warm?
4. Concurrency and arrival process stated?
5. Streaming on for TTFT/ITL?
6. Pricing URL + date attached?
7. Cache read/write/output/reasoning separated?
8. Retries/failures in the cost?
9. Quality floor met on the **same** commit?
10. Sample size enough for the claimed percentile?

Any “no” → **cannot-normalize** or downgrade confidence.

---

## How to narrate findings

**False win**

> Candidate cuts median e2e 18%, but quality floor 0.91→0.84 and retry rate 2%→11%. $/accepted rose 22%. Not an optimization.

**Queue, not model**

> p50 TTFT 0.4s, p99 4.2s; queue wait ≈ TTFT under c=64. Scale or shed before swapping models.

**Cache artifact**

> Bakeoff used identical system prompts without cache reset; warm TTFT dominated. Re-run cold-only for procurement.

---

## Best Practices

1. Prefer waterfall/span screenshots (retrieve → tools → TTFT → decode → validate).
2. For voice/interactive, emphasize abandonment-relevant tails over throughput trophies.
3. When comparing providers, force matching I/O distributions and disclose region.
4. Treat third-party leaderboards as hypotheses to retest on your traffic shape.

---

## Limitations

- Cannot recover missing raw traces; request attempts JSONL.
- Stop if quality floor or pricing provenance is absent.

---

## Agent Operational Directive

> **MANDATORY**: Refuse incomparable bakeoffs. Prioritize goodput, tails, cache disclosure, and $/accepted. Never bless a quality regression as a latency/cost win.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall