Skip to content

ericrisco/rsc-harness / skills/llm-pipeline

llm-pipeline

Unverified

Use when wiring several LLM calls into a production flow: typed contracts between steps, a router/gateway so 429s, timeouts and outages fail over instead of taking you down, and cost control via caching, model tiers and abort caps. NOT single-prompt wording (that is `prompt-engineering`), NOT a model-driven tool loop (that is `building-agents`).

llm-pipeline

Wire multiple LLM calls into a reliable, controllable production pipeline. You chain steps where one call's validated output feeds the next, put a router in front of providers so an outage fails over instead of taking you down, and engineer the cross-cutting concerns: timeouts, bounded retries, fallbacks, caching, and cost caps.

Treat the LLM as an unreliable network dependency, not a local function call. Every rule below follows from that: providers have outages, rate limits, and latency tails, so no single provider is a single point of failure and no call is allowed to run unbounded.

Do you even need a pipeline?

This skill is the orchestration around calls. If you only have one call, you are in the wrong place.

SituationGo to
Make one prompt better, few-shot, system-prompt design../prompt-engineering/SKILL.md
One call must return a typed object validated against a schema../structured-extraction/SKILL.md
The model decides its own next step / tool to call../building-agents/SKILL.md
Chunk/embed/retrieve context to stuff into a prompt../rag/SKILL.md
Pure spend ledger / attribution / dashboard../cost-tracking/SKILL.md
Fixed multi-step flow + reliability layerhere

A pipeline is a DAG you designed. The moment the model picks its own next step, it is an agent — go build that instead.

Design the chain as a typed DAG

Each step is a pure-ish function: (typed input) -> (typed output via structured output). Chaining small single-purpose steps beats one mega-prompt — reported ~20% output-quality gain — because each step is debuggable, cacheable, and retryable in isolation.

Rules:

  • The structured output of step N is the input contract of step N+1. Validate it (Pydantic / JSON Schema) at the seam. A schema-valid object that fails validation here never poisons the next call.
  • Keep steps small and single-purpose. "Extract entities" and "classify sentiment" are two steps, not one prompt doing both. Smaller steps route to cheaper models and cache better.
  • Mark independent steps for parallel fan-out. If step B and step C both only need step A's output, run them concurrently — see ../parallel/SKILL.md. Sequential only where there is a real data dependency.
  • Tag each step idempotent or side-effecting. Retries and replays must be safe; a step that writes to a DB or sends an email is not safe to blindly retry.
# Bad: free text flows between steps; step 2 silently mis-parses step 1
entities = client.responses.create(model="gpt-4o", input=f"Extract entities: {doc}").output_text
summary  = client.responses.create(model="gpt-4o", input=f"Summarize for {entities}").output_text

# Good: each step emits a validated object; the seam is a contract you assert on
from pydantic import BaseModel

class Entities(BaseModel):
    people: list[str]
    orgs: list[str]

step1 = client.responses.parse(model="gpt-4o", input=f"Extract: {doc}",
                               text_format=Entities, timeout=30)
ents: Entities = step1.output_parsed          # parse fails HERE, not downstream
summary = summarize(ents)                      # typed input, not a string blob

Put a gateway/router in front

Use a router so a failure on one deployment fails over to another, and so you can swap or load-balance models from config without touching call sites.

LiteLLM is the de-facto open-source LLM gateway (current stable line v1.83.3-stable). It exposes a unified OpenAI-format completion() across 100+ providers, with built-in retry/fallback, cost tracking, and budget management — usable as a Python SDK or as a Proxy Server.

Fallback semantics worth knowing: a request to an order=1 deployment that fails (connection error, 404, 429, ...) auto-tries order=2, then order=3; each order level gets its own num_retries before escalating; exhausting orders falls through to configured fallbacks. There are specialized buckets — content_policy_fallbacks (ContentPolicyViolationError), context_window_fallbacks (ContextWindowExceededError), and default_fallbacks.

from litellm import Router

router = Router(
    model_list=[
        {"model_name": "smart",
         "litellm_params": {"model": "anthropic/claude-sonnet-4-6", "timeout": 30}},
        {"model_name": "smart-backup",
         "litellm_params": {"model": "openai/gpt-4o", "timeout": 30}},
    ],
    fallbacks=[{"smart": ["smart-backup"]}],
    context_window_fallbacks=[{"smart": ["smart-backup"]}],
    num_retries=2,            # per order level, bounded
    timeout=30,               # hard cap, never unbounded
)
resp = router.completion(model="smart",
                         messages=[{"role": "user", "content": prompt}])

Build vs buy: raw SDK + a thin retry/timeout wrapper (fewest deps, fine for one provider) → LiteLLM SDK/proxy (widest provider coverage, fallbacks for free) → self-hosted open-source gateway (Bifrost: Go, Apache-2.0, per-key budgets and rate limits) → hosted gateway (Portkey, when you want it operated for you). Full config — all fallback buckets, redis cache params, budget/rate-limit settings, cost callbacks — is in references/litellm-router.md.

Reliability mechanics

  • Always set a timeout. A 30–60s hard cap per call. Without it, one slow upstream hangs the request indefinitely.
  • Bounded retries with exponential backoff + jitter. LiteLLM defaults: backoff from INITIAL_RETRY_DELAY 0.2s up to MAX_RETRY_DELAY 10s, with jitter to avoid thundering herds. Never while True.
  • Only retry idempotent steps. Retrying a step that sent an email or charged a card double-fires. Gate retries behind the idempotency tag from the DAG design.
  • Circuit-break and degrade gracefully. When a provider is down and fallbacks are exhausted, return a partial result, a cached result, or a cheaper-model result — never hang and never 500 the user if a degraded answer exists.
# Bad: naked call in a for-loop; no timeout, unbounded effect, no fallback
for _ in range(10000):
    try:
        return client.chat.completions.create(model="gpt-4o", messages=msgs)
    except Exception:
        continue          # hammers a down provider, blows the budget, may never exit

# Good: router does bounded retries + backoff + fallback; you degrade on exhaustion
try:
    return router.completion(model="smart", messages=msgs)   # timeout + num_retries set
except Exception:
    return cached_or_cheaper_answer(msgs)                     # graceful degradation

Caching: two distinct layers

The cheapest, fastest, most reliable call is the one you never made — so cache and tier before you tune prompts. Wording is the last lever, not the first.

LayerWhat it matchesSafetyEnable when
Prefix / prompt cache (provider-native)Exact prefix of the promptAlways safe (same input → same cached compute)Always; put the stable prefix first
Semantic cache (your gateway)Embedding-similar prior queryRisky: weak embedder → false hitsParaphrased FAQ-style queries, approximate answers OK

Prefix cache is transparent and free to enable. OpenAI caches automatically at ~50% off cached input tokens, no write penalty, no storage fee — first request full price, prefix hits half price. Anthropic is explicit (you mark cache breakpoints) and deeper: cache read = 0.1× base input (~90% off), 5-minute write = 1.25× base, 1-hour write = 2× base, delivering ~90% cost and ~85% latency reduction on long stable prefixes. For both, put the stable content first (system prompt, instructions, fixed context) and the variable content last so the prefix matches.

Semantic cache matches an embedding-similar prior query and returns that prior response. The quality is dominated by the embedding model — a weak embedder produces false cache hits: a confidently wrong answer for a similar-but-different question (GPTCache is documented returning incorrect saved responses for similar prompts). So: strong embedder, tuned similarity threshold, and never on correctness-critical paths. Threshold tuning, embedding choice, TTL, and the multi-tier semantic → prefix → inference order are in references/caching-layers.md.

Cost & latency control

  • Model-tier routing, cheap-first. Run the cheap tier, escalate only on low confidence or detected complexity. Anchor prices (Anthropic, 2026): Haiku 4.5 $1/$5 per M in/out, Sonnet 4.6 $3/$15, Opus 4.7 $5/$25 — flagship-for-everything is 5× the cost for zero quality gain on easy calls.
  • Budget caps that ABORT, not just log. Per-request and per-tenant caps that kill a runaway loop. A logged-but-uncapped budget still lets a bug spend $10k overnight.
  • Token and latency budgets per step, and stream output for perceived latency on user-facing calls.

The ledger — attribution, per-team dashboards, monthly reporting — is not this skill; that is ../cost-tracking/SKILL.md. This skill owns the controls (tiers, caching, caps that abort).

Observability hooks

Log per step, every call: model, tokens_in/out, cost, latency_ms, cache_hit, fallback_used, retry_count. These are exactly the fields you debug a production incident from ("why did p99 spike?" → fallback_used + retry_count). Wire them into the tracing backbone in ../observability/SKILL.md, and measure output quality with ../agent-eval/SKILL.md — logging is not evaluation.

Anti-patterns

Anti-patternWhy it bitesDo instead
Naked call, no timeoutOne hung provider stalls the whole requestHard timeout on every call (30–60s)
Unbounded while True retryThundering herd, blown budget, infinite hangBounded retries + exp backoff (0.2s→10s) + jitter
Retrying a side-effecting stepDouble-writes, double-chargesIdempotency tag; only retry pure steps
Free text between stepsStep N+1 silently mis-parsesValidated structured output as the contract
Semantic cache with a weak embedderConfident WRONG answers from false hitsStrong embedder + tuned threshold, or prefix cache only
Flagship model for everything5–25× the cost, no quality gain on easy callsTier routing, cheap-first, escalate
No fallback configuredProvider outage = your outageRouter model group + fallbacks
Treating schema-valid as correctPerfectly-shaped wrong answers shipValidate semantics + eval (agent-eval)

Verify

Run scripts/verify.sh <file-or-dir> against your pipeline/gateway code. It is offline and read-only, and checks statically: every completion/chat call site has an explicit timeout, retries are bounded (no while True retry loops), at least one fallback is configured when a router/model_list is present, no hardcoded sk-/provider key literals (must be env-sourced), and any YAML/JSON config parses and lists ≥2 model entries so a fallback target exists. It prints PASS/FAIL per check and exits non-zero on any FAIL; an empty or clean target exits 0.

writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall