Skip to content

ericrisco/rsc-harness / skills/building-agents

building-agents

Unverified

Use when building or restructuring an LLM agent — provider adapter, tool calling, structured output, RAG, agent loop, eval gate, cost routing, tracing, MCP server — model-agnostic across OpenAI/Anthropic/Gemini/OSS so a model swap is a config change. NOT vector-store SQL alone (that is `postgresdb`) or service deployment (that is `deployment`).

FileKindSize
SKILL.mdskill md19.4 KB
evals/README.mdmarkdown2.9 KB
evals/cases.yamlconfig6.3 KB
references/agent-loops-and-harness.mdmarkdown14.4 KB
references/evals-and-observability.mdmarkdown15.6 KB
references/mcp-servers.mdmarkdown11.6 KB
references/provider-abstraction.mdmarkdown20.8 KB
references/tools-and-rag.mdmarkdown15.6 KB
scripts/verify.shexecutablescript4.6 KB

references/evals-and-observability.md

View on GitHub
# Evals, tracing & cost control — the production line

An LLM feature without an eval gate is a demo. This file ships a runnable, provider-
agnostic eval harness (the one `scripts/verify.sh` dry-runs), an LLM-as-judge that is
itself model-agnostic, OTel GenAI tracing, and a closed-loop cost controller: caching,
batching, routing/cascades, and hard budgets. Python 3.12+, Pydantic v2.

## Eval-first

Define success criteria **before** building. Two complementary suites:

- **Capability evals** — can the agent do the new thing at all? Built once, when you add
  a feature.
- **Regression evals** — does a prompt/model/code change break what already worked? Run
  on every PR against a stored baseline.

Both reduce to: a golden set, graders, metrics, and a threshold that gates merge.

## Golden sets

JSONL, one case per line, versioned as fixtures next to the code:

```json
{"input": "Refund order ord_8123 for 500 cents, defective", "expected": {"tool": "refund", "order_id": "ord_8123"}, "metadata": {"suite": "capability", "tags": ["refund"]}}
```

```python
import json
from pathlib import Path

from pydantic import BaseModel


class Case(BaseModel):
    input: str
    expected: dict
    metadata: dict = {}


def load_golden(path: str | Path) -> list[Case]:
    return [Case.model_validate_json(line) for line in Path(path).read_text().splitlines() if line.strip()]
```

Build the golden set from **real traffic** (sampled, labeled), not invented prompts; keep
a strict train/eval split so prompts can't be tuned against the eval set (a form of
leakage). Version it: a golden set is a first-class artifact, reviewed like code.

## Graders

Three grader kinds, one interface. The LLM-as-judge uses the same `LLMProvider`, so the
judge is provider-agnostic and can run on a *different* model than the candidate.

```python
from typing import Protocol

from provider_abstraction import CompletionRequest, LLMProvider, Message
from provider_abstraction import complete_structured


class Grader(Protocol):
    name: str
    def grade(self, case: Case, output: str) -> float: ...   # 0.0–1.0


class ExactGrader:
    name = "exact"
    def grade(self, case: Case, output: str) -> float:
        return float(output.strip() == str(case.expected.get("text", "")).strip())


class SchemaGrader:
    name = "schema"
    def __init__(self, schema: type[BaseModel]) -> None:
        self.schema = schema
    def grade(self, case: Case, output: str) -> float:
        try:
            self.schema.model_validate_json(output)
            return 1.0
        except Exception:
            return 0.0


class Verdict(BaseModel):
    score: float          # 0.0–1.0
    reasoning: str


class JudgeGrader:
    name = "judge"
    RUBRIC = ("Score 0.0–1.0 how well the ANSWER satisfies the CRITERIA. "
              "Be strict; reward only grounded, complete, correct answers. Return JSON.")

    def __init__(self, judge: LLMProvider, judge_model: str) -> None:
        self.judge, self.model = judge, judge_model

    async def grade(self, case: Case, output: str) -> float:
        v = await complete_structured(self.judge, CompletionRequest(model=self.model, messages=[
            Message(role="system", content=self.RUBRIC),
            Message(role="user", content=f"CRITERIA:\n{case.expected}\n\nANSWER:\n{output}")]), Verdict)
        return v.score
```

Judge hygiene: use a **different model** for judge vs candidate to avoid self-preference
bias; prefer **pairwise** (A-vs-B) when comparing two systems and **pointwise** (score one)
for absolute gates; randomize A/B order to cancel position bias; and **calibrate** the
judge against ~50 human labels before trusting it in a gate (target judge-human agreement
> 0.8).

## Metrics

```python
def accuracy(scores: list[float]) -> float:
    return sum(s >= 0.5 for s in scores) / len(scores)


def pass_at_k(successes_per_case: list[int], k: int) -> float:
    # pass@k = fraction of cases with >=1 success in k attempts.
    return sum(s >= 1 for s in successes_per_case) / len(successes_per_case)


def pass_caret_k(all_success_per_case: list[bool]) -> float:
    # pass^k = fraction of cases where ALL k attempts succeeded (stability).
    return sum(all_success_per_case) / len(all_success_per_case)


def percentile(values: list[float], p: float) -> float:
    s = sorted(values)
    idx = min(len(s) - 1, int(round((p / 100) * (len(s) - 1))))
    return s[idx]
```

Track at minimum: `accuracy`, `pass@1`/`pass@3`, `faithfulness` (judge/citation grader),
answer relevance, `cost_per_task`, `p50`/`p95` latency, and `tool_call_validity_rate`
(fraction of tool calls that pass schema validation). Recommended gates: capability
`pass@3 ≥ 0.90`; regression `pass^3 = 1.0` on release-critical paths.

## Eval runner

The full async runner. It supports a **dry-run** mode (`EVAL_DRY_RUN=1` or `--dry-run`)
that uses a stub provider and never hits live APIs — this is exactly what `verify.sh`
invokes as a smoke test.

```python
# evals/run.py
import argparse
import asyncio
import json
import os
import statistics
import sys
import time
from pathlib import Path

from provider_abstraction import (CompletionRequest, CompletionResponse, LLMProvider,
                                   Message, Usage, get_provider)


class StubProvider:
    """Deterministic offline provider for dry-runs — no network, no keys."""
    async def complete(self, req: CompletionRequest) -> CompletionResponse:
        return CompletionResponse(text='{"ok": true}', usage=Usage(input_tokens=10, output_tokens=5))
    async def stream(self, req):  # pragma: no cover - unused in dry-run
        yield
    async def embed(self, texts: list[str]) -> list[list[float]]:
        return [[0.0] * 1536 for _ in texts]


async def run(golden_path: str, dry_run: bool) -> int:
    cases = [json.loads(line) for line in Path(golden_path).read_text().splitlines() if line.strip()]
    provider: LLMProvider = StubProvider() if dry_run else get_provider(os.environ["LLM"])
    model = "stub" if dry_run else os.environ["LLM"].split(":", 1)[1]
    rows = []
    for c in cases:
        t0 = time.perf_counter()
        resp = await provider.complete(CompletionRequest(model=model,
            messages=[Message(role="user", content=c["input"])]))
        ok = SchemaGrader(_AnyJSON).grade(_Case(c), resp.text)
        rows.append({"score": ok, "cost": resp.usage.cost_usd, "ms": (time.perf_counter() - t0) * 1000})

    n = len(rows) or 1
    metrics = {
        "accuracy": sum(r["score"] for r in rows) / n,
        "cost_per_task": sum(r["cost"] for r in rows) / n,
        "p95_latency_ms": statistics.quantiles([r["ms"] for r in rows], n=20)[-1] if len(rows) > 1 else rows[0]["ms"],
    }
    thresholds = {"accuracy": 0.0 if dry_run else 0.9}   # dry-run only checks the pipeline runs
    Path("eval-report.json").write_text(json.dumps(metrics, indent=2))
    failed = {k: metrics[k] for k, lo in thresholds.items() if metrics[k] < lo}
    print(json.dumps({"metrics": metrics, "failed": failed, "dry_run": dry_run}, indent=2))
    return 1 if failed else 0


if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--golden", default="evals/golden.jsonl")
    ap.add_argument("--dry-run", action="store_true", default=os.environ.get("EVAL_DRY_RUN") == "1")
    args = ap.parse_args()
    sys.exit(asyncio.run(run(args.golden, args.dry_run)))
```

```python
# tiny shims referenced above so the file is self-contained
from pydantic import BaseModel


class _AnyJSON(BaseModel):
    model_config = {"extra": "allow"}


class _Case:
    def __init__(self, d: dict) -> None:
        self.input, self.expected, self.metadata = d["input"], d.get("expected", {}), d.get("metadata", {})
```

## Regression gates in CI

Run the suite on every PR; fail if any metric regresses past tolerance versus a stored
baseline. GitHub Actions:

```yaml
name: eval-gate
on: pull_request
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -e . pydantic openai anthropic
      - name: Run evals
        env:
          LLM: ${{ vars.LLM }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python evals/run.py --golden evals/golden.jsonl
      - name: Compare against baseline
        run: python evals/compare.py eval-report.json evals/baseline.json --tolerance 0.02
```

```python
# evals/compare.py — fail if any metric regresses more than tolerance vs baseline.
import json
import sys

current, baseline, _, tol = sys.argv[1], sys.argv[2], sys.argv[3], float(sys.argv[4])
cur = json.load(open(current))
base = json.load(open(baseline))
regressions = {k: (base[k], cur[k]) for k in ("accuracy",)
               if k in base and cur.get(k, 0) < base[k] - tol}
if regressions:
    print(f"REGRESSION: {regressions}")
    sys.exit(1)
print("no regressions")
```

Guard against flaky graders: run judge grades `n=3` and require majority; never put a
grader with judge-human agreement below your threshold into a release gate.

## Tracing (OTel GenAI)

Emit OpenTelemetry **GenAI semantic-convention** spans (`gen_ai.*`). Dashboards
(Langfuse / Phoenix / Braintrust) are swappable OTLP backends — you emit spans, you swap
the exporter.

```python
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))  # OTEL_EXPORTER_OTLP_ENDPOINT
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("agent")


async def traced_complete(p: LLMProvider, system: str, req: CompletionRequest) -> CompletionResponse:
    with tracer.start_as_current_span("chat") as span:
        span.set_attribute("gen_ai.system", system)               # "openai" | "anthropic" | ...
        span.set_attribute("gen_ai.request.model", req.model)
        span.set_attribute("gen_ai.request.temperature", req.temperature)
        resp = await p.complete(req)
        span.set_attribute("gen_ai.usage.input_tokens", resp.usage.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", resp.usage.output_tokens)
        span.set_attribute("gen_ai.usage.cost_usd", resp.usage.cost_usd)
        return resp
```

Wrap each tool call and each agent step in its own child span; the agent loop's `run_id`
becomes the trace id, so a whole run is one tree. **Redact PII before it lands in a span**
(spans persist to a backend) — never trace raw user content unfiltered.

## Cost & latency control

### Caching

Provider prompt caching (Anthropic `cache_control`, OpenAI/Gemini automatic cached input)
is abstracted behind the adapter (`cache_system` flag). On top of that, an **app-level
semantic cache** returns a prior answer when a new request is near-identical:

```python
import hashlib


class SemanticCache:
    def __init__(self, provider: LLMProvider, retriever, threshold: float = 0.95) -> None:
        self.provider, self.retriever, self.threshold = provider, retriever, threshold

    async def get_or_compute(self, prompt: str, compute) -> tuple[str, bool]:
        [vec] = await self.provider.embed([prompt])
        hits = await self.retriever.search(vec, k=1, flt={"kind": "cache"})
        if hits and hits[0].get("sim", 0) >= self.threshold:
            return hits[0]["content"], True                       # cache hit
        answer = await compute()
        await self.retriever.upsert(prompt, vec, answer, meta={"kind": "cache"})
        return answer, False
```

### Batching

For offline workloads (backfills, nightly evals, bulk classification), use provider batch
APIs (≈ −50% cost class) instead of synchronous calls. Submit a JSONL of requests, poll,
collect — latency is hours, not seconds, which is fine offline.

### Model routing / cascades

Classify the task, send it to the **cheapest model that passes the eval for that task
class**, and escalate only on a failed self-check — never default to the flagship.

```python
async def route_and_run(task: str, classify, providers: dict[str, tuple[LLMProvider, str]]) -> str:
    tier = await classify(task)                          # "cheap" | "default" | "smart"
    order = {"cheap": ["cheap", "default", "smart"],
             "default": ["default", "smart"],
             "smart": ["smart"]}[tier]
    for name in order:
        provider, model = providers[name]
        resp = await provider.complete(CompletionRequest(model=model,
            messages=[Message(role="user", content=task)]))
        if await _self_check(provider, model, task, resp.text):    # escalate only on failure
            return resp.text
    return resp.text                                     # smart tier is terminal


async def _self_check(provider, model, task: str, answer: str) -> bool:
    v = await complete_structured(provider, CompletionRequest(model=model, messages=[
        Message(role="user", content=f"Does this answer fully and correctly address the task? "
                                      f"TASK: {task}\nANSWER: {answer}")]), Verdict)
    return v.score >= 0.8
```

### Budgets

An immutable per-tenant cost tracker with a hard stop. Frozen dataclass = auditable, never
silently mutated.

```python
from dataclasses import dataclass, field


class BudgetExceededError(Exception):
    pass


@dataclass(frozen=True, slots=True)
class CostRecord:
    tenant: str
    model: str
    input_tokens: int
    output_tokens: int
    cost_usd: float


@dataclass(frozen=True, slots=True)
class CostTracker:
    budgets_usd: dict[str, float] = field(default_factory=dict)   # per-tenant cap
    records: tuple[CostRecord, ...] = ()

    def add(self, r: CostRecord) -> "CostTracker":
        spent = self.spent(r.tenant) + r.cost_usd
        cap = self.budgets_usd.get(r.tenant, float("inf"))
        if spent > cap:
            raise BudgetExceededError(f"{r.tenant}: ${spent:.4f} exceeds ${cap:.2f}")
        return CostTracker(budgets_usd=self.budgets_usd, records=(*self.records, r))

    def spent(self, tenant: str) -> float:
        return sum(rec.cost_usd for rec in self.records if rec.tenant == tenant)
```

Compute `cost_usd` from the dated pricing table — never hardcode a price in business logic.

| Model (provider:id) | $/MTok in | $/MTok out | Notes |
|---|---|---|---|
| anthropic:claude-haiku-4-5 | 1.00 | 5.00 | cheap/fast; caching −90% read, batch −50% |
| anthropic:claude-sonnet-4-6 | 3.00 | 15.00 | default; 1M-token context flat-rate |
| anthropic:claude-opus-4-8 | 5.00 | 25.00 | flagship Opus (4.8, May 2026); re-verify the id |
| openai:gpt-5.5 | 5.00 | 30.00 | flagship; >272K prompt billed 2× in / 1.5× out |
| openai:gpt-5.4 | 2.50 | 15.00 | cached input −90% |
| openai:gpt-5.2-codex | 1.75 | 14.00 | code-tuned |
| gemini:gemini-3.5-flash | 1.50 | 9.00 | cached 0.15; 1M context |
| gemini:gemini-3.1-pro | — | — | paid-only since 2026-04-01 |

All figures **(as of 2026-06; verify before quoting)**. Load them from config and refresh
the table before trusting any number.

## Anti-patterns

- **Overfitting prompts to the eval set** — you measure memorization, not capability.
  Hold out a fresh split.
- **Happy-path-only evals** — include adversarial, empty, and malformed inputs.
- **Chasing accuracy while cost/latency drift** — gate on `cost_per_task` and `p95`, not
  just accuracy.
- **Flaky graders in release gates** — calibrate and majority-vote, or keep them advisory.
- **Tracing PII without redaction** — spans persist; redact before emit.

## See also

- `provider-abstraction.md` — the `LLMProvider` the runner, judge, and router all use.
- `agent-loops-and-harness.md` — budgets and `AgentState` the `CostTracker` complements.
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall