Skip to content

ericrisco/rsc-harness / skills/building-agents

building-agents

Unverified

Use when building or restructuring an LLM agent — provider adapter, tool calling, structured output, RAG, agent loop, eval gate, cost routing, tracing, MCP server — model-agnostic across OpenAI/Anthropic/Gemini/OSS so a model swap is a config change. NOT vector-store SQL alone (that is `postgresdb`) or service deployment (that is `deployment`).

FileKindSize
SKILL.mdskill md19.4 KB
evals/README.mdmarkdown2.9 KB
evals/cases.yamlconfig6.3 KB
references/agent-loops-and-harness.mdmarkdown14.4 KB
references/evals-and-observability.mdmarkdown15.6 KB
references/mcp-servers.mdmarkdown11.6 KB
references/provider-abstraction.mdmarkdown20.8 KB
references/tools-and-rag.mdmarkdown15.6 KB
scripts/verify.shexecutablescript4.6 KB

evals/cases.yaml

View on GitHub
skill: building-agents

should_trigger:
  - prompt: "We're hardwired to the OpenAI SDK everywhere and the boss wants us to be able to switch to Claude or Gemini without a rewrite. How should I restructure this?"
    why: "Provider-agnostic restructuring / 'swap models is a config change' is the core thesis of the skill (the capability interface + adapter)."
  - prompt: "Add function calling to our support assistant so it can look up orders and issue refunds, but I don't want it to blow up if the model sends garbage arguments."
    why: "Adding tool calling with schema validation + typed contracts + idempotency on side-effecting tools is squarely in scope."
  - prompt: "Build me a chatbot that answers questions strictly from our internal handbook PDFs and says 'I don't know' when the answer isn't there."
    why: "RAG over a private corpus with cite-or-refuse grounding — a named trigger — even though the user never says 'RAG', 'embeddings', or 'retrieval'."
  - prompt: "Our agent sometimes spins forever and runs up a huge API bill. Help me make the loop safe and put a cost ceiling on each task."
    why: "Bounded agent loop with step cap, timeouts, and token/cost budgets — the loop + cost-control trigger — phrased as a symptom, not a tool name."
  - prompt: "Before we ship this LLM feature, I want a quality gate in CI that fails the build if accuracy or faithfulness drops on our test set."
    why: "Golden set + graders + regression gate as a CI exit code is the 'eval before ship' decision rule and an explicit trigger."
  - prompt: "I want to expose our internal pricing tools to several different AI clients so multiple teams can reuse them. What's the right way to wrap them?"
    why: "Cross-client tool reuse out-of-process = build an MCP server, an explicit trigger; the user describes the reuse need rather than naming MCP."
  - prompt: "Stand up tracing for our LLM calls so I can see token usage and cost per request in Langfuse without locking us to one backend."
    why: "Vendor-neutral OTel GenAI observability with a swappable exporter is an explicit trigger."

should_not_trigger:
  - prompt: "Reword this prompt so the model gives more concise answers — no code changes, just better wording."
    route_to: "none"
    why: "Pure prompt-wording improvement with no architecture is prompt engineering; the skill's 'Do NOT use' list excludes it explicitly."
  - prompt: "Write a one-off script that calls the OpenAI API once to summarize a string I paste in. No tools, no tests, throwaway."
    route_to: "none"
    why: "One-shot throwaway prompt with no tools/eval/production path — the skill says just call the SDK directly."
  - prompt: "Tune Anthropic prompt caching and extended thinking in our anthropic-only client.py to cut latency on a file that only imports the anthropic SDK."
    route_to: "external:claude-api"
    why: "Single-vendor Anthropic-SDK internals tuning in a file that only imports anthropic defers to the dedicated Anthropic skill; this skill stays multi-provider."
  - prompt: "Write the pgvector schema and an HNSW-indexed nearest-neighbour SQL query, plus tuning advice — just the database side, no LLM calls."
    route_to: "postgresdb"
    why: "Pure SQL / pgvector indexing with no embeddings pipeline, retrieval grounding, or agent is Postgres territory."
  - prompt: "Set up the CI pipeline and Docker deploy for our FastAPI service that happens to host an LLM endpoint — focus on the deployment, not the model code."
    route_to: "deployment"
    why: "Shipping/CI/containerization of the service is deployment; the skill itself points to deployment for that concern."
  - prompt: "Audit our auth handler for injection and secret-leak vulnerabilities — it's a normal REST endpoint, not an agent."
    route_to: "security-audit"
    why: "A security-only review of a non-agent endpoint is security-audit, not LLM agent architecture."
  - prompt: "Bootstrap the 01-TOOLS and 02-DOCS workspace layout for this repo and generate the root CLAUDE.md."
    route_to: "harness"
    why: "Workspace scaffolding (the 01-TOOLS/02-DOCS layout) is explicitly delegated to harness."

capability:
  - scenario: "Our codebase calls `OpenAI().chat.completions.create(model='gpt-5.5', ...)` directly in three handlers. We need to support switching to Anthropic Claude or Gemini via config, with strict JSON output for a sentiment classifier and one side-effecting `create_invoice` tool. Lay out the architecture and the key code."
    must_include:
      - "Defines a capability/provider interface (e.g. an LLMProvider Protocol with a normalized request/response) that business logic programs against instead of a vendor SDK"
      - "Model id is resolved from config/registry (e.g. 'provider:model'), never hardcoded in handler logic"
      - "Per-provider adapters that hide vendor quirks (OpenAI tools/function vs Anthropic system-as-top-level-param + input_schema / tool-forcing for structured output)"
      - "Strict structured output via JSON Schema with validation that fails loudly (e.g. Pydantic model_validate), not try/except json.loads that swallows errors"
      - "The create_invoice tool has a typed args model with extra='forbid', validates arguments BEFORE side effects, and carries an idempotency key"
      - "Returns a structured ToolResult (status/summary/next_actions), not the raw API blob"
      - "Mentions selecting the cheapest model that passes an eval rather than defaulting to the flagship"
  - scenario: "We have a RAG support bot and want to make it production-ready before launch. Show how to bound the agent loop, eval it, and observe it."
    must_include:
      - "Bounded agent loop: explicit step cap, per-tool timeout, and a token/cost budget (no while True)"
      - "Retrieval grounded with cite-or-refuse: cite chunk ids and refuse when no chunk passes a similarity threshold rather than hallucinating"
      - "A golden set plus graders (exact/schema/LLM-as-judge) with a CI threshold that exits non-zero to block the merge"
      - "Reports metrics including accuracy/faithfulness, p95 latency, and cost per task"
      - "OTel GenAI tracing with gen_ai.* attributes (model, input/output tokens, cost) and a swappable OTLP exporter/backend"
      - "Retries only transient errors (429/5xx/timeout) with backoff+jitter, not 4xx like 400/401"
      - "Routing/cascade to the cheapest model whose eval passes, escalating only on a failed self-check"
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall