ericrisco/rsc-harness / skills/rag
rag
VerifiedUse when building grounded Q&A over your own corpus — chunk, retrieve hybrid, rerank, ground, cite chunk ids, refuse when the sources fall short — or when the right document is retrieved but the answer is still wrong, invented, or unmeasured. NOT operating the store itself — collection schema, HNSW ef_search, quantization (that is `vector-db`).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 11.4 KB |
| evals/README.md | markdown | 1.0 KB |
| evals/cases.yaml | config | 3.6 KB |
| references/evaluation.md | markdown | 4.3 KB |
| references/pipeline.md | markdown | 6.1 KB |
| scripts/verify.shexecutable | script | 6.6 KB |
evals/cases.yaml
View on GitHubskill: rag
should_trigger:
- prompt: "Build Q&A over our internal handbook so the bot only answers from it and adds citations."
why: Core RAG ask — grounded answers from a private corpus with citations is exactly the pipeline rag owns.
- prompt: "My RAG returns the right document but the answer is still wrong."
why: Non-obvious — the store retrieved correctly, so this is a passage/rerank/grounding problem inside the pipeline, not a vector-db issue.
- prompt: "The assistant keeps inventing facts that aren't in our docs — how do I ground it so it stops?"
why: Grounding + refusal contract; forcing answer-only-from-context and an explicit refusal is rag's job.
- prompt: "How do I measure if my retrieval is any good — context recall vs context precision?"
why: RAG-specific evaluation metrics; rag carries faithfulness/recall/precision even though the general harness is agent-eval.
- prompt: "Add contextual retrieval — prepend an LLM context blurb to each chunk before embedding to cut failed retrievals."
why: Anthropic Contextual Retrieval technique lives in the rag pipeline (contextualize stage), not in embeddings-search.
- prompt: "Wire a hybrid search plus rerank flow: dense + BM25 fused, then a cross-encoder before generation."
why: The retrieve → rerank → generate funnel is the heart of the rag pipeline.
- prompt: "El bot s'inventa coses que no surten als nostres documents i vull respostes amb cites."
why: Catalan — grounding + citations over a private corpus is the rag pipeline.
should_not_trigger:
- prompt: "Set up a Qdrant collection with binary quantization and tune ef_search for recall."
route_to: vector-db
why: Store operations — collection schema, quantization, HNSW knobs — are the layer under rag, not the pipeline.
- prompt: "Which embedding model and chunk size give the best recall for a code corpus?"
route_to: embeddings-search
why: Embedding-model choice and chunk-size science are the layer beside rag; rag uses a chunker but does not own the experiments.
- prompt: "Extract the tables from these scanned invoice PDFs into clean text."
route_to: document-processing
why: File-to-text / OCR is pre-RAG ingestion; rag assumes text already exists.
- prompt: "Build the chat UI with sessions and channel routing for our support bot."
route_to: chatbot
why: Product surface (sessions, UI, channels) — chatbot calls the rag brain, it is not the retrieval pipeline itself.
- prompt: "Pull the buyer name, total, and due date out of this contract into a fixed JSON schema."
route_to: structured-extraction
why: Schema-constrained field extraction, not a grounded prose Q&A answer.
capability:
- scenario: "Design a grounded RAG pipeline over a 500-PDF policy corpus that currently hallucinates and sometimes answers despite the right document being present."
must_include:
- Names the full pipeline stages retrieve → rerank → ground → cite → refuse.
- Specifies hybrid retrieval (dense + BM25) fused with Reciprocal Rank Fusion (RRF).
- Uses a reranker (Cohere Rerank 3.5 or a local cross-encoder) with a top-N → top-K funnel (~150 → ~20).
- Grounding prompt forces answer-only-from-context, inline citations of chunk ids, and an explicit refusal on insufficient context.
- Defines an eval plan with faithfulness plus context recall/precision and a CI threshold gate.
- Routes file-to-text ingestion to document-processing and store operations to vector-db.
- Mentions contextual retrieval (prepend context before embedding) as a recall lever for the policy corpus.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |