Skip to content

ericrisco/rsc-harness / skills/rag

Use when building grounded Q&A over your own corpus — chunk, retrieve hybrid, rerank, ground, cite chunk ids, refuse when the sources fall short — or when the right document is retrieved but the answer is still wrong, invented, or unmeasured. NOT operating the store itself — collection schema, HNSW ef_search, quantization (that is `vector-db`).

rag — own the retrieve → rerank → ground → cite → refuse pipeline

You own the pipeline that turns a corpus plus a question into a grounded, cited answer: chunk, optionally contextualize, index, retrieve hybrid, rerank, assemble a grounded prompt, cite the sources, and refuse when the context does not contain the answer.

You are judged by retrieval quality and answer faithfulness, not by raw vector math. If you find yourself tuning HNSW parameters, you wandered into the store underneath you (../vector-db/SKILL.md). If you are comparing embedding models or chunk sizes, that is the science beside you (embeddings-search).

The pipeline, and where each stage hands off

Each stage is a real branch — most failures live in one specific stage, and several stages delegate to a sibling skill rather than living here.

StageWhat you doHands off to
IngestGet clean text out of PDFs/DOCX/HTML/OCRyou assume text exists → ../document-processing/SKILL.md
ChunkHeading/semantic-aware splits with overlap, stable idsmodel, dims + chunk-size science → embeddings-search
ContextualizePrepend an LLM-written context blurb per chunk (optional)stays here
IndexEmbed + write dense vectors and a BM25/keyword indexyou upsert, it owns the knobs → ../vector-db/SKILL.md
RetrieveHybrid dense + BM25, fuse with RRF, top ~150hybrid query mechanics → ../vector-db/SKILL.md
RerankCross-encoder over the 150, keep top ~20stays here
Ground + citeSystem prompt: answer only from context, cite chunk idsstays here
RefuseOutput "I don't have enough information" on weak contextstays here
EvaluateFaithfulness, answer relevancy, context precision/recallgeneral harness → agent-eval

Three neighbors are not stages at all. Surfacing this answer inside a chat product (sessions, channels, UI) is ../chatbot/SKILL.md — it calls you, not the reverse. A multi-step tool loop with state, where retrieval is one tool among many, is ../building-agents/SKILL.md. Pulling schema-constrained fields out of text instead of a grounded prose answer is structured-extraction. rag is the retrieval brain those products call.

Retrieval is the bottleneck — measure recall before touching the prompt

Naive RAG pipelines fail at the retrieval step in up to ~40% of cases even when the correct document is in the corpus (StackAI / Lushbinary, 2026-06-02). Why this matters: if the right passage never reaches the model, no prompt wording can save the answer. So your first move on a broken pipeline is never the prompt — it is measuring whether retrieval delivered the goods.

Bad   → "answers are wrong, let me lower temperature and reword the system prompt."
Good  → measure context recall on a golden set; if the right chunk isn't in the top-K,
         fix chunking + hybrid + rerank first. Only then touch grounding.

Order of attack when answers are wrong: context recall → context precision → grounding prompt → generation params. The last item almost never moves the needle.

Chunking — the highest-leverage single fix

Chunk on structure, not on a blind character count. Why: too-small loses the context a passage needs to be interpretable; too-large dilutes the embedding so the relevant sentence gets averaged away (StackAI; EdenAI 2025, accessed 2026-06-02).

  • Split on headings/sections first, then sub-split long sections to a target window.
  • Keep overlap (~10–20% of the window) so a fact spanning a boundary survives in one chunk.
  • Attach a stable chunk_id and source at creation — you will need them end to end for citations (see below). Never embed text and discard where it came from.
# Heading-aware split sketch; real size tuning belongs in embeddings-search.
def chunk_markdown(doc_id, text, target=800, overlap=120):
    sections, buf, head = [], [], None
    for line in text.splitlines():
        if line.startswith("#"):
            if buf: sections.append((head, "\n".join(buf))); buf = []
            head = line.lstrip("# ").strip()
        else:
            buf.append(line)
    if buf: sections.append((head, "\n".join(buf)))
    out, i = [], 0
    for head, body in sections:
        for start in range(0, max(1, len(body)), target - overlap):
            piece = body[start:start + target]
            out.append({"chunk_id": f"{doc_id}#{i}", "source": doc_id,
                        "heading": head, "text": piece})
            i += 1
    return out

The "what window/overlap maximizes recall for this corpus" study is embeddings-search; here you just need structurally sane chunks that keep their ids.

Contextual retrieval — prepend context before you embed

Anthropic's Contextual Retrieval (Sept 2024) prepends a short LLM-generated blurb to each chunk before embedding and before BM25 indexing, so an isolated chunk knows what document and section it belongs to. Why it matters: it cuts failed retrievals by ~35% (contextual embeddings alone), ~49% (contextual embeddings + contextual BM25), and ~67% once reranking is added (anthropic.com/news/contextual-retrieval, 2024-09; accessed 2026-06-02).

<document>{{WHOLE_DOC}}</document>
Here is the chunk we want to situate within the whole document:
<chunk>{{CHUNK}}</chunk>
Give a short, standalone context (1–2 sentences) that situates this chunk within the
document for search retrieval. Answer only with the context, nothing else.

Embed context + "\n" + chunk_text (not the bare chunk). The full implementation — caching the document prompt, batching, the BM25 side, and a runnable retrieve → rerank → answer skeleton — lives in references/pipeline.md.

Hybrid retrieval + RRF + rerank

Dense vectors miss exact terms (codes, names, error strings); BM25 keyword search catches them but misses paraphrase. Combine both, fuse with Reciprocal Rank Fusion (RRF), then rerank with a cross-encoder. The default-best quality/cost funnel (Microsoft Cloud Blog 2025-02-04; StackAI, accessed 2026-06-02):

retrieve ~150 candidates (dense + BM25, fused with RRF)
   → rerank all 150 with a cross-encoder
   → keep top ~20 for the prompt

The actual hybrid query (named vectors, sparse-dense, server-side fusion) is vector-db. Here you own the funnel and the reranker choice:

  • Cohere Rerank 3.5 — managed cross-encoder, context length 4096, SOTA on BEIR and multilingual, available via Cohere API, Bedrock, Pinecone, Azure (docs.cohere.com/changelog/ rerank-v3.5, accessed 2026-06-02). Use when you want quality without hosting a model.
  • A local cross-encoder (e.g. a bge-reranker) — use when data cannot leave your network or you need zero per-call cost; you pay in GPU/latency instead.
# Rerank the fused candidates down to the prompt set; keep ids intact.
import cohere
co = cohere.ClientV2()
ranked = co.rerank(model="rerank-v3.5", query=q,
                   documents=[c["text"] for c in candidates], top_n=20)
top = [candidates[r.index] for r in ranked.results]  # each still carries chunk_id + source

Ground the prompt — answer only from context, cite, refuse

Properly grounded RAG reduces hallucination rates by up to ~71%; poorly grounded pipelines still hallucinate in up to ~40% of responses even with the right doc retrieved (Confident AI / Maxim 2025, accessed 2026-06-02). The prompt must do three things: bind the answer to the context, force inline citations, and provide an explicit refusal path.

You answer ONLY using the information inside <context>. Do not use prior knowledge.
Cite every claim with the chunk id it came from, like [chunk_id]. Multiple ids are fine.
If the context does not contain enough information to answer, reply exactly:
"I don't have enough information in the provided sources to answer that."
(Spanish corpora: "No tengo suficiente información en las fuentes para responder.")

<context>
[doc12#3] {chunk text...}
[doc12#4] {chunk text...}
</context>

Question: {{question}}

A grounding prompt without a refusal path is a bug — it converts "missing context" into a confident fabrication. The refusal clause is what turns retrieval failures into honest non-answers.

Citations — carry ids the whole way

The answer can only link back if a stable id survives every stage: chunk → retrieve → rerank → prompt → answer. Why: if you embed text and drop the id at index time, there is nothing for a citation to point at, and you cannot debug which passage produced a wrong claim.

  • Put chunk_id and source on the chunk at creation; keep them on the object through rerank.
  • Render them into the <context> block ([chunk_id] text) so the model can quote them.
  • Map cited ids back to source URLs/pages when you render the answer to the user.

Evaluating it — metrics, not vibes

Faithfulness is not correctness: an answer can be faithful to a wrong chunk. Measure four RAGAS metrics on a small golden Q/A set (RAGAS docs; Cohorte 2025; Confident AI, accessed 2026-06-02):

Failure symptomMetric that catches itFirst fix
Answer states things not in the sourcesfaithfulness (claims supported by context)grounding prompt + refusal
Answer is on-topic but doesn't address the questionanswer relevancyprompt / query rewriting
Relevant chunks exist but rank below junkcontext precisionreranker, RRF weights
The needed chunk never gets retrievedcontext recallchunking, contextual retrieval, hybrid

Build a 30–50 question golden set with known-good answers, score with RAGAS, and gate CI below a threshold (e.g. faithfulness ≥ 0.90, context recall ≥ 0.85). Full formulas, thresholds, and the CI snippet are in references/evaluation.md. The general-purpose eval harness is agent-eval; the RAG-specific metrics live here.

Anti-patterns

Anti-patternWhy it breaksDo instead
Fixed char-count chunking, blind to structureSplits mid-sentence; dilutes embeddingsHeading/semantic chunks with overlap
Rerank disabled — stuff top-50 raw into promptNoise drowns the right passage; cost balloonsRetrieve ~150 → rerank → keep ~20
No refusal path in the grounding promptMissing context becomes confident fabricationExplicit "I don't have enough information"
Embed text, drop the chunk idNothing to cite or debugCarry chunk_id+source end to end
"It looks good" eval on vibesRegressions ship silentlyGolden set + RAGAS + CI threshold gate
Embedding the query differently from the corpusQuery and chunks land in different spacesSame model + same preprocessing both sides
Dense-only, ignoring BM25/keywordMisses exact codes/names/error stringsHybrid dense + BM25 fused with RRF
Tuning temperature to fix wrong answersGeneration is rarely the bottleneckMeasure context recall first
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 196kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 193kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 193kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 193kUpdated Claude CodeCodexVerifiedInstall