alivirgo/Major-AI-Skills / plugins/mas-ai-workflows/skills/rag-retrieval-audit
rag-retrieval-audit
VerifiedDiagnose missing evidence in a RAG pipeline with labeled queries, chunk/trace inspection, Recall@K/MRR/nDCG, and stage-isolated failure taxonomy. Use when retrieval looks fine but answers fail, or when index/chunk/filter/rerank changes need before/after proof.
RAG Retrieval Audit AI Skill Guide (Claude)
Overview & Engine Architecture
A RAG system fails in stages. Most “the model is wrong” tickets are retrieval, indexing, chunking, filter, or context-assembly failures that a stronger generator only hides with fluent guesses. This skill audits the retriever path only: did the answer-bearing evidence enter the final model context at the application’s real cutoff K?
Claude operates as a Principal RAG Reliability Engineer, specializing in stage-isolated failure taxonomy, ID-based Recall@K / MRR / nDCG, chunk-boundary forensics, hybrid-search arm health, and ACL-safe relevance denominators.
┌─────────────────────────────────────────────────────────────┐
│ RAG Retrieval Audit Stack │
│ │
│ Corpus governance │
│ ├── Source docs → parse → chunk → embed → index revision │
│ └── Metadata: ACL, tenant, version/hash, embedding_model │
│ │
│ Query path │
│ ├── Rewrite / expand → dense ± sparse (BM25) → fuse (RRF) │
│ ├── Optional rerank → filters/thresholds → context pack │
│ └── Generator (out of scope for retrieval metrics) │
│ │
│ Audit harness │
│ ├── Labeled queries + eligible relevant chunk/doc IDs │
│ ├── Per-query traces (retrieved IDs, ranks, scores, filters│
│ └── Metrics @ app K: hit-rate, Recall@K, Precision@K, MRR, │
│ nDCG; separate faithfulness later │
└─────────────────────────────────────────────────────────────┘
When to use / when not to
Use when
- Answers are confident but wrong, and you need to know if evidence was missing vs ignored.
- You are changing chunk size, embeddings, hybrid weights, filters, or rerankers.
- Production has zero-result or “wrong doc family” clusters.
- Compliance requires proving which sources were eligible for a user.
Do not use when
- The task is pure prompt/style polish with a fixed golden context (use generation evals).
- You lack any labeled relevant IDs and refuse LLM-as-judge proxies (build labels first via
@ai-evaluation-dataset). - Someone asks to bypass ACLs so recall looks better — refuse; shrink the relevance denominator instead.
Operational Capabilities & Agent Directives
- Freeze one baseline: index revision, embedding model ID, chunker version,
K, filters, fusion weights, reranker, and prompt context budget before any experiment. - Label at chunk (or passage) IDs, not vague “the handbook.” Prefer
source_id + content_hash + chunk_id. - Evaluate at the application’s real K — the count of chunks that actually reach the generator after truncation, not an aspirational top-50.
- Separate stages: ingestion miss ≠ parse fail ≠ chunk split ≠ ranking miss ≠ filter drop ≠ context truncate ≠ faithfulness drift.
- Never compare raw cosine/dot scores across embedding models as if calibrated.
- Change one variable at a time; keep a frozen baseline run for every PR.
- ACL hygiene: documents the user cannot access are not “misses”; exclude them from the relevance denominator.
Failure taxonomy (diagnose in this order)
| Stage | Symptom | Typical root cause |
|---|---|---|
| Ingestion | Relevant source_id absent from index | Sync lag, failed job, wrong collection, deleted tombstone |
| Parse | Doc present; answer text garbled/missing | PDF/table/OCR loss; HTML boilerplate; empty extracted body |
| Chunk | Answer split across boundaries; half-table rows | Fixed-token splits; tiny overlap; footer pollution in embeddings |
| Embed/index | Near-zero or nonsense scores; empty after migration | Wrong embedding_model on query vs corpus; metric mismatch |
| Retrieve | Relevant ID never in top-K (any arm) | Weak dense match on IDs/codes; BM25 arm dead; too-small fetch_k |
| Fuse/threshold | Hybrid “works” but lexical-only hits vanish | Similarity floor applied after RRF to fused list |
| Filter/ACL | Empty or wrong-tenant hits | Post-filter after ANN (fetch_k then filter) vs pre-filter; over-tight metadata |
| Rerank/pack | ID retrieved then dropped from final prompt | Reranker demotion; duplicate collapse; token budget truncation |
| Generate | Correct chunks in prompt; answer still wrong | Faithfulness — not a retrieval fix (track separately) |
Practitioner pattern (r/Rag): treat “RAG is inaccurate” as retrieval until proven otherwise; only then score faithfulness against the assembled context.
Metrics — what each number means
Use ID-based metrics when gold chunk/doc IDs exist (preferred). Use Ragas-style context recall / context precision (claim attribution vs reference) when you only have reference answers.
| Metric | Question | Order-sensitive | Reach for it when |
|---|---|---|---|
| Hit-rate@K | ≥1 relevant ID in top K? | no | Cheap CI smoke |
| Recall@K | Fraction of gold IDs in top K | no | Missing evidence is costly |
| Precision@K | Fraction of top K that are relevant | no | Context budget is tight |
| MRR | Reciprocal rank of first relevant | yes | Generator mostly uses top 1–2 |
| nDCG@K | Graded ranking quality | yes | Reranker / graded labels |
| Context recall (Ragas) | Reference claims supportable from retrieved text | n/a | No gold chunk IDs yet |
| Context precision (Ragas) | Relevant chunks ranked early vs noise | yes | Noise displaces signal |
Diagnostic pairs
- Recall↓, MRR irrelevant → chunking / embedding / not in index.
- Recall steady, MRR↓ → ranking / missing reranker / fusion weights.
- Both OK, answers wrong → faithfulness or packing/truncation (log final prompt context).
- Recall↑ but MRR↓ after a change → more gold in the set but buried — generation often worsens.
Do not gate releases only on end-to-end answer accuracy; it masks 20–40% retrieval regressions when the model bluffs.
Audit procedure
1. Inputs to collect (block if missing)
- Bounded query set (≥30 real user questions; synthetic-only sets are optional extras).
- Per query: eligible relevant chunk IDs under the same ACL as production.
- Index revision / collection name, embedding model, chunker config,
K, filters, hybrid weights, reranker ID. - Authorization to mutate only a test index unless prod change is approved.
2. Per-query trace schema (mandatory fields)
Log for every audited query:
{
"query_id": "q-1042",
"query_text": "...",
"user_acl": ["tenant:acme", "role:support"],
"index_revision": "idx-2026-10-01T12:00Z",
"embedding_model": "bge-m3",
"k_final": 8,
"query_rewrite": null,
"filters_applied": {"tenant": "acme"},
"arms": {
"dense": [{"id": "c1", "rank": 1, "score": 0.71}],
"bm25": [{"id": "c9", "rank": 1, "score": 12.4}]
},
"fused": [{"id": "c1", "rank": 1, "rrf": 0.032, "sources": ["dense"]}],
"after_rerank": [{"id": "c9", "rank": 1, "score": 0.88}],
"final_context_ids": ["c9", "c1", "c3"],
"final_context_chars": 6120,
"gold_ids": ["c9"],
"notes": ""
}
Critical: log final_context_ids (what the model saw), not only first-stage retrieval. Truncation and rerank drops are common silent failures.
3. Score and classify
For each query, mark primary failure stage from the taxonomy. Aggregate:
- Mean Hit-rate@K, Recall@K, Precision@K, MRR, nDCG@K at production K.
- Slice by: zero-result, filter-excluded gold, BM25-only share of top-K, document family, language.
4. Single-variable experiments
Preserve baseline. Typical order of levers (stop when Recall@K and MRR meet gate):
- Confirm gold sources are indexed and parsed (spot-open chunk text).
- Fix chunk boundaries around tables/codes; add overlap; strip repeated footers.
- Raise candidate pool (
fetch_k/ multi-arm top-N) before filters when the store post-filters. - Prefer pre-filter (or filtered ANN) over fetch-then-filter when tenants are sparse in the ANN shortlist (LangChain FAISS filter ordering).
- Hybrid: ensure BM25-only hits are not killed by a post-fusion vector similarity floor (r/Rag hybrid bug reports).
- Add cross-encoder rerank after fusion when RRF alone is flat on homogeneous jargon.
- Only then retune embeddings or rembed the corpus — with
embedding_modelstamped on every chunk.
5. Deliverable
Ship:
- Per-query failure table (query_id, stage, gold_ids, final_context_ids, hit, ranks).
- Aggregate metrics before/after with identical
Kand ACL rules. - Reproducible config dump (models, chunker, index revision, fusion, filters).
- Explicit “not retrieval” bucket for faithfulness follow-ups.
- ACL note: inaccessible gold excluded from denominators — no bypass recommendations.
Production example: ID-based retrieval eval harness
"""Minimal RAG retrieval audit: Recall@K, Precision@K, MRR, hit-rate."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Callable, Sequence
@dataclass(frozen=True)
class Case:
query_id: str
query: str
gold_ids: frozenset[str] # eligible under the same ACL
def recall_at_k(retrieved: Sequence[str], gold: frozenset[str], k: int) -> float:
if not gold:
return float("nan") # exclude ACL-empty cases from means
top = list(retrieved)[:k]
return len(gold.intersection(top)) / len(gold)
def precision_at_k(retrieved: Sequence[str], gold: frozenset[str], k: int) -> float:
top = list(retrieved)[:k]
if not top:
return 0.0
return len(gold.intersection(top)) / len(top)
def mrr(retrieved: Sequence[str], gold: frozenset[str], k: int) -> float:
for i, doc_id in enumerate(list(retrieved)[:k], start=1):
if doc_id in gold:
return 1.0 / i
return 0.0
def hit_rate(retrieved: Sequence[str], gold: frozenset[str], k: int) -> float:
return 1.0 if gold.intersection(list(retrieved)[:k]) else 0.0
def audit(
cases: Sequence[Case],
retrieve_final_context_ids: Callable[[str], Sequence[str]],
k: int,
) -> list[dict]:
rows = []
for case in cases:
ids = list(retrieve_final_context_ids(case.query))
rows.append(
{
"query_id": case.query_id,
"hit": hit_rate(ids, case.gold_ids, k),
"recall": recall_at_k(ids, case.gold_ids, k),
"precision": precision_at_k(ids, case.gold_ids, k),
"mrr": mrr(ids, case.gold_ids, k),
"retrieved": ids[:k],
"gold": sorted(case.gold_ids),
}
)
return rows
# Wire retrieve_final_context_ids to YOUR packer (post-filter, post-rerank, post-truncate).
# Prefer LlamaIndex RetrieverEvaluator or Ragas IDBasedContextRecall when already in-stack:
# https://developers.llamaindex.ai/python/examples/evaluation/retrieval/retriever_eval/
# https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/
Canary without a full golden set
Log fusion_source per top-K hit (dense | bm25 | both). If BM25-only share flatlines after a deploy, hybrid may be silently vector-only.
Technical troubleshooting matrix
| Failure signature | Root cause | Diagnostic & fix |
|---|---|---|
| Empty results for one tenant | ANN then metadata filter; fetch_k too small | Pre-filter or raise fetch_k; verify filter field cardinality |
| Exact error codes / SKUs missed | Dense-only retrieval | Add BM25/sparse arm; query rewrite that keeps literals |
| Hybrid no better than dense | Post-RRF similarity threshold drops BM25-only | Threshold on dense arm only; keep lexical survivors |
| High similarity, wrong answer | Chunk lacks full fact (boundary/table split) | Structural chunking; inspect gold span vs chunk text |
| Correct doc, wrong chunk | Boilerplate-dominated embeddings | Strip headers/footers; smaller semantic units |
| Scores ~0 but “good” answers | Distance/metric or embed-model mismatch | Pin same embed_model on index build and query |
| Recall OK, users still complain | Faithfulness / truncation of packed context | Diff retrieved vs final_context_ids; generation eval |
| Metric improved, UX worse | Recall↑ MRR↓ | Add reranker; report both metrics in CI |
Best practices
- Build the eval set from real support/search queries, not only synthetic FAQ paraphrases.
- Version corpus + chunker + embedding model together; stamp
embedding_modelon chunks for gradual rembeds. - Deduplicate near-identical chunks before fusion to stop footer clones from crowding top-K.
- Keep retrieval CI separate from answer faithfulness CI (
@ai-evaluation-dataset,@ai-citation-verification). - For evidence-dense docs, prefer structural/hierarchical chunk checks; for sparse factoid QA, simple token/sentence baselines are often enough — measure, don’t assume.
- Gate merges on Hit-rate@K + Recall@K + MRR with thresholds chosen from your baseline, not internet folklore.
Limitations
- This skill does not certify production quality for every vector DB or framework version.
- LLM-judge context metrics are proxies; disagreement with ID-based labels should be investigated.
- Multimodal/table-heavy corpora need parser-specific audits beyond text metrics.
- Stop and ask if ACL rules, index write permission, or success thresholds are missing.
Related skills
@ai-evaluation-dataset— build/version labeled query sets and gold IDs@llm-json-contract-check— structured judge / scorer outputs@prompt-regression-gate— prompt/packer changes with frozen retrieval@ai-citation-verification— citation support against retrieved passages@ai-pii-redaction-review— redact traces before sharing eval dumps@llm-cost-latency-benchmark— cost/latency of rerank + larger K@qdrant/@chromadb/@elasticsearch— store-specific filter and hybrid knobs
Agent Operational Directive
MANDATORY: Audit retrieval with stage isolation and metrics at production
K. Log final packed context IDs. Change one variable at a time against a frozen baseline. Never recommend ACL bypasses to inflate recall. Report retrieval metrics separately from answer quality, latency, and cost.
Source anchors (research)
- Ragas Context Recall / Context Precision
- LlamaIndex RetrieverEvaluator
- LangChain FAISS filter-after-ANN empty results
- LlamaIndex zero/near-zero scores / embed mismatch
- r/Rag: retrieval-vs-generation split; chunk boundary failures; post-RRF threshold killing BM25
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |