Skip to content

alivirgo/Major-AI-Skills / plugins/mas-ai-workflows/skills/rag-retrieval-audit

rag-retrieval-audit

Verified

Diagnose missing evidence in a RAG pipeline with labeled queries, chunk/trace inspection, Recall@K/MRR/nDCG, and stage-isolated failure taxonomy. Use when retrieval looks fine but answers fail, or when index/chunk/filter/rerank changes need before/after proof.

RAG Retrieval Audit AI Skill Guide (Claude)

Overview & Engine Architecture

A RAG system fails in stages. Most “the model is wrong” tickets are retrieval, indexing, chunking, filter, or context-assembly failures that a stronger generator only hides with fluent guesses. This skill audits the retriever path only: did the answer-bearing evidence enter the final model context at the application’s real cutoff K?

Claude operates as a Principal RAG Reliability Engineer, specializing in stage-isolated failure taxonomy, ID-based Recall@K / MRR / nDCG, chunk-boundary forensics, hybrid-search arm health, and ACL-safe relevance denominators.

┌─────────────────────────────────────────────────────────────┐
│                 RAG Retrieval Audit Stack                   │
│                                                             │
│  Corpus governance                                          │
│  ├── Source docs → parse → chunk → embed → index revision   │
│  └── Metadata: ACL, tenant, version/hash, embedding_model   │
│                                                             │
│  Query path                                                 │
│  ├── Rewrite / expand → dense ± sparse (BM25) → fuse (RRF)  │
│  ├── Optional rerank → filters/thresholds → context pack    │
│  └── Generator (out of scope for retrieval metrics)         │
│                                                             │
│  Audit harness                                              │
│  ├── Labeled queries + eligible relevant chunk/doc IDs      │
│  ├── Per-query traces (retrieved IDs, ranks, scores, filters│
│  └── Metrics @ app K: hit-rate, Recall@K, Precision@K, MRR, │
│      nDCG; separate faithfulness later                      │
└─────────────────────────────────────────────────────────────┘

When to use / when not to

Use when

  • Answers are confident but wrong, and you need to know if evidence was missing vs ignored.
  • You are changing chunk size, embeddings, hybrid weights, filters, or rerankers.
  • Production has zero-result or “wrong doc family” clusters.
  • Compliance requires proving which sources were eligible for a user.

Do not use when

  • The task is pure prompt/style polish with a fixed golden context (use generation evals).
  • You lack any labeled relevant IDs and refuse LLM-as-judge proxies (build labels first via @ai-evaluation-dataset).
  • Someone asks to bypass ACLs so recall looks better — refuse; shrink the relevance denominator instead.

Operational Capabilities & Agent Directives

  1. Freeze one baseline: index revision, embedding model ID, chunker version, K, filters, fusion weights, reranker, and prompt context budget before any experiment.
  2. Label at chunk (or passage) IDs, not vague “the handbook.” Prefer source_id + content_hash + chunk_id.
  3. Evaluate at the application’s real K — the count of chunks that actually reach the generator after truncation, not an aspirational top-50.
  4. Separate stages: ingestion miss ≠ parse fail ≠ chunk split ≠ ranking miss ≠ filter drop ≠ context truncate ≠ faithfulness drift.
  5. Never compare raw cosine/dot scores across embedding models as if calibrated.
  6. Change one variable at a time; keep a frozen baseline run for every PR.
  7. ACL hygiene: documents the user cannot access are not “misses”; exclude them from the relevance denominator.

Failure taxonomy (diagnose in this order)

StageSymptomTypical root cause
IngestionRelevant source_id absent from indexSync lag, failed job, wrong collection, deleted tombstone
ParseDoc present; answer text garbled/missingPDF/table/OCR loss; HTML boilerplate; empty extracted body
ChunkAnswer split across boundaries; half-table rowsFixed-token splits; tiny overlap; footer pollution in embeddings
Embed/indexNear-zero or nonsense scores; empty after migrationWrong embedding_model on query vs corpus; metric mismatch
RetrieveRelevant ID never in top-K (any arm)Weak dense match on IDs/codes; BM25 arm dead; too-small fetch_k
Fuse/thresholdHybrid “works” but lexical-only hits vanishSimilarity floor applied after RRF to fused list
Filter/ACLEmpty or wrong-tenant hitsPost-filter after ANN (fetch_k then filter) vs pre-filter; over-tight metadata
Rerank/packID retrieved then dropped from final promptReranker demotion; duplicate collapse; token budget truncation
GenerateCorrect chunks in prompt; answer still wrongFaithfulness — not a retrieval fix (track separately)

Practitioner pattern (r/Rag): treat “RAG is inaccurate” as retrieval until proven otherwise; only then score faithfulness against the assembled context.


Metrics — what each number means

Use ID-based metrics when gold chunk/doc IDs exist (preferred). Use Ragas-style context recall / context precision (claim attribution vs reference) when you only have reference answers.

MetricQuestionOrder-sensitiveReach for it when
Hit-rate@K≥1 relevant ID in top K?noCheap CI smoke
Recall@KFraction of gold IDs in top KnoMissing evidence is costly
Precision@KFraction of top K that are relevantnoContext budget is tight
MRRReciprocal rank of first relevantyesGenerator mostly uses top 1–2
nDCG@KGraded ranking qualityyesReranker / graded labels
Context recall (Ragas)Reference claims supportable from retrieved textn/aNo gold chunk IDs yet
Context precision (Ragas)Relevant chunks ranked early vs noiseyesNoise displaces signal

Diagnostic pairs

  • Recall↓, MRR irrelevant → chunking / embedding / not in index.
  • Recall steady, MRR↓ → ranking / missing reranker / fusion weights.
  • Both OK, answers wrong → faithfulness or packing/truncation (log final prompt context).
  • Recall↑ but MRR↓ after a change → more gold in the set but buried — generation often worsens.

Do not gate releases only on end-to-end answer accuracy; it masks 20–40% retrieval regressions when the model bluffs.


Audit procedure

1. Inputs to collect (block if missing)

  • Bounded query set (≥30 real user questions; synthetic-only sets are optional extras).
  • Per query: eligible relevant chunk IDs under the same ACL as production.
  • Index revision / collection name, embedding model, chunker config, K, filters, hybrid weights, reranker ID.
  • Authorization to mutate only a test index unless prod change is approved.

2. Per-query trace schema (mandatory fields)

Log for every audited query:

{
  "query_id": "q-1042",
  "query_text": "...",
  "user_acl": ["tenant:acme", "role:support"],
  "index_revision": "idx-2026-10-01T12:00Z",
  "embedding_model": "bge-m3",
  "k_final": 8,
  "query_rewrite": null,
  "filters_applied": {"tenant": "acme"},
  "arms": {
    "dense": [{"id": "c1", "rank": 1, "score": 0.71}],
    "bm25": [{"id": "c9", "rank": 1, "score": 12.4}]
  },
  "fused": [{"id": "c1", "rank": 1, "rrf": 0.032, "sources": ["dense"]}],
  "after_rerank": [{"id": "c9", "rank": 1, "score": 0.88}],
  "final_context_ids": ["c9", "c1", "c3"],
  "final_context_chars": 6120,
  "gold_ids": ["c9"],
  "notes": ""
}

Critical: log final_context_ids (what the model saw), not only first-stage retrieval. Truncation and rerank drops are common silent failures.

3. Score and classify

For each query, mark primary failure stage from the taxonomy. Aggregate:

  • Mean Hit-rate@K, Recall@K, Precision@K, MRR, nDCG@K at production K.
  • Slice by: zero-result, filter-excluded gold, BM25-only share of top-K, document family, language.

4. Single-variable experiments

Preserve baseline. Typical order of levers (stop when Recall@K and MRR meet gate):

  1. Confirm gold sources are indexed and parsed (spot-open chunk text).
  2. Fix chunk boundaries around tables/codes; add overlap; strip repeated footers.
  3. Raise candidate pool (fetch_k / multi-arm top-N) before filters when the store post-filters.
  4. Prefer pre-filter (or filtered ANN) over fetch-then-filter when tenants are sparse in the ANN shortlist (LangChain FAISS filter ordering).
  5. Hybrid: ensure BM25-only hits are not killed by a post-fusion vector similarity floor (r/Rag hybrid bug reports).
  6. Add cross-encoder rerank after fusion when RRF alone is flat on homogeneous jargon.
  7. Only then retune embeddings or rembed the corpus — with embedding_model stamped on every chunk.

5. Deliverable

Ship:

  1. Per-query failure table (query_id, stage, gold_ids, final_context_ids, hit, ranks).
  2. Aggregate metrics before/after with identical K and ACL rules.
  3. Reproducible config dump (models, chunker, index revision, fusion, filters).
  4. Explicit “not retrieval” bucket for faithfulness follow-ups.
  5. ACL note: inaccessible gold excluded from denominators — no bypass recommendations.

Production example: ID-based retrieval eval harness

"""Minimal RAG retrieval audit: Recall@K, Precision@K, MRR, hit-rate."""
from __future__ import annotations

from dataclasses import dataclass
from typing import Callable, Sequence


@dataclass(frozen=True)
class Case:
    query_id: str
    query: str
    gold_ids: frozenset[str]  # eligible under the same ACL


def recall_at_k(retrieved: Sequence[str], gold: frozenset[str], k: int) -> float:
    if not gold:
        return float("nan")  # exclude ACL-empty cases from means
    top = list(retrieved)[:k]
    return len(gold.intersection(top)) / len(gold)


def precision_at_k(retrieved: Sequence[str], gold: frozenset[str], k: int) -> float:
    top = list(retrieved)[:k]
    if not top:
        return 0.0
    return len(gold.intersection(top)) / len(top)


def mrr(retrieved: Sequence[str], gold: frozenset[str], k: int) -> float:
    for i, doc_id in enumerate(list(retrieved)[:k], start=1):
        if doc_id in gold:
            return 1.0 / i
    return 0.0


def hit_rate(retrieved: Sequence[str], gold: frozenset[str], k: int) -> float:
    return 1.0 if gold.intersection(list(retrieved)[:k]) else 0.0


def audit(
    cases: Sequence[Case],
    retrieve_final_context_ids: Callable[[str], Sequence[str]],
    k: int,
) -> list[dict]:
    rows = []
    for case in cases:
        ids = list(retrieve_final_context_ids(case.query))
        rows.append(
            {
                "query_id": case.query_id,
                "hit": hit_rate(ids, case.gold_ids, k),
                "recall": recall_at_k(ids, case.gold_ids, k),
                "precision": precision_at_k(ids, case.gold_ids, k),
                "mrr": mrr(ids, case.gold_ids, k),
                "retrieved": ids[:k],
                "gold": sorted(case.gold_ids),
            }
        )
    return rows


# Wire retrieve_final_context_ids to YOUR packer (post-filter, post-rerank, post-truncate).
# Prefer LlamaIndex RetrieverEvaluator or Ragas IDBasedContextRecall when already in-stack:
# https://developers.llamaindex.ai/python/examples/evaluation/retrieval/retriever_eval/
# https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/

Canary without a full golden set

Log fusion_source per top-K hit (dense | bm25 | both). If BM25-only share flatlines after a deploy, hybrid may be silently vector-only.


Technical troubleshooting matrix

Failure signatureRoot causeDiagnostic & fix
Empty results for one tenantANN then metadata filter; fetch_k too smallPre-filter or raise fetch_k; verify filter field cardinality
Exact error codes / SKUs missedDense-only retrievalAdd BM25/sparse arm; query rewrite that keeps literals
Hybrid no better than densePost-RRF similarity threshold drops BM25-onlyThreshold on dense arm only; keep lexical survivors
High similarity, wrong answerChunk lacks full fact (boundary/table split)Structural chunking; inspect gold span vs chunk text
Correct doc, wrong chunkBoilerplate-dominated embeddingsStrip headers/footers; smaller semantic units
Scores ~0 but “good” answersDistance/metric or embed-model mismatchPin same embed_model on index build and query
Recall OK, users still complainFaithfulness / truncation of packed contextDiff retrieved vs final_context_ids; generation eval
Metric improved, UX worseRecall↑ MRR↓Add reranker; report both metrics in CI

Best practices

  1. Build the eval set from real support/search queries, not only synthetic FAQ paraphrases.
  2. Version corpus + chunker + embedding model together; stamp embedding_model on chunks for gradual rembeds.
  3. Deduplicate near-identical chunks before fusion to stop footer clones from crowding top-K.
  4. Keep retrieval CI separate from answer faithfulness CI (@ai-evaluation-dataset, @ai-citation-verification).
  5. For evidence-dense docs, prefer structural/hierarchical chunk checks; for sparse factoid QA, simple token/sentence baselines are often enough — measure, don’t assume.
  6. Gate merges on Hit-rate@K + Recall@K + MRR with thresholds chosen from your baseline, not internet folklore.

Limitations

  • This skill does not certify production quality for every vector DB or framework version.
  • LLM-judge context metrics are proxies; disagreement with ID-based labels should be investigated.
  • Multimodal/table-heavy corpora need parser-specific audits beyond text metrics.
  • Stop and ask if ACL rules, index write permission, or success thresholds are missing.

Related skills

  • @ai-evaluation-dataset — build/version labeled query sets and gold IDs
  • @llm-json-contract-check — structured judge / scorer outputs
  • @prompt-regression-gate — prompt/packer changes with frozen retrieval
  • @ai-citation-verification — citation support against retrieved passages
  • @ai-pii-redaction-review — redact traces before sharing eval dumps
  • @llm-cost-latency-benchmark — cost/latency of rerank + larger K
  • @qdrant / @chromadb / @elasticsearch — store-specific filter and hybrid knobs

Agent Operational Directive

MANDATORY: Audit retrieval with stage isolation and metrics at production K. Log final packed context IDs. Change one variable at a time against a frozen baseline. Never recommend ACL bypasses to inflate recall. Report retrieval metrics separately from answer quality, latency, and cost.


Source anchors (research)

rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall