ericrisco/rsc-harness / skills/embeddings-search
embeddings-search
VerifiedUse when choosing an embedding model, chunk size, or query form, when semantic search returns irrelevant results, when adding hybrid BM25+vector or a reranker, or when a retrieval change needs a number (recall@k, nDCG, MRR). NOT operating the store — index tuning, quantization (that is `vector-db`) — nor the retrieve-to-answer loop (that is `rag`).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 10.6 KB |
| evals/README.md | markdown | 725 B |
| evals/cases.yaml | config | 3.3 KB |
| references/evaluation.md | markdown | 3.2 KB |
| references/models.md | markdown | 3.4 KB |
| scripts/verify.shexecutable | script | 5.2 KB |
references/evaluation.md
View on GitHub# Retrieval evaluation
Turn "search is bad" into a number you can move. Everything here runs offline against a labeled
query set — no LLM judge, no network.
## 1. Build the golden query set
The golden set is the asset. Aim for **30–50+** entries; more is better. Each entry is a real
question plus the doc ids that genuinely answer it.
```json
[
{"query": "how long do refunds take", "relevant": ["doc_412", "doc_77"]},
{"query": "SKU AB-9931 specs", "relevant": ["doc_1290"]},
{"query": "puc cancel·lar la subscripció?", "relevant": ["doc_88", "doc_640"]}
]
```
Sourcing tips:
- Pull real queries from logs; don't invent only easy ones — include the exact-match and the
multilingual cases that break dense-only retrieval.
- Label relevance against the **chunk/doc id your retriever returns**, not the source PDF.
- Keep it in version control. A change to the set invalidates old numbers.
## 2. Metrics — what each one catches
- **recall@k**: fraction of an entry's relevant docs found in the top-k. Catches *the right
chunk never came back* (a recall problem no reranker can fix).
- **nDCG@k**: discounted gain that rewards relevant docs ranked higher (uses `log2(rank+1)`
discount). Catches *right docs, wrong order*.
- **MRR**: mean of `1/rank` of the first relevant doc. Catches *the one answer is buried deep*.
## 3. Runnable skeleton
```python
import math
def recall_at_k(retrieved, relevant, k):
top = set(retrieved[:k])
rel = set(relevant)
return len(top & rel) / len(rel) if rel else 0.0
def ndcg_at_k(retrieved, relevant, k):
rel = set(relevant)
dcg = sum(1.0 / math.log2(i + 2) for i, d in enumerate(retrieved[:k]) if d in rel)
ideal = sum(1.0 / math.log2(i + 2) for i in range(min(len(rel), k)))
return dcg / ideal if ideal else 0.0
def mrr(retrieved, relevant):
rel = set(relevant)
for i, d in enumerate(retrieved):
if d in rel:
return 1.0 / (i + 1)
return 0.0
def evaluate(golden, search_fn, k=10):
rs, ns, ms = [], [], []
for row in golden:
got = search_fn(row["query"]) # -> ordered list of doc ids
rs.append(recall_at_k(got, row["relevant"], k))
ns.append(ndcg_at_k(got, row["relevant"], k))
ms.append(mrr(got, row["relevant"]))
n = len(golden)
return {
f"recall@{k}": sum(rs) / n,
f"ndcg@{k}": sum(ns) / n,
"mrr": sum(ms) / n,
}
```
## 4. A/B one change at a time
```text
baseline = evaluate(golden, search_v0) # record it
candidate = evaluate(golden, search_v1) # changed ONE thing
# compare on the SAME golden set; keep the change only if the metric you care about went up
```
Change exactly one variable per run: the embedding model, OR the chunk size/strategy, OR the
fusion (`k` in RRF, fan-in), OR the reranker. Stacking changes means you can't attribute the
delta — and a regression hides inside an improvement.
Typical decision pattern:
- recall@k low → upstream: model choice, chunking, add sparse/hybrid. A reranker won't help.
- recall@k high but nDCG/MRR low → ordering: add or tune the reranker.
- exact-term queries failing while paraphrases pass → add BM25/sparse and re-measure.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |