Skip to content

ericrisco/rsc-harness / skills/embeddings-search

embeddings-search

Verified

Use when choosing an embedding model, chunk size, or query form, when semantic search returns irrelevant results, when adding hybrid BM25+vector or a reranker, or when a retrieval change needs a number (recall@k, nDCG, MRR). NOT operating the store — index tuning, quantization (that is `vector-db`) — nor the retrieve-to-answer loop (that is `rag`).

FileKindSize
SKILL.mdskill md10.6 KB
evals/README.mdmarkdown725 B
evals/cases.yamlconfig3.3 KB
references/evaluation.mdmarkdown3.2 KB
references/models.mdmarkdown3.4 KB
scripts/verify.shexecutablescript5.2 KB

references/evaluation.md

View on GitHub
# Retrieval evaluation

Turn "search is bad" into a number you can move. Everything here runs offline against a labeled
query set — no LLM judge, no network.

## 1. Build the golden query set

The golden set is the asset. Aim for **30–50+** entries; more is better. Each entry is a real
question plus the doc ids that genuinely answer it.

```json
[
  {"query": "how long do refunds take", "relevant": ["doc_412", "doc_77"]},
  {"query": "SKU AB-9931 specs",        "relevant": ["doc_1290"]},
  {"query": "puc cancel·lar la subscripció?", "relevant": ["doc_88", "doc_640"]}
]
```

Sourcing tips:
- Pull real queries from logs; don't invent only easy ones — include the exact-match and the
  multilingual cases that break dense-only retrieval.
- Label relevance against the **chunk/doc id your retriever returns**, not the source PDF.
- Keep it in version control. A change to the set invalidates old numbers.

## 2. Metrics — what each one catches

- **recall@k**: fraction of an entry's relevant docs found in the top-k. Catches *the right
  chunk never came back* (a recall problem no reranker can fix).
- **nDCG@k**: discounted gain that rewards relevant docs ranked higher (uses `log2(rank+1)`
  discount). Catches *right docs, wrong order*.
- **MRR**: mean of `1/rank` of the first relevant doc. Catches *the one answer is buried deep*.

## 3. Runnable skeleton

```python
import math

def recall_at_k(retrieved, relevant, k):
    top = set(retrieved[:k])
    rel = set(relevant)
    return len(top & rel) / len(rel) if rel else 0.0

def ndcg_at_k(retrieved, relevant, k):
    rel = set(relevant)
    dcg = sum(1.0 / math.log2(i + 2) for i, d in enumerate(retrieved[:k]) if d in rel)
    ideal = sum(1.0 / math.log2(i + 2) for i in range(min(len(rel), k)))
    return dcg / ideal if ideal else 0.0

def mrr(retrieved, relevant):
    rel = set(relevant)
    for i, d in enumerate(retrieved):
        if d in rel:
            return 1.0 / (i + 1)
    return 0.0

def evaluate(golden, search_fn, k=10):
    rs, ns, ms = [], [], []
    for row in golden:
        got = search_fn(row["query"])          # -> ordered list of doc ids
        rs.append(recall_at_k(got, row["relevant"], k))
        ns.append(ndcg_at_k(got, row["relevant"], k))
        ms.append(mrr(got, row["relevant"]))
    n = len(golden)
    return {
        f"recall@{k}": sum(rs) / n,
        f"ndcg@{k}":   sum(ns) / n,
        "mrr":         sum(ms) / n,
    }
```

## 4. A/B one change at a time

```text
baseline   = evaluate(golden, search_v0)        # record it
candidate  = evaluate(golden, search_v1)         # changed ONE thing
# compare on the SAME golden set; keep the change only if the metric you care about went up
```

Change exactly one variable per run: the embedding model, OR the chunk size/strategy, OR the
fusion (`k` in RRF, fan-in), OR the reranker. Stacking changes means you can't attribute the
delta — and a regression hides inside an improvement.

Typical decision pattern:
- recall@k low → upstream: model choice, chunking, add sparse/hybrid. A reranker won't help.
- recall@k high but nDCG/MRR low → ordering: add or tune the reranker.
- exact-term queries failing while paraphrases pass → add BM25/sparse and re-measure.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall