Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/lancedb

lancedb

Verified

<!--
CIANCHOSAINT wholesale-copy of cianfhoghlaim/cianfhoghlaim @ main branch.

Original: cianfhoghlaim/cianfhoghlaim (per the openspec/changes/cianchosaint-repo-bootstrap-v2/specs/cianchosaint-bootstrap-v2/spec.md).
Migrated to cianchosaint: 2026-08-23
Licence: BUSL-1.1 (per LICENSE.md)

Wholesale-copied from cianfhoghlaim/cianfhoghlaim@lancedb/SKILL.md.

UPDATED 2026-09-26 (per openspec/changes/2026-09-26-lancedb-v0.39-lance-v2-upgrade-v1,
Stage 5 of the package-version-drift saga): The 24 BIEP companion tables are
now in Lance v2 format (the v2 format was added in lancedb 0.10 and is the
default in 0.39). The migration was in-place (one-time cutover per the
locked Q3). For existing v1 tables, run `scripts/migrate_lance_v1_to_v2.py`.
Per https://www.lancedb.com/blog/lance-file-format-2-2-taming-complex-data:
'The migration path is straightforward: existing projects can follow the
batched migration examples in the docs to upgrade their tables.'
-->
---
name: lancedb
description: Expert assistance for vector database development with LanceDB. Use when users need vector search, semantic search, RAG applications, hybrid search, multimodal embeddings, time-travel / versioned RAG, LanceDB Cloud, Lance + Iceberg, Ibis + lance_scan, the embeddings registry, or production-scale vector storage. Powers the BIEP `cianfhoghlaim.lc.<subject>.<level>_<lang>` companion tables (the canonical v1 CocoIndex `mount_table_target` target for the 6 LC subjects + `gov.ie` circulars) using the shared `BAAI/bge-m3` 1024-d embedder.
---

# LanceDB - Embedded Vector Database

**Version:** >=0.39,<1.0 | **Last Updated:** 2026-09-26

## Overview

LanceDB is an open-source, embedded vector database for AI applications:

- **Embedded**: Runs in-process without a separate server
- **Multimodal**: Store vectors, text, images, and audio together
  ("fat table" pattern with BLOBs + vectors in the same row)
- **Scalable**: Billion-scale vectors with disk-based indexes
- **Cloud-Native**: S3-compatible storage with serverless option
  (LanceDB Cloud, regions: us-east-1, us-west-2, eu-west-1, ap-south-1)
- **HNSW-backed IVF Indexing**: HNSW is **not** a top-level Python
  index in LanceDB — it is exposed only as a sub-index inside IVF
  partitions: `IVF_HNSW_FLAT` (no quant), `IVF_HNSW_SQ` (scalar quant,
  best recall/latency trade-off), `IVF_HNSW_PQ` (product quant).
  Use the `IvfHnswSq` config class in async Python.
- **MVCC Safety**: Multi-version concurrency control for safe concurrent
  operations (use `lancedb.connect(...)` + `SerialDatabaseExecutor`)
- **Hybrid Search**: Combine vector and full-text search with RRF reranking
- **Time-travel**: `table.checkout(version)` for versioned RAG, A/B testing
  of embedding models, knowledge-base audits
- **Lance Namespace / Iceberg**: expose Lance tables as Iceberg to PyIceberg
  consumers
- **Lance + Ray**: distributed indexing for > 1M rows
- **Ibis + DuckDB**: federated SQL over Lance via `lance_scan()`

**Documentation**: https://lancedb.github.io/lancedb/

## When to Use This Skill

Activate when users need:

- "Build a RAG application with vector search"
- "Store and search embeddings"
- "Implement semantic search"
- "Combine vector and full-text search" (vector + BM25 with RRF reranking)
- "Store multimodal data (images, text)" — use the multimodal "fat table"
  pattern, not the "pointer strategy"
- "Version my RAG index / A/B test embedding models" — use
  `table.checkout(version)` and `table.version`
- "Use OpenAI / Cohere / HuggingFace / Gemini / Bedrock / Ollama
  embeddings" — use the `get_registry().get(...)` pattern
- "Query Lance from DuckDB / marimo" — use Ibis + `lance_scan()`
- "Expose Lance as Iceberg to PyIceberg" — use `lance.namespace`
- "Deploy to LanceDB Cloud" — see the LanceDB Cloud regions +
  auto-compaction + auto-reindexing section
- "Run a TS / Next.js app against Lance" — use the modern TS API
  (`search()` not `vectorSearch()`)

## Core Concepts

### 1. Connection and Tables

```python
import lancedb
import pyarrow as pa

# Local connection
db = lancedb.connect("data/my-database")

# Cloud connection
db = lancedb.connect("db://my-database", api_key="...", region="us-east-1")

# S3 connection
db = lancedb.connect("s3://my-bucket/lancedb")

# Create table with data
data = [
    {"id": 1, "text": "Hello world", "vector": [0.1] * 128},
    {"id": 2, "text": "Goodbye world", "vector": [0.2] * 128}
]
table = db.create_table("documents", data=data)

# Open existing table
table = db.open_table("documents")

# List tables
print(db.table_names())
```

### 2. Schema Definition

```python
import pyarrow as pa

schema = pa.schema([
    pa.field("id", pa.int64()),
    pa.field("text", pa.string()),
    pa.field("vector", pa.list_(pa.float32(), 128)),  # 128-dim vector
    pa.field("metadata", pa.string()),
    pa.field("tags", pa.list_(pa.string()))
])

table = db.create_table("documents", schema=schema)
```

### 3. Vector Search

```python
# Basic search
query_vector = [0.15] * 128
results = table.search(query_vector).limit(10).to_pandas()

# With metadata filtering (pre-filter)
results = (table.search(query_vector)
          .where("category = 'tech'")
          .limit(10)
          .to_pandas())

# Specify distance metric
results = (table.search(query_vector)
          .metric("cosine")  # cosine, l2, dot
          .limit(10)
          .to_pandas())

# Select specific columns
results = (table.search(query_vector)
          .select(["id", "text"])
          .limit(10)
          .to_pandas())
```

### 4. Full-Text Search

```python
# Create FTS index
table.create_fts_index("text")

# Perform full-text search
results = table.search("machine learning", query_type="fts").limit(10).to_pandas()

# Multilingual / Irish FTS — use the right tokenizer
table.create_fts_index(
    "text",
    tokenizer="en_stem",  # or "default", "whitespace", "raw", "jieba" (zh),
                          # "no", "fr", "de", "es", "it", "pt", "ru", …
    with_stopwords=["the", "a", "an"],  # optional stopword list
)

# Lowercase + ASCII folding for multilingual corpora
table.create_fts_index("text", language="English", stem=True, remove_stop_words=True)
```

**For multilingual corpora (Irish, Scottish Gaelic, Welsh, Breton)**,
use the `default` tokenizer (Unicode-aware) and pre-normalise the
text via `unicodedata.normalize("NFKC", text)` before insert. For
English-heavy corpora, use `en_stem` for Porter-stemmed matching.

### 5. Hybrid Search

```python
# Combine vector and full-text search with RRF (Reciprocal Rank Fusion)
results = (table.search(query_type="hybrid")
          .vector(query_vector)
          .text("machine learning")
          .limit(10)
          .rerank(method="rrf")
          .to_pandas())

# Pre-filter vs post-filter
results = (table.search(query_vector)
          .where("category = 'tech'")     # pre-filter (faster, selective)
          .limit(10)
          .to_pandas())

results = (table.search(query_vector)
          .where("score > 0.8")
          .prefilter(False)               # post-filter (slower, non-selective)
          .limit(10)
          .to_pandas())

# refine_factor oversampling for higher accuracy
results = (table.search(query_vector)
          .refine_factor(10)              # oversample 10x then re-rank
          .limit(10)
          .to_pandas())
```

### 6. Indexing

**IVF-PQ Index** (for large datasets):
```python
table.create_index(
    metric="cosine",
    index_type="IVF_PQ",
    num_partitions=256,
    num_sub_vectors=16
)
```

**HNSW-backed IVF Index** (HNSW is NOT top-level — see vector-index.md):
```python
table.create_index(
    metric="cosine",
    index_type="IVF_HNSW_SQ",   # or "IVF_HNSW_FLAT" for unquantized
    num_partitions=64,          # num_rows // 1_048_576 starting point
    m=20,
    ef_construction=150,
)

# Async / config-object form (preferred for new code):
from lancedb.index import IvfHnswSq
await table.create_index("vector", config=IvfHnswSq(
    distance_type="cosine", num_partitions=64, m=20, ef_construction=150,
))
```

## Common Patterns

### RAG Application

```python
import lancedb
from lancedb.embeddings import get_registry
from lancedb.pydantic import LanceModel, Vector

# Setup embedding model
model = get_registry().get("openai").create(name="text-embedding-3-small")

class Document(LanceModel):
    text: str = model.SourceField()
    vector: Vector(model.ndims()) = model.VectorField()
    source: str
    chunk_id: int

# Create table
db = lancedb.connect("~/.lancedb")
table = db.create_table("documents", schema=Document)

# Add documents (embeddings auto-generated)
docs = [
    {"text": "LanceDB is a vector database", "source": "docs", "chunk_id": 1},
    {"text": "Vector search enables semantic retrieval", "source": "blog", "chunk_id": 2}
]
table.add(docs)

# Create indexes
table.create_fts_index("text")
table.create_index(metric="cosine")

# Search
def search_documents(query: str, limit: int = 5):
    results = (table.search(query, query_type="hybrid")
              .limit(limit)
              .to_pydantic(Document))
    return results

results = search_documents("How does semantic search work?")
```

### Image Similarity

```python
from transformers import CLIPProcessor, CLIPModel
import torch
from PIL import Image

# Load CLIP
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

def get_image_embedding(image_path):
    image = Image.open(image_path)
    inputs = processor(images=image, return_tensors="pt")
    with torch.no_grad():
        embedding = model.get_image_features(**inputs)
    return embedding[0].numpy().tolist()

# Create image database
db = lancedb.connect("./image_db")
data = [
    {"path": img, "embedding": get_image_embedding(img), "category": cat}
    for img, cat in image_files
]
table = db.create_table("images", data)
table.create_index(metric="cosine")

# Search similar images
query_embedding = get_image_embedding("query.jpg")
results = table.search(query_embedding).limit(10).to_pandas()
```

### Incremental Updates

```python
# Add new data
new_data = [{"id": 3, "text": "New document", "vector": [0.3] * 128}]
table.add(new_data)

# Update records
table.update(where="id = 1", values={"text": "Updated text"})

# Delete records
table.delete("id IN (1, 2)")

# Compact table (merge fragments)
table.compact()
```

### Multi-Tenant Setup

```python
def get_tenant_table(tenant_id: str):
    db = lancedb.connect(f"s3://my-bucket/{tenant_id}")
    return db.open_table("documents")

# Each tenant has isolated data
tenant_a = get_tenant_table("tenant-a")
tenant_b = get_tenant_table("tenant-b")
```

## TypeScript Usage (modern API)

The **deprecated** `vectorSearch(...)` API has been replaced by
`search(...)` with explicit `queryType`. Use the modern API:

```typescript
import * as lancedb from "@lancedb/lancedb";
import { embedding, rerankers } from "@lancedb/lancedb";
import { Field, Float32, FixedSizeList, Schema, Utf8 } from "apache-arrow";
import "dotenv/config";

const db = await lancedb.connect("data/my-database");

// Declarative schema via the embeddings registry
const openai = embedding.getRegistry().get("openai");
const embedModel = openai.create({ model: "text-embedding-3-small" });

const schema = new Schema([
  new Field("id", new Utf8(), false),
  new Field("text", new Utf8(), false),
  new Field("vector", new FixedSizeList(1024, new Field("item", new Float32())), false),
]);

// Vector search (modern API — `search()`, not `vectorSearch()`)
const table = await db.openTable("documents");
const results = await table
  .search(Array.from(embedModel.embed("machine learning")))
  .limit(10)
  .toArray();

// Hybrid search with RRF reranking
const hybrid = await table
  .search(queryType: "hybrid", Array.from(embedModel.embed("machine learning")), "machine learning")
  .rerank(rerankers.RRFReranker())
  .limit(10)
  .toArray();

// LanceDB Cloud
// .env: LANCEDB_URI=db://my-database, LANCEDB_API_KEY=...
const cloud = await lancedb.connect(process.env.LANCEDB_URI!, {
  apiKey: process.env.LANCEDB_API_KEY!,
  region: "eu-west-1",
});
```

**For the full modern TS reference**, see
[`references/typescript-modern-api.md`](references/typescript-modern-api.md).

## Distance Metrics

| Metric | Use Case | Range |
|--------|----------|-------|
| `l2` (Euclidean) | General purpose | [0, inf), lower = similar |
| `cosine` | Unnormalized embeddings | [-1, 1], higher = similar |
| `dot` | Normalized embeddings | [-1, 1], higher = similar |

## Index Selection Guide

| Scenario | Index Type | Parameters |
|----------|-----------|------------|
| <100K vectors | None (brute force) | - |
| Memory constrained | IVF_PQ | num_partitions=256 |
| Accuracy critical | IVF_HNSW_SQ | num_partitions = num_rows // 1_048_576, m=20, ef_construction=150 |
| Best recall/size | IVF_HNSW_FLAT | m=20, ef_construction=150, no quant (raw + HNSW) |
| Max compression | IVF_RQ | num_partitions = num_rows // 4096, RaBitQ |
| Small dim (≤256) | IVF_PQ | num_partitions = num_rows // 4096, num_sub_vectors = dim // 8 |

## Advanced Patterns

### Time-travel / Versioned RAG

```python
# Get the current version
current = table.version  # e.g. 7

# Query a specific historical version (read-only)
historical = table.checkout(current - 1)
results = historical.search(query_vector).limit(10).to_pandas()

# Restore a previous version (writes a new version with the old data)
table.restore(4)  # creates version 8 with the data from version 4

# A/B test two embedding models: index the same data twice
# (once with model A, once with model B), then search both tables
# and compare scores.
```

Use cases: model rollback, A/B testing, knowledge-base audits,
reproducible experiments. See
[`references/time-travel-rag.md`](references/time-travel-rag.md).

### Embeddings Registry (10+ providers)

```python
from lancedb.embeddings import get_registry

registry = get_registry()

# OpenAI
model = registry.get("openai").create(name="text-embedding-3-small")

# Cohere
model = registry.get("cohere").create(name="embed-english-v3.0")

# HuggingFace (local)
model = registry.get("huggingface").create(
    name="sentence-transformers/all-MiniLM-L6-v2",
    device="cuda",
)

# Sentence-Transformers
model = registry.get("sentence-transformers").create(
    name="BAAI/bge-large-en-v1.5",
)

# ColBERT (multi-vector)
model = registry.get("colbert").create(name="colbert-ir/colbertv2.0")

# Gemini
model = registry.get("gemini").create(name="text-embedding-004")

# Bedrock
model = registry.get("bedrock").create(name="amazon.titan-embed-text-v1")

# Ollama (local)
model = registry.get("ollama").create(name="nomic-embed-text")

# OpenCLIP (multimodal image+text)
model = registry.get("open-clip").create(name="ViT-B-32")
```

Use `LanceModel` with `SourceField()` and `VectorField()` to bind the
embedder to a column:

```python
from lancedb.pydantic import LanceModel, Vector

class Document(LanceModel):
    text: str = model.SourceField()
    vector: Vector(model.ndims()) = model.VectorField()
    filename: str
```

See [`references/embed-functions-registry.md`](references/embed-functions-registry.md).

### Multimodal "fat table" pattern

```python
import pyarrow as pa

# Store images / audio / PDFs as BLOBs in the same row as the embedding
schema = pa.schema([
    pa.field("id", pa.int64()),
    pa.field("filename", pa.string()),
    pa.field("image_blob", pa.large_list(pa.uint8())),  # BLOB
    pa.field("image_embedding", pa.list_(pa.float32(), 768)),  # CLIP
    pa.field("description", pa.string()),
    pa.field("tags", pa.list_(pa.string())),
])

table = db.create_table("multimodal", schema=schema)

# Insert
table.add([{
    "id": 1,
    "filename": "recipe.png",
    "image_blob": open("recipe.png", "rb").read(),  # bytes
    "image_embedding": clip_embed("recipe.png"),
    "description": "Beef stew with root vegetables",
    "tags": ["main", "irish"],
}])

# Range-read the BLOB only for top-K results (avoid full-row reads)
results = table.search(query_vec).limit(10).to_pandas()
for r in results.itertuples():
    img = pa.ipc.open_stream(r.image_blob).read()  # or just use the bytes
```

For very large BLOBs (> 1 MB), prefer the "pointer strategy" — store
the BLOB in S3/R2 and put the URL in the row.

### Lance Namespace / Iceberg

Expose a Lance table to Iceberg consumers (PyIceberg, DuckDB
`iceberg_attach`):

```python
import lance.namespace

# Connect to an Iceberg REST catalog (e.g. Lakekeeper, Polaris)
ns = lance.namespace.connect(
    "iceberg",
    REST_URL="http://lakekeeper:8181/catalog",
    S3_ENDPOINT="http://minio:9000",
    S3_ACCESS_KEY_ID="...",
    S3_SECRET_ACCESS_KEY="...",
)

# Register an existing Lance table as an Iceberg table
ns.create_namespace("cianfhoghlaim")
ns.create_table("cianfhoghlaim.leabharlann_books", metadata={"lance_uri": "s3://lance/leabharlann_books"})

# Query from PyIceberg
from pyiceberg.catalog import load_catalog
catalog = load_catalog("cianfhoghlaim", type="rest", uri="http://lakekeeper:8181/catalog")
tbl = catalog.load_table("cianfhoghlaim.leabharlann_books")
df = tbl.scan().to_pandas()
```

### Ibis + DuckDB `lance_scan()`

```python
import duckdb

con = duckdb.connect()
con.execute("INSTALL lance; LOAD lance;")
con.execute(
    "CREATE VIEW books AS SELECT * FROM lance_scan('s3://lance/leabharlann_books')"
)
# Now SQL-federated queries over Lance from marimo notebooks:
df = con.execute(
    "SELECT filename, text FROM books WHERE subject = 'irish' LIMIT 10"
).df()
```

### Geospatial + FTS

```python
# Compute distance with the built-in geo operator
results = table.search("best fish and chips", query_type="hybrid")
    .where("distance(lat, lon, 53.2707, -9.0568) < 5")  # within 5 km of Galway
    .prefilter(False)  # post-filter (geo is non-selective)
    .limit(10)
    .to_pandas()
```

### `explain_plan` / `analyze_plan` / `drop_index`

```python
# Diagnose slow queries
plan = table.search(query_vector).limit(10).explain_plan(verbose=True)
print(plan)

# Run the query with metrics
metrics = table.search(query_vector).limit(10).analyze_plan()

# Drop an index (e.g. before a bulk insert, then re-create)
table.drop_index("vector_idx")
# … bulk insert …
table.create_index(metric="cosine", index_type="HNSW", m=20, ef_construction=150)
```

## Best Practices

1. **Use Float16 for vectors** — 50% storage savings
2. **Store metadata with vectors** — avoid joins
3. **Pre-filter when the filter is selective** (> 1% of rows);
   **post-filter** when the filter is non-selective
4. **Use `refine_factor`** for high-accuracy queries
5. **Compact regularly** — merge fragments for performance
6. **Batch inserts** — 1000-10000 rows at a time
7. **Drop indexes before bulk inserts > 50K rows** (HNSW rebuild is slow)
8. **Use the embeddings registry** — never roll your own embedder
9. **Use the multimodal "fat table"** for small BLOBs (< 1 MB);
   use the "pointer strategy" (S3 + URL) for larger
10. **Use `time-travel`** for model rollback, A/B testing, and audits
11. **FTS tokenizer matters for multilingual corpora** — use
    `default` (Unicode) for Irish, Welsh, etc.; `en_stem` for English
12. **TS: use `search()`, not the deprecated `vectorSearch()`**
13. **The Cianfhoghlaim canonical embedding model is `BAAI/bge-m3`**
    (1024-d, multilingual, 100+ languages including all 6
    Celtic languages, MIT-licensed). Use it for any
    multilingual RAG corpus. Cache the model weights at
    `stedding/huggingface/hub/models--BAAI--bge-m3/`.
14. **Reconciled model selection** (resolves the
    `EMBEDDINGS.md` conflict): the
    `MultiModelEmbedder` in `.agents/skills/cocoindex/SKILL.md`
    routes Irish (`ga`) to `GaBERT`
    (`DCU-NLP/bert-base-irish-cased-v1`, 768-d) for
    linguistic accuracy (séimhiú, urú, dialectal variation),
    and everything else to `BGE-M3` (1024-d, multilingual).
    Both stored in the same LanceDB table with a `model_name`
    column.

## SQL Queries

```python
# Full SQL support via DataFusion
results = db.sql("SELECT * FROM documents WHERE score > 0.8").to_pandas()

# Aggregations
stats = db.sql("""
    SELECT category, AVG(score) as avg_score
    FROM documents
    GROUP BY category
""").to_pandas()
```

## Deployment Options

**OSS (Self-Hosted):**
```python
db = lancedb.connect("./data")  # Local
db = lancedb.connect("s3://bucket/path")  # S3
```

**LanceDB Cloud (4 regions):**
```python
# .env
# LANCEDB_URI=db://my-database
# LANCEDB_API_KEY=...

db = lancedb.connect(
    "db://my-database",
    api_key=os.environ["LANCEDB_API_KEY"],
    region="us-east-1",  # us-east-1, us-west-2, eu-west-1, ap-south-1
)
```

LanceDB Cloud features:
- **Auto-compaction** — runs every 5 minutes (no manual `compact()`)
- **Auto-reindexing** — re-creates the HNSW index on every 1k writes
- **Serverless** — no instance management
- **Multi-region** — pick the region closest to your data

**For self-hosted Cloudflare R2 + Lance** (the Cianfhoghlaim production target),
use the rclone-sidecar Compose pattern (see
`references/hosting-lancedb-docker-compose.md` or the deleted
`docs/lance/lancedb.compose.yaml` for the upstream example).

## Troubleshooting

### Slow Searches
- Create an index for datasets >100K
- Use pre-filtering for selective filters
- Check if compaction is needed

### Out of Memory
- Use disk-based indexes (IVF-PQ)
- Enable compression with Float16
- Query with projections

### Schema Mismatch
- Verify vector dimensions match
- Check data types in schema

## Resources

- **Documentation**: https://lancedb.github.io/lancedb/
- **GitHub**: https://github.com/lancedb/lancedb
- **Examples**: https://github.com/lancedb/vectordb-recipes
- **Blog**: https://blog.lancedb.com/

## 2026-06 updates (from the `upstream-package-monitoring` openspec change)

- **Lance Format v2.2** — 50%+ storage reduction vs Parquet; 68× faster
  blob reads. The Cianfhoghlaim migration to format v2.2 is tracked by
  `openspec/changes/lancedb-format-v22-migration/`.
- **Lance Blob V2** — 4 storage modes: Inline / Packed / Dedicated /
  External. Cianfhoghlaim recommends **Packed** for leabharlann assets (where
  blobs are <1 MB) and **Dedicated** for upstream blog payloads
  (where blobs can be >10 MB).
- **LanceDB embedder model note** — the value actually exported by
  `cocoindex/_lifespan.py:70` is
  `BAAI/bge-large-en-v1.5` (English-only, 1024-dim), NOT `BAAI/bge-m3`
  as some CocoIndex v1 App docstrings claim. Both are 1024-dim so the
  discrepancy is latent. Apps whose docstrings claim `bge-m3` will be
  corrected in a follow-up openspec change.
- **LanceDB upstream monitor** — `lancedb_blog.yml` in
  `infrastructure/firecrawl/monitors/upstream_packages/` is the
  Firecrawl monitor that detects Lance Format / Lance Blob / multimodal
  / Lance Namespace releases via the LLM-judge `--goal` filter.
  See the `cianfhoghlaim-cocoindex-v1` skill for the 14 v1 CocoIndex Apps
  that consume these updates.

## Examples

See [`./examples/`](./examples/) for upstream LanceDB reference
notebooks (8 total, ~17 MB). Highlights:

- `data_engineering_lance_Advance_RAG_LOTR_main.ipynb` — Lord of
  the Rings multi-document RAG baseline
- `data_engineering_lance_Advanced_RAG_Context_Enrichment_Window_*.ipynb`
  — context-enrichment window RAG pattern
- `data_engineering_lance_Chunking_Analysis_*.ipynb` — chunking
  strategy comparison
- `data_engineering_lance_cognee-RAG_cognee_demo.ipynb` — Cognee
  + LanceDB hybrid
- `data_engineering_lance_ColPali-vision-retriever_colpali.ipynb`
  — ColPali vision retriever on LanceDB
- `data_engineering_lance_Geospatial-Recommendation-System_*.ipynb`
  — geospatial point recommender
- `data_engineering_lance_multi-document-agentic-rag_main.ipynb`
  — multi-document agentic RAG
- `data_engineering_lance_multimodal-recipe-agent_*.ipynb` —
  multimodal (image + text) recipe agent

## British-Isles Education pipeline — Canonical Cianfhoghlaim pattern (post-v4)

The post-v4 lc6 pipeline (`openspec/changes/lc6-biep/`) uses
LanceDB as the **vector companion** to DuckLake. Each of the
7 v1 CocoIndex Apps (6 LC subjects + `government_circulars`)
mounts a LanceDB table via the canonical `mount_table_target`
pattern (the v1 conformance R4 contract). The 24+1 BIEP tables
all live under `cianfhoghlaim.lc.<subject>.<level>_<lang>` and
`cianfhoghlaim.education.ie.gov_circulars`:

```python
import lancedb
from lancedb.pydantic import LanceModel, Vector
from lancedb.embeddings import get_registry

# Shared Cianfhoghlaim embedder: BAAI/bge-m3 (1024-d, multilingual)
embedder = get_registry().get("sentence-transformers").create(
    name="BAAI/bge-m3",
    device="mps",  # M-series Mac native
)

class MathChunk(LanceModel):
    text: str = embedder.SourceField()
    vector: Vector(embedder.ndims()) = embedder.VectorField()
    subject: str
    level: str
    language: str
    module_id: str
    source_pdf: str

# Connect via the canonical Cianfhoghlaim pattern (local + S3-compatible Garage)
db = lancedb.connect("s3://garage/lance")

# 1 of the 24 BIEP per-subject tables (en/higher/Mathematics)
table = db.open_table("cianfhoghlaim.lc.mathematics.higher_en")

# Hybrid vector + BM25 search with RRF reranking (the canonical Cianfhoghlaim pattern)
results = (
    table.search(query_type="hybrid")
    .vector(embedder.embed("algebraic fractions").tolist())
    .text("algebraic fractions")
    .where("subject = 'mathematics' AND level = 'higher'")
    .rerank(method="rrf")
    .limit(10)
    .to_pandas()
)
```

**British-Isles Education pipeline use case:**

- **24+1 LanceDB companion tables** — one per
  `(subject, level, language)` partition, plus
  `cianfhoghlaim.education.ie.gov_circulars` for the `gov.ie`
  circulars. Each table holds 1024-d `BAAI/bge-m3` vectors
  over BAML-extracted chunks.
- **Bilingual RAG** — the `BAAI/bge-m3` embedder is
  multilingual, so a single query in English retrieves both
  English and Gaeilge chunks (ranked via the
  `BAML ExtractCrossLinguisticConcept` mapping).
- **`lance_scan()` from DuckDB** — the marimo notebooks
  join `lance_scan('s3://garage/lance/cianfhoghlaim.lc.<subject>.*')`
  against the DuckLake `curriculum_syllabus` tables in a single
  federated SQL query.
- **Time-travel for syllabus versions** — each BIEP table is
  versioned per academic year, so `table.checkout(version)`
  lets the notebooks compare the 2025 vs 2026 syllabus.
- **Reconciled embedder note** — the canonical BIEP embedder
  is `BAAI/bge-m3` (multilingual). The 2026-06 release-note
  claim that `_lifespan.py:70` exports `BAAI/bge-large-en-v1.5`
  is a latent discrepancy (both are 1024-d, so consumers see
  no error). The BIEP explicitly uses `bge-m3` for Gaeilge
  + English coverage; the `_lifespan.py` constant will be
  corrected to `bge-m3` in a follow-up openspec change.

Cross-references:
- [`.agents/skills/dlt/SKILL.md`](../dlt/SKILL.md) — the DLT
  sources that produce the BAML rows
- [`.agents/skills/cocoindex/SKILL.md`](../cocoindex/SKILL.md) —
  the canonical `mount_table_target` pattern (v1 conformance R4)
- [`.agents/skills/motherduck/SKILL.md`](../motherduck/SKILL.md) —
  the 4 Dives that query these tables
- [`.agents/skills/duckdb/SKILL.md`](../duckdb/SKILL.md) — the
  `lance_scan()` federated SQL layer
- [`.agents/skills/marimo/SKILL.md`](../marimo/SKILL.md) — the 6
  per-subject notebooks that join DuckLake + LanceDB
- [`.agents/skills/baml/SKILL.md`](../baml/SKILL.md) — the
  `ExtractCrossLinguisticConcept` cross-linguistic alignment


## Version policy

Per [`docs/SKILL-VERSION-HEADER-POLICY.md`](../../../../docs/SKILL-VERSION-HEADER-POLICY.md) (the canonical refresh policy):

- The canonical refresh cadence is weekly (Stage 7 GH workflow) + on-demand (per openspec change)
- The canonical command is `python3 scripts/audit/refresh_skill_versions.py`
- The canonical package→skill mapping is at [`scripts/audit/PACKAGE_TO_SKILL_MAP.md`](../../../../scripts/audit/PACKAGE_TO_SKILL_MAP.md)
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall