Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/lancedb

lancedb

Verified

references/multimodal-fat-tables.md

View on GitHub
# Multimodal "Fat Table" Pattern

LanceDB supports storing BLOBs (images, audio, PDFs) alongside
vectors and metadata in the same row — the "fat table" pattern. This
is the recommended approach for small-to-medium BLOBs (< 1 MB).

## Schema

```python
import pyarrow as pa

schema = pa.schema([
    pa.field("id", pa.int64()),
    pa.field("filename", pa.string()),
    # BLOB: image / audio / PDF bytes
    pa.field("image_blob", pa.large_list(pa.uint8())),
    # Vector: CLIP / OpenCLIP embedding
    pa.field("image_embedding", pa.list_(pa.float32(), 768)),
    # Metadata
    pa.field("description", pa.string()),
    pa.field("tags", pa.list_(pa.string())),
    # Optional: pre-computed text embedding for hybrid search
    pa.field("text_embedding", pa.list_(pa.float32(), 1024)),
])

table = db.create_table("multimodal", schema=schema)
```

## Insert

```python
import clip
import io
from PIL import Image

def embed_image(image_bytes: bytes) -> list[float]:
    image = Image.open(io.BytesIO(image_bytes))
    return clip.encode(image).tolist()

# Single insert
table.add([{
    "id": 1,
    "filename": "recipe.png",
    "image_blob": open("recipe.png", "rb").read(),
    "image_embedding": embed_image(open("recipe.png", "rb").read()),
    "description": "Beef stew with root vegetables",
    "tags": ["main", "irish"],
}])

# Batch insert
rows = []
for path in image_paths:
    with open(path, "rb") as f:
        data = f.read()
    rows.append({
        "id": hash(path) & 0x7fffffff,
        "filename": path,
        "image_blob": data,
        "image_embedding": embed_image(data),
        "description": caption(path),  # from a vision LLM
    })
table.add(rows)
```

## Range-read on BLOB

LanceDB supports range-reads on `large_list` columns, so you can
fetch only the BLOB for the top-K results:

```python
results = table.search(query_vec).limit(10).to_pandas()
for r in results.itertuples():
    # The image_blob is already a list of bytes (pyarrow large_list)
    image = Image.open(io.BytesIO(bytes(r.image_blob)))
    image.show()
```

If you only need the embedding + metadata (not the BLOB), use
`select(["id", "filename", "image_embedding", "description"])`:

```python
results = (table.search(query_vec)
          .select(["id", "filename", "description", "image_embedding"])
          .limit(10)
          .to_pandas())
```

## Fat table vs pointer strategy

| | Fat table | Pointer strategy |
|:--|:--|:--|
| **BLOB size** | < 1 MB recommended | Any size |
| **Storage** | Lance (one copy) | S3/R2 (BLOB) + Lance (URL) |
| **Read pattern** | Single query | Two queries (metadata + BLOB fetch) |
| **Best for** | Small images, audio clips, page previews | Full videos, large PDFs, raw datasets |
| **Cons** | Slower for very large BLOBs | Extra round-trip for BLOB fetch |

**Rule of thumb**: if the BLOB is < 1 MB, use the fat table. If
> 1 MB, use the pointer strategy.

## Hybrid multimodal search

```python
# Vector search on the image embedding
results = (table.search(query_vec)
          .select(["id", "filename", "image_embedding", "text_embedding", "description"])
          .limit(20)
          .to_pandas())

# Re-rank with the text embedding (hybrid)
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
scores = reranker.predict([(query, r.description) for r in results.itertuples()])
results["rerank_score"] = scores
results = results.sort_values("rerank_score", ascending=False).head(10)
```

## Cianfhoghlaim example

The `cocoindex/leabharlann_embedding.py` CocoIndex
v1 App embeds the books corpus as text + image (BGE-large-en-v1.5
for text; the multimodal pattern is delegated to a separate
`cocoindex/image_embedding.py` flow if/when
the leabharlann includes cover images).

The external `multimodal-recipe-agent` and `multimodal-search`
examples in the upstream lancedb/vectordb-recipes repo (deleted
with the docs) are the canonical multimodal fat-table patterns.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall