Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/cocoindex

cocoindex

Verified

references/multimodal-image-search.md

View on GitHub
# Multimodal Image Search in CocoIndex v1

The canonical pattern for image + text search using CLIP-style
embeddings. Two flavours: single-vector (CLIP) and multi-vector
(ColPali).

## Single-vector (CLIP + Qdrant)

```python
import cocoindex as coco
from cocoindex.connectors import localfs, qdrant
from cocoindex.resources.file import FileLike, PatternFilePathMatcher

# Assumes you've wrapped CLIP in a ContextKey + SentenceTransformerEmbedder
# (or a custom @coco.fn that calls CLIP).

@coco.fn(memo=True)
async def process_image(file: FileLike, target) -> None:
    image_bytes = await file.read_bytes()
    # Embed via CLIP
    embedding = await clip_embed(image_bytes)
    target.declare_row(row=ImageRecord(
        id=hash(file.file_path.path) & 0x7fffffff,
        filename=str(file.file_path.path),
        embedding=embedding,
    ))
```

## Multi-vector (ColPali + Qdrant MaxSim)

ColPali produces a multi-vector representation (one vector per
image patch), which is better for document / page retrieval. CocoIndex
v1 supports this via the `MultiVector` schema in Qdrant:

```python
# In a @dataclass row:
@dataclass
class DocPage:
    id: int
    image_id: str
    patch_vectors: list[list[float]]  # N patches × D dims
```

Qdrant's MaxSim scoring picks the best patch match per document.

## Cianfhoghlaim examples

- `cocoindex/leabharlann_embedding.py` — text-only
  (BGE-large-en-v1.5) for the leabharlann books corpus
- The external `docs/cocoindex/image_search/` example (now in
  upstream cocoindex repo) is the canonical CLIP + Qdrant + FastAPI
  pattern
- The external `docs/cocoindex/image_search_colpali/` example is
  the canonical ColPali + Qdrant MaxSim pattern
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall