Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/dlt

references/dlt-transformations.md

View on GitHub
# dlt Transformations (chained dlt → dlt)

`dlt.transformer` is the canonical pattern for **chaining dlt
sources together** within a single pipeline. The second source
consumes rows from the first, applies a transformation, and
yields the result.

## Basic pattern

```python
@dlt.resource(name="chunks")
def chunks(pdf_paths: list[str]):
    for path in pdf_paths:
        text = extract_pdf_text(path)
        for i, chunk in enumerate(chunk_text(text)):
            yield {"text": chunk, "source": path, "chunk_id": i}

@dlt.transformer(data_from=chunks)
def embed_chunks(chunk: dict):
    """Consume one row from `chunks` and yield the embedded version."""
    yield {
        **chunk,
        "embedding": embed(chunk["text"]),  # list[float]
    }

# Run: chunks → embed_chunks → destination
pipeline = dlt.pipeline(destination="duckdb", dataset_name="curriculum")
load_info = pipeline.run(embed_chunks(pdf_paths))
```

## SQL-based transformations

For SQL-on-data transformations (e.g. aggregations, joins), use
`dlt.transformer` with a SQL query:

```python
@dlt.transformer(data_from=chunks)
def chunk_stats(chunk: dict):
    """Compute word count per chunk."""
    yield {
        "chunk_id": chunk["chunk_id"],
        "word_count": len(chunk["text"].split()),
    }
```

Or use the `dlt` SQL integration directly:

```python
pipeline = dlt.pipeline(destination="duckdb", dataset_name="curriculum")
pipeline.run([
    chunks(["ncca.pdf"]),       # raw chunks
    "SELECT source, COUNT(*) AS chunk_count FROM chunks GROUP BY source",  # SQL aggregation
])
```

## Ibis aggregation

For complex analytics, use Ibis (a portable dataframe API) on top
of dlt:

```python
import ibis

pipeline = dlt.pipeline(destination="duckdb", dataset_name="curriculum")
con = pipeline.dataset().ibis()  # Ibis connection to the dlt-managed DuckDB

# Now run Ibis expressions
chunks_table = con.table("chunks")
stats = (
    chunks_table
    .group_by("source")
    .aggregate(chunk_count=chunks_table.count(), avg_length=chunks_table.text.length().mean())
    .execute()
)
```

## Multi-yield from a single resource

A `dlt.resource` can yield multiple types of rows:

```python
@dlt.resource(name="document_analysis", primary_key="id")
def document_analysis(pdf_paths: list[str]):
    for path in pdf_paths:
        text = extract_pdf_text(path)
        # Yield the raw text
        yield {"id": path, "type": "raw_text", "text": text}
        # Yield the chunks
        for i, chunk in enumerate(chunk_text(text)):
            yield {"id": f"{path}:{i}", "type": "chunk", "text": chunk}
        # Yield the entities
        entities = b.ExtractEntities(text)
        for e in entities:
            yield {"id": f"{path}:{e.name}", "type": "entity", "name": e.name, "kind": e.type}
```

The destination has 3 tables: `raw_text`, `chunk`, `entity`.

## Cianfhoghlaim usage

The Cianfhoghlaim stack uses `dlt.transformer` in:

- `dlt/british_isles/ireland/` — chain raw pages → parsed
  curriculum → BAML-extracted outcomes
- `dlt/leabharlann/` — chain Takeout files
  → extracted text → BAML entities

## Reference

- The `Transformations _ dlt Docs.md` reference (25K) was in
  `docs/dlt/` (deleted with the `sync-skills-from-docs` change).
  The same content is in the dltHub docs at
  <https://dlthub.com/docs/dlt-ecosystem/transformations>
- The `ibis` skill for Ibis patterns
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall