Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/dlt

references/performance-optimisation.md

View on GitHub
# Performance Optimisation

For dlt pipelines that handle > 100k rows, performance matters.
This reference covers the 4 main optimisation levers.

## 1. Parallelised resources

By default, dlt resources are sequential. To parallelise:

```python
@dlt.resource(parallelized=True)
def chunks(pdf_paths: list[str]):
    for path in pdf_paths:
        text = extract_pdf_text(path)
        for chunk in chunk_text(text):
            yield {"text": chunk, "source": path}
```

The `parallelized=True` flag tells dlt to use multiple workers
(via `concurrent.futures.ThreadPoolExecutor` or
`ProcessPoolExecutor`, depending on the resource).

## 2. `add_limit(N)` for testing

Cap a resource for testing without modifying the source:

```python
load_info = pipeline.run(
    add_limit(curriculum(pdf_paths), 100),  # only 100 rows
)
```

Useful for development (skip the 10-minute load, test the schema
instead).

## 3. File rotation

For > 1M rows, dlt buffers to memory and writes in chunks. The
default chunk size is good for most cases, but for very large
loads, tune:

```python
@dlt.resource(
    file_max_items=100_000,  # rotate to a new file every 100k rows
    workers=4,  # number of parallel workers
)
def chunks(pdf_paths: list[str]):
    ...
```

## 4. Multi-pipeline-in-one-process

For multi-destination fan-out, run a single pipeline with multiple
destinations. State is unified:

```python
pipeline = dlt.pipeline(
    pipeline_name="curriculum",
    destination="duckdb",
    dataset_name="curriculum",
    progress="log",  # log progress instead of TUI
)
load_info = pipeline.run([
    curriculum_chunks("ncca.pdf"),
    lancedb_adapter(curriculum_chunks("ncca.pdf"), embed=["text"]),
    cognee_destination(curriculum_chunks("ncca.pdf")),
])
```

One `pipeline.run()`, three destinations, one `pipeline.last_trace`.

## Async generators

For I/O-bound sources, use `async` resources:

```python
@dlt.resource(parallelized=True)
async def chunks(pdf_paths: list[str]):
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_chunks(session, path) for path in pdf_paths]
        for chunks in asyncio.as_completed(tasks):
            for chunk in await chunks:
                yield chunk
```

## 5. `apply_hints` for incremental loading

For append-only sources that grow over time, use incremental
loading to skip already-seen rows:

```python
@dlt.resource(
    primary_key="id",
    write_disposition="merge",
)
def events():
    for event in fetch_events():
        yield event
```

dlt tracks the highest seen `id` and only inserts new ones. See the
`add-incremental-loading` sub-skill for the full pattern.

## Performance checklist

- [ ] `parallelized=True` for any source > 1k rows
- [ ] `add_limit(N)` for any test run
- [ ] `file_max_items` tuned for > 1M row loads
- [ ] `write_disposition="merge"` with explicit `primary_key` for
  upserts (not append)
- [ ] Multi-destination fan-out via single `pipeline.run(...)`
- [ ] `apply_hints` for incremental loading on growing sources
- [ ] `dlt pipeline ... trace` to inspect state and timing

## Reference

- The `dlt_optimisation.py` reference (900+ lines) was in `docs/dlt/`
  (deleted with the `sync-skills-from-docs` change). The same content
  is in the upstream dltHub docs under
  [performance](https://dlthub.com/docs/performance)
- The `add-incremental-loading` sub-skill
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall