Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/dlt


name: dlt description: Master routing skill for data load tool (dlt 1.28.1, June 2026). Use this to understand dlt rules, decide which sub-skill to invoke, and apply the Cianfhoghlaim dlt conventions (DuckLake/DuckDB destination, USE_LOCAL_SCRAPES offline fallback, relative imports only, type-safe BAML-driven pipelines, multi-destination fan-out to LanceDB / Memgraph / Graphiti, and Dagster dlt_assets wrapping). Powers the British-Isles Education pipeline (6 LC subjects + gov.ie circulars). Notes the 1.27 dlt[hub] plugin split and the 1.28 refresh > replace deprecation.

DLT Master Router & Rules (Cianfhoghlaim)

You are operating within the cianfhoghlaim stack which uses dlt (data load tool) for extracting and loading data. This skill is the router + decision tree + project rules for all dlt operations.

1.1 Live version (verified 2026-06-29)

  • Latest: dlt 1.30.0 (released Sep 2026) on PyPI.
  • Python: requires-python = ">=3.10, <3.15" — Python 3.9 dropped in 1.28.1; Python 3.14 supported (experimental).
  • Source count: 8,000+ sources (was 5,000+ in Wave 1).
  • Yanked: 1.27.0 and 1.27.1 — data-loss bug ("incremental merge truncates destination table"). Pin dlt>=1.27.2,<1.28 if you must, or upgrade to 1.28.1.
  • CLI split (1.27.0): pip install dlt[hub] is now required for dlt dashboard, dlt pipeline ... show, dlt pipeline ... mcp. dlt ai is now dlthub ai.
  • refresh > replace (1.28.0): the replace write-disposition switch is deprecated; use the refresh parameter instead.
  • Lance destination (1.25.0) + Lance REST Namespace (1.27.0): a lance destination now exists alongside lancedb — use the former for local/S3/Az/GCS Lance files, the latter for LanceDB Cloud.
  • Native Polars (1.27.0): @dlt.resource can yield Polars DataFrame or LazyFrame directly (auto-routed through Arrow).
  • Databricks Zerobus (1.27.0): databricks_adapter(..., insert_api="zerobus").
  • dlt.Relation.join(...) (1.26.0) and dlt.current.interval() (1.26.0) for relational composition and time-windowed incrementals.

1. Project rules (PRESERVED from the original skill, with one fix)

When assuming the data-engineer persona, use these rules:

  • Destinations: dlt.pipeline(..., destination="ducklake") for production (MotherDuck / DuckLake) or destination="duckdb" for local dev. Set USE_DUCKLAKE=true to switch to MotherDuck.
  • Tests: disable plugins during testing by setting DLT_DISABLE_PLUGINS=true.
  • Canonical env vars (per the 2026-08-01-lakehouse-and-reproducible-deploy-v1 openspec change — now exported by all 5 data-plane stacks secrets.env):
    • USE_DUCKLAKE (default true) — switch to MotherDuck
    • USE_LOCAL_SCRAPES (default true) — offline fallback
    • MOTHERDUCK_TOKEN (resolved from infisical://dev-baile/motherduck/token)
    • MOTHERDUCK_MODE (default byob)
    • MOTHERDUCK_DATABASE (default cianfhoghlaim)
    • MOTHERDUCK_S3_BUCKET (default ducklake-cianfhoghlaim)
    • MOTHERDUCK_S3_ENDPOINT (default http://lakehouse-garage:3900)
    • DUCKLAKE_BUCKET (default ducklake-cianfhoghlaim)
    • CIANFHOGHLAIM_EMBED_MODEL (default BAAI/bge-m3)
    • CIANFHOGHLAIM_EMBED_DIM (default 1024)
  • Source location: dlt_sources lives at dlt_sources/ (NOT data_platform/dlt_sources/, which is a deprecated path mentioned in the old skill; NOT dlt/, which was the pre-v4 path).
  • Imports: All cianfhoghlaim.data_platform... absolute imports have been removed; use relative or local dlt_sources imports (e.g. from cianfhoghlaim.dlt.british_isles.ireland.education.ncca...).
  • Offline fallback: USE_LOCAL_SCRAPES=true routes extraction to stedding/ingest_queue/ (the curated local cache) instead of live web scraping (avoids API rate limits and credit drain).
  • Absolute namespaces (per project AGENTS.md): NEVER import cianfhoghlaim.data_platform... from within the data platform — use relative imports.
  • Ingestion cache (per project AGENTS.md): Test with USE_LOCAL_SCRAPES=true before live web scraping to avoid API rate limits.

2. Decision tree → sub-skill or reference

When tasked with dlt operations or data exploration, use this guide to invoke the most appropriate resource:

Data exploration & notebooks (notebooks/)

  • explore-data: Use to analyze datasets and create an analysis_plan.md artifact
  • build-notebook: Use to assemble or regenerate a marimo notebook from an analysis_plan.md

Pipeline creation & maintenance

  • create-filesystem-pipeline: Use to build pipelines that read from local files. Highly relevant for the USE_LOCAL_SCRAPES offline fallback pattern
  • add-incremental-loading: Use to add state and incremental extraction to a filesystem pipeline
  • create-rest-api-pipeline: Use for generic REST / HTTP API sources
  • dlt-init-openapi (3rd-party): Use to auto-generate a verified dlt source from any OpenAPI spec
  • dlt init <verified-source> (1.28+ recommended): For any of the 28 verified sources listed at https://dlthub.com/docs/dlt-ecosystem/verified-sources (Airtable, GitHub, Stripe, Notion, Postgres replication, MongoDB, Salesforce, HubSpot, Kafka, Slack, etc.). Verified sources are downloaded into the working directory.

Type-safe pipelines (BAML → dlt)

  • baml-dlt-integration — see references/type-safe-pipeline.md for the canonical BAML → Pydantic → columns=... pattern

Destinations

  • LanceDB — see references/destinations-lancedb.md (the lancedb_adapter(source, embed=[...]) pattern)
  • Cognee + Memgraph — see references/destinations-cognee-memgraph.md (knowledge-graph destination)
  • Graphiti — see references/destinations-graphiti.md (temporal knowledge graph)

Performance & optimisation

  • Parallelised resources, add_limit, file rotation — see references/performance-optimisation.md

Dagster integration

  • @dlt_assets wrapping — see references/dagster-dlt-assets.md (the canonical pattern for scheduling DLT inside Dagster)

Transformations (dlt → dlt)

  • @dlt.transformer, SQL-based, Ibis aggregation — see references/dlt-transformations.md

Deployment

  • Dagster asset with scheduling — see the dagster skill
  • Serverless webhook (HTTP-triggered) — see references/deploy-gcp-cloud-function-webhook.md
  • Serverless scheduled — see references/deploy-modal.md
  • DAG-transformed downstream — hand off to the sqlmesh skill via sqlmesh init -t dlt --dlt-pipeline <name> dialect (see references/sqlmesh-init.md)

Search & RAG

  • Vectorise for RAG — see references/destinations-lancedb.md (the lancedb_adapter(source, embed=[...]) pattern)
  • Knowledge graph — see references/destinations-cognee-memgraph.md

OpenAPI source generation

  • dlt-init-openapi — see references/openapi-generator.md for auto-generated verified sources from any OpenAPI spec

Browser scraping

  • crawl4ai — see references/crawl4ai-dlt-summary.md (alternative to Firecrawl for JS-heavy sites)

3. Project-specific recipes (Cianfhoghlaim patterns)

Type-safe pipeline (BAML → dlt → oRPC → MCP)

import dlt
from cianfhoghlaim.baml import b  # post-v4 codegen path
from cianfhoghlaim.baml.types import PrimaryLearningOutcome  # auto-generated
from pydantic import BaseModel

class PrimaryOutcomeRow(BaseModel):
    """Mirror of the BAML class, with dlt column types."""
    stage: str
    curriculum_area: str
    learning_outcome: str

@dlt.resource(name="primary_outcomes", write_disposition="merge", primary_key=["stage", "curriculum_area", "learning_outcome"])
def primary_outcomes(pdf_path: str) -> list[PrimaryOutcomeRow]:
    """Extract primary learning outcomes from an NCCA PDF via BAML."""
    text = extract_pdf_text(pdf_path)
    outcomes = b.ExtractPrimaryLearningOutcomes(text)
    for o in outcomes:
        yield PrimaryOutcomeRow(
            stage=o.stage,
            curriculum_area=o.curriculum_area,
            learning_outcome=o.learning_outcome,
        )

pipeline = dlt.pipeline(
    pipeline_name="ireland_primary_curriculum",
    destination="ducklake",
    dataset_name="cianfhoghlaim.education.ie",
)
load_info = pipeline.run(primary_outcomes("ncca_primary.pdf"))
print(load_info)

The BAML class is the single source of truth — both the Pydantic BaseModel and the dlt primary_key derive from it.

Canonical Cianfhoghlaim pattern — British-Isles Education pipeline (lc6)

The post-v4 canonical pipeline ingests one of the six Irish Leaving Certificate subjects (Mathematics, Chemistry, Geography, Gaeilge, English, Computer Science) from a BAML-extracted NCCA/SEC PDF, fan-outs to DuckLake + LanceDB, and is scheduled by Dagster via @dlt_assets:

import dlt
from cianfhoghlaim.dlt.british_isles.ireland.education.ncca import (
    mathematics_syllabus as mathematics_syllabus_source,
)
from cianfhoghlaim.baml.education.lc_extraction.curriculum_syllabus import b


@dlt.resource(
    name="lc_mathematics_syllabus",
    write_disposition="merge",
    primary_key=["subject", "level", "language", "module_id"],
)
def mathematics_syllabus(pdf_path: str):
    """Extract Mathematics LC syllabus modules from an NCCA PDF via BAML."""
    doc = b.ExtractCurriculumSyllabus(text=open(pdf_path).read())
    for module in doc.modules:
        yield {
            "subject": "mathematics",
            "level": doc.level,
            "language": doc.language,
            "module_id": module.id,
            "title": module.title,
            "hours": module.hours,
            "learning_outcomes": module.learning_outcomes,
        }


pipeline = dlt.pipeline(
    pipeline_name="lc6_ncca",
    destination="ducklake",
    dataset_name="cianfhoghlaim.leaving_cert",
)
load_info = pipeline.run(
    mathematics_syllabus("leaving_certificate/mathematics/en/...syllabus.pdf")
)
print(load_info)

The same pattern is repeated for the other 5 LC subjects (chemistry_syllabus, geography_syllabus, gaeilge_syllabus, english_syllabus, computer_science_syllabus) plus a government_circulars resource for gov.ie education circulars. Each resource is wrapped in @dlt_assets in orchestration/defs/2_materials/ and contributes to the 7 v1 CocoIndex Apps (6 LC + government_circulars) and the 4 MotherDuck Dives (lc_syllabus_topics, lc_exam_difficulty, lc_marking_complexity, gov_circulars_archive).

Multi-destination fan-out (DuckDB + LanceDB + Memgraph)

import dlt
from dlt.destinations import duckdb
from lancedb import lancedb_adapter
from cognee import add as cognee_add, cognify

@dlt.resource(name="curriculum_chunks")
def chunks(pdf_path: str):
    text = extract_pdf_text(pdf_path)
    for chunk in chunk_text(text):
        yield {"text": chunk, "source": pdf_path}

# Fan out to 3 destinations
pipeline = dlt.pipeline(destination="duckdb", dataset_name="curriculum")
load_info = pipeline.run([
    chunks("ncca.pdf"),                                    # → DuckDB
    lancedb_adapter(chunks("ncca.pdf"), embed=["text"]),    # → LanceDB
    cognee_destination(chunks("ncaa.pdf")),                 # → Cognee
])

One pipeline run, three destinations. State is unified (one pipeline.last_trace, not three).

Dagster asset wrapping (@dlt_assets)

from dagster_dlt import DagsterDltResource, dlt_assets
import dlt

@dlt_assets(
    dlt_source=ireland_curriculum_source(),
    dlt_pipeline=dlt.pipeline(
        pipeline_name="ireland_curriculum",
        destination="ducklake",
        dataset_name="cianfhoghlaim.education.ie",
    ),
)
def ireland_curriculum_assets(context, dlt_run_resource: DagsterDltResource):
    yield from dlt_run_resource.run(context=context)

# Schedule
@schedule(cron_schedule="0 2 * * *", job=ireland_curriculum_assets_job)  # 02:00 UTC daily
def ireland_curriculum_schedule(): ...

See references/dagster-dlt-assets.md for the full pattern with multiprocess_executor, parallel assets, and incremental loading.

4. Performance & anti-patterns

✅ Do:

  • Use parallelized=True for any resource fetching > 1k rows
  • Use add_limit(N) to cap a source for testing
  • Use file rotation (dlt.pipeline(..., progress="log") + chunked writes) for > 1M rows
  • Use write_disposition="merge" with an explicit primary_key for upserts
  • Use columns=PydanticModel for type-safe pipelines (the BAML pattern)
  • Pre-validate inputs before the API call (catches bad PDFs early)

❌ Don't:

  • Fetch all data in a single fetch_all() call (OOM risk for > 1M rows)
  • Use write_disposition="merge" without a primary_key (silently appends duplicates)
  • Import cianfhoghlaim.data_platform.dlt_sources from within cianfhoghlaim/ (use relative imports (from .dlt...)
  • Hand-write DDL for the destination (let dlt infer the schema from the resource yield)
  • Run live web scraping without USE_LOCAL_SCRAPES=true first (drains API credits and risks rate limits)
  • Add a BAML client inline in a function (use a named client in baml/clients.baml)
  • Pin dlt==1.27.0 or dlt==1.27.1 — both YANKED from PyPI for a data-loss bug; the fix is 1.27.2 (or upgrade to ≥ 1.28.1).
  • Use write_disposition="replace" (deprecated in 1.28.0) — use the refresh parameter instead.
  • Call dlt ai ... (moved to dlthub ai ... in 1.27.0).
  • Call dlt dashboard without pip install dlt[hub] (1.27.0 split).

5. Reference index

The 3 reference files in references/ that were previously orphaned are now linked from here:

New references (added by the sync-skills-from-docs change):

6. Cross-references

  • The dlt skill is consumed by: data-engineer agent (the primary user)
  • The dlt skill collaborates with: baml skill (type-safe pipelines), dagster skill (@dlt_assets wrapping), sqlmesh skill (sqlmesh init -t dlt), lancedb skill (vector destination), cognee skill (knowledge-graph destination), motherduck skill (MotherDuck destination)
  • The dlt skill feeds into: explore-data (analysis_plan.md) and build-notebook (marimo notebooks) for downstream visualisation

7. Examples

See ./examples/ for upstream dlt reference

British-Isles Education pipeline use case (BIEP, post-v4)

The dlt skill is the data-load backbone for the British-Isles Education pipeline (openspec/changes/lc6-biep/). It drives:

  • 7 per-subject BAML extraction stages (syllabus / exam paper layout / marking scheme / cross-linguistic / syllabus diagram) across the 6 LC subjects (Mathematics, Chemistry, Geography, Gaeilge, English, Computer Science) — 42 lc5/lc6 Dagster assets total in orchestration/defs/2_materials/.
  • gov.ie circulars — the government_circulars resource mirrors the 7th v1 CocoIndex App (government_circulars), landing pages from gov.ie/.../circulars/... into cianfhoghlaim.education.ie.gov_circulars_archive (one of the 4 canonical MotherDuck Dives).
  • Dual-language fan-out — every resource is partitioned by language (en / ga) so the Gaeilge syllabus runs in parallel with English via the same @dlt_assets wrapper.
  • BAML → dlt columns — the lc5/lc6 extraction schemas (e.g. ExtractCurriculumSyllabus, ExtractMarkingSchemeGuideline) are the primary_key source-of-truth for the merge write disposition.

Cross-references:

2026-06 updates (from the upstream-package-monitoring openspec change)

  • dltHub Pro launched 2026-04-14. The Pro tier adds 9,700+ known source contexts that DLT can pull from in one call. The Cianfhoghlaim dev plan is tracked by openspec/changes/dlt-pro-source-registry/.

  • Cortex Code (Snowflake's AI assistant, launched ~9 weeks before dltHub Pro) integrates directly with the dlt Pro source registry.

  • ADE-Bench (the AI data-engineer benchmark) reported 65% task success on Snowflake via Cortex Code vs 58% for Claude Code. The paper's key finding: "without the workbench, the agent leaked credentials" — directly validates Cianfhoghlaim's strict-secret-hydration mandate (see docs/secrets/secrets_management_plan.md for the Infisical + Locket + mise three-way contract).

  • dlthub upstream monitor — dlthub_blog.yml in bonneagar/firecrawl/monitors/upstream_packages/ is the Firecrawl monitor that detects new source-context additions, ADE-Bench results, and Cortex Code integration updates via the LLM-judge --goal filter. The n8n workflow bonneagar/stacks/n8n/workflows/upstream-blog-monitor.json writes the payload to s3://cianfhoghlaim-upstream-webhooks/dlthub/...jsonl and triggers the Dagster asset upstream_blog_monitor_ingest. notebooks:

  • data_engineering_dlt_small-data-sf-2025_elvis.ipynb — Dremio "small data" workshop (SF 2025): dlt pipelines for small files, REST API ingestion patterns, and DuckDB destination examples.

v7 flattening migration notes (added 2026-07-19)

Per openspec/changes/2026-07-14-fix-foundation-v7-flattening-and-baml-drift-v1:

  • DLT sources are now at cianfhoghlaim.dlt.british_isles.<jurisdiction>.education.<source>
  • The canonical CLI entry point is python -m cianfhoghlaim.dlt.cli run-pipeline (NOT python -m cianfhoghlaim.dlt.run_pipeline which was the pre-v7 name)
  • The new BIEP v3 jurisdiction pipelines live at:
    • dlt_sources/british_isles/ireland/education/ireland_jurisdiction_pipeline.py
    • dlt_sources/british_isles/england/education/england_jurisdiction_pipeline.py
    • dlt_sources/british_isles/sct_wls_ni/education/sct_wls_ni_jurisdiction_pipeline.py
    • dlt_sources/british_isles/crown_dependencies/education/crown_dependencies_jurisdiction_pipeline.py
  • Each pipeline uses dlt.common.curriculum_registry.SubjectRegistry for canonical subject metadata
  • The destination is configured via dlt.common.destinations_cianfhoghlaim.get_dlt_destination(use_ducklake=True)
  • The 4 destinations: local DuckDB, MotherDuck BYOB, Garage S3 + Lakekeeper, R2 + Lakekeeper
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 197kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall