cianfhoghlaim/cianchosaint / .agents/skills/dlt
dlt
Verifiedname: dlt
description: Master routing skill for data load tool (dlt 1.28.1, June 2026). Use this to understand dlt rules, decide which sub-skill to invoke, and apply the Cianfhoghlaim dlt conventions (DuckLake/DuckDB destination, USE_LOCAL_SCRAPES offline fallback, relative imports only, type-safe BAML-driven pipelines, multi-destination fan-out to LanceDB / Memgraph / Graphiti, and Dagster dlt_assets wrapping). Powers the British-Isles Education pipeline (6 LC subjects + gov.ie circulars). Notes the 1.27 dlt[hub] plugin split and the 1.28 refresh > replace deprecation.
DLT Master Router & Rules (Cianfhoghlaim)
You are operating within the cianfhoghlaim stack which uses dlt
(data load tool) for extracting and loading data. This skill is the
router + decision tree + project rules for all dlt operations.
1.1 Live version (verified 2026-06-29)
- Latest:
dlt 1.30.0(released Sep 2026) on PyPI. - Python:
requires-python = ">=3.10, <3.15"— Python 3.9 dropped in 1.28.1; Python 3.14 supported (experimental). - Source count: 8,000+ sources (was 5,000+ in Wave 1).
- Yanked: 1.27.0 and 1.27.1 — data-loss bug ("incremental merge
truncates destination table"). Pin
dlt>=1.27.2,<1.28if you must, or upgrade to 1.28.1. - CLI split (1.27.0):
pip install dlt[hub]is now required fordlt dashboard,dlt pipeline ... show,dlt pipeline ... mcp.dlt aiis nowdlthub ai. refresh>replace(1.28.0): thereplacewrite-disposition switch is deprecated; use therefreshparameter instead.- Lance destination (1.25.0) + Lance REST Namespace (1.27.0): a
lancedestination now exists alongsidelancedb— use the former for local/S3/Az/GCS Lance files, the latter for LanceDB Cloud. - Native Polars (1.27.0):
@dlt.resourcecan yield Polars DataFrame or LazyFrame directly (auto-routed through Arrow). - Databricks Zerobus (1.27.0):
databricks_adapter(..., insert_api="zerobus"). dlt.Relation.join(...)(1.26.0) anddlt.current.interval()(1.26.0) for relational composition and time-windowed incrementals.
1. Project rules (PRESERVED from the original skill, with one fix)
When assuming the data-engineer persona, use these rules:
- Destinations:
dlt.pipeline(..., destination="ducklake")for production (MotherDuck / DuckLake) ordestination="duckdb"for local dev. SetUSE_DUCKLAKE=trueto switch to MotherDuck. - Tests: disable plugins during testing by setting
DLT_DISABLE_PLUGINS=true. - Canonical env vars (per the 2026-08-01-lakehouse-and-reproducible-deploy-v1
openspec change — now exported by all 5 data-plane stacks
secrets.env):USE_DUCKLAKE(defaulttrue) — switch to MotherDuckUSE_LOCAL_SCRAPES(defaulttrue) — offline fallbackMOTHERDUCK_TOKEN(resolved frominfisical://dev-baile/motherduck/token)MOTHERDUCK_MODE(defaultbyob)MOTHERDUCK_DATABASE(defaultcianfhoghlaim)MOTHERDUCK_S3_BUCKET(defaultducklake-cianfhoghlaim)MOTHERDUCK_S3_ENDPOINT(defaulthttp://lakehouse-garage:3900)DUCKLAKE_BUCKET(defaultducklake-cianfhoghlaim)CIANFHOGHLAIM_EMBED_MODEL(defaultBAAI/bge-m3)CIANFHOGHLAIM_EMBED_DIM(default1024)
- Source location:
dlt_sourceslives atdlt_sources/(NOTdata_platform/dlt_sources/, which is a deprecated path mentioned in the old skill; NOTdlt/, which was the pre-v4 path). - Imports: All
cianfhoghlaim.data_platform...absolute imports have been removed; use relative or localdlt_sourcesimports (e.g.from cianfhoghlaim.dlt.british_isles.ireland.education.ncca...). - Offline fallback:
USE_LOCAL_SCRAPES=trueroutes extraction tostedding/ingest_queue/(the curated local cache) instead of live web scraping (avoids API rate limits and credit drain). - Absolute namespaces (per project AGENTS.md): NEVER import
cianfhoghlaim.data_platform...from within the data platform — use relative imports. - Ingestion cache (per project AGENTS.md): Test with
USE_LOCAL_SCRAPES=truebefore live web scraping to avoid API rate limits.
2. Decision tree → sub-skill or reference
When tasked with dlt operations or data exploration, use this guide to invoke the most appropriate resource:
Data exploration & notebooks (notebooks/)
explore-data: Use to analyze datasets and create ananalysis_plan.mdartifactbuild-notebook: Use to assemble or regenerate a marimo notebook from ananalysis_plan.md
Pipeline creation & maintenance
create-filesystem-pipeline: Use to build pipelines that read from local files. Highly relevant for theUSE_LOCAL_SCRAPESoffline fallback patternadd-incremental-loading: Use to add state and incremental extraction to a filesystem pipelinecreate-rest-api-pipeline: Use for generic REST / HTTP API sourcesdlt-init-openapi(3rd-party): Use to auto-generate a verified dlt source from any OpenAPI specdlt init <verified-source>(1.28+ recommended): For any of the 28 verified sources listed athttps://dlthub.com/docs/dlt-ecosystem/verified-sources(Airtable, GitHub, Stripe, Notion, Postgres replication, MongoDB, Salesforce, HubSpot, Kafka, Slack, etc.). Verified sources are downloaded into the working directory.
Type-safe pipelines (BAML → dlt)
baml-dlt-integration— seereferences/type-safe-pipeline.mdfor the canonical BAML → Pydantic →columns=...pattern
Destinations
- LanceDB — see
references/destinations-lancedb.md(thelancedb_adapter(source, embed=[...])pattern) - Cognee + Memgraph — see
references/destinations-cognee-memgraph.md(knowledge-graph destination) - Graphiti — see
references/destinations-graphiti.md(temporal knowledge graph)
Performance & optimisation
- Parallelised resources,
add_limit, file rotation — seereferences/performance-optimisation.md
Dagster integration
@dlt_assetswrapping — seereferences/dagster-dlt-assets.md(the canonical pattern for scheduling DLT inside Dagster)
Transformations (dlt → dlt)
@dlt.transformer, SQL-based, Ibis aggregation — seereferences/dlt-transformations.md
Deployment
- Dagster asset with scheduling — see the
dagsterskill - Serverless webhook (HTTP-triggered) — see
references/deploy-gcp-cloud-function-webhook.md - Serverless scheduled — see
references/deploy-modal.md - DAG-transformed downstream — hand off to the
sqlmeshskill viasqlmesh init -t dlt --dlt-pipeline <name> dialect(seereferences/sqlmesh-init.md)
Search & RAG
- Vectorise for RAG — see
references/destinations-lancedb.md(thelancedb_adapter(source, embed=[...])pattern) - Knowledge graph — see
references/destinations-cognee-memgraph.md
OpenAPI source generation
dlt-init-openapi— seereferences/openapi-generator.mdfor auto-generated verified sources from any OpenAPI spec
Browser scraping
crawl4ai— seereferences/crawl4ai-dlt-summary.md(alternative to Firecrawl for JS-heavy sites)
3. Project-specific recipes (Cianfhoghlaim patterns)
Type-safe pipeline (BAML → dlt → oRPC → MCP)
import dlt
from cianfhoghlaim.baml import b # post-v4 codegen path
from cianfhoghlaim.baml.types import PrimaryLearningOutcome # auto-generated
from pydantic import BaseModel
class PrimaryOutcomeRow(BaseModel):
"""Mirror of the BAML class, with dlt column types."""
stage: str
curriculum_area: str
learning_outcome: str
@dlt.resource(name="primary_outcomes", write_disposition="merge", primary_key=["stage", "curriculum_area", "learning_outcome"])
def primary_outcomes(pdf_path: str) -> list[PrimaryOutcomeRow]:
"""Extract primary learning outcomes from an NCCA PDF via BAML."""
text = extract_pdf_text(pdf_path)
outcomes = b.ExtractPrimaryLearningOutcomes(text)
for o in outcomes:
yield PrimaryOutcomeRow(
stage=o.stage,
curriculum_area=o.curriculum_area,
learning_outcome=o.learning_outcome,
)
pipeline = dlt.pipeline(
pipeline_name="ireland_primary_curriculum",
destination="ducklake",
dataset_name="cianfhoghlaim.education.ie",
)
load_info = pipeline.run(primary_outcomes("ncca_primary.pdf"))
print(load_info)
The BAML class is the single source of truth — both the
Pydantic BaseModel and the dlt primary_key derive from it.
Canonical Cianfhoghlaim pattern — British-Isles Education pipeline (lc6)
The post-v4 canonical pipeline ingests one of the six Irish Leaving
Certificate subjects (Mathematics, Chemistry, Geography, Gaeilge,
English, Computer Science) from a BAML-extracted NCCA/SEC PDF,
fan-outs to DuckLake + LanceDB, and is scheduled by Dagster via
@dlt_assets:
import dlt
from cianfhoghlaim.dlt.british_isles.ireland.education.ncca import (
mathematics_syllabus as mathematics_syllabus_source,
)
from cianfhoghlaim.baml.education.lc_extraction.curriculum_syllabus import b
@dlt.resource(
name="lc_mathematics_syllabus",
write_disposition="merge",
primary_key=["subject", "level", "language", "module_id"],
)
def mathematics_syllabus(pdf_path: str):
"""Extract Mathematics LC syllabus modules from an NCCA PDF via BAML."""
doc = b.ExtractCurriculumSyllabus(text=open(pdf_path).read())
for module in doc.modules:
yield {
"subject": "mathematics",
"level": doc.level,
"language": doc.language,
"module_id": module.id,
"title": module.title,
"hours": module.hours,
"learning_outcomes": module.learning_outcomes,
}
pipeline = dlt.pipeline(
pipeline_name="lc6_ncca",
destination="ducklake",
dataset_name="cianfhoghlaim.leaving_cert",
)
load_info = pipeline.run(
mathematics_syllabus("leaving_certificate/mathematics/en/...syllabus.pdf")
)
print(load_info)
The same pattern is repeated for the other 5 LC subjects
(chemistry_syllabus, geography_syllabus, gaeilge_syllabus,
english_syllabus, computer_science_syllabus) plus a
government_circulars resource for gov.ie education circulars.
Each resource is wrapped in @dlt_assets in
orchestration/defs/2_materials/ and contributes
to the 7 v1 CocoIndex Apps (6 LC + government_circulars) and
the 4 MotherDuck Dives (lc_syllabus_topics,
lc_exam_difficulty, lc_marking_complexity,
gov_circulars_archive).
Multi-destination fan-out (DuckDB + LanceDB + Memgraph)
import dlt
from dlt.destinations import duckdb
from lancedb import lancedb_adapter
from cognee import add as cognee_add, cognify
@dlt.resource(name="curriculum_chunks")
def chunks(pdf_path: str):
text = extract_pdf_text(pdf_path)
for chunk in chunk_text(text):
yield {"text": chunk, "source": pdf_path}
# Fan out to 3 destinations
pipeline = dlt.pipeline(destination="duckdb", dataset_name="curriculum")
load_info = pipeline.run([
chunks("ncca.pdf"), # → DuckDB
lancedb_adapter(chunks("ncca.pdf"), embed=["text"]), # → LanceDB
cognee_destination(chunks("ncaa.pdf")), # → Cognee
])
One pipeline run, three destinations. State is unified (one
pipeline.last_trace, not three).
Dagster asset wrapping (@dlt_assets)
from dagster_dlt import DagsterDltResource, dlt_assets
import dlt
@dlt_assets(
dlt_source=ireland_curriculum_source(),
dlt_pipeline=dlt.pipeline(
pipeline_name="ireland_curriculum",
destination="ducklake",
dataset_name="cianfhoghlaim.education.ie",
),
)
def ireland_curriculum_assets(context, dlt_run_resource: DagsterDltResource):
yield from dlt_run_resource.run(context=context)
# Schedule
@schedule(cron_schedule="0 2 * * *", job=ireland_curriculum_assets_job) # 02:00 UTC daily
def ireland_curriculum_schedule(): ...
See references/dagster-dlt-assets.md for the full pattern with
multiprocess_executor, parallel assets, and incremental loading.
4. Performance & anti-patterns
✅ Do:
- Use
parallelized=Truefor any resource fetching > 1k rows - Use
add_limit(N)to cap a source for testing - Use file rotation (
dlt.pipeline(..., progress="log")+ chunked writes) for > 1M rows - Use
write_disposition="merge"with an explicitprimary_keyfor upserts - Use
columns=PydanticModelfor type-safe pipelines (the BAML pattern) - Pre-validate inputs before the API call (catches bad PDFs early)
❌ Don't:
- Fetch all data in a single
fetch_all()call (OOM risk for > 1M rows) - Use
write_disposition="merge"without aprimary_key(silently appends duplicates) - Import
cianfhoghlaim.data_platform.dlt_sourcesfrom withincianfhoghlaim/(use relative imports (from .dlt...) - Hand-write DDL for the destination (let dlt infer the schema from the resource yield)
- Run live web scraping without
USE_LOCAL_SCRAPES=truefirst (drains API credits and risks rate limits) - Add a BAML client inline in a function (use a named client in
baml/clients.baml) - Pin
dlt==1.27.0ordlt==1.27.1— both YANKED from PyPI for a data-loss bug; the fix is 1.27.2 (or upgrade to ≥ 1.28.1). - Use
write_disposition="replace"(deprecated in 1.28.0) — use therefreshparameter instead. - Call
dlt ai ...(moved todlthub ai ...in 1.27.0). - Call
dlt dashboardwithoutpip install dlt[hub](1.27.0 split).
5. Reference index
The 3 reference files in references/ that were previously
orphaned are now linked from here:
references/dlthub.md— the generic dltHub expert skill (501 lines; write_disposition matrix, REST API source, sources + destinations reference)references/dlthub-codebase-analysis.md— dltHub code-level design-patterns analysis (decorator, builder, factory, repository, strategy; resource / source / write-disposition patterns)references/dlt-baml-orpc-mcp-typesafe-pipeline-analysis.md— the full type-safe pipeline architecture (BAML → dlt columns → oRPC contract → MCP tool)
New references (added by the sync-skills-from-docs change):
references/destinations-lancedb.mdreferences/destinations-cognee-memgraph.mdreferences/destinations-graphiti.mdreferences/performance-optimisation.mdreferences/dagster-dlt-assets.mdreferences/openapi-generator.mdreferences/deploy-gcp-cloud-function-webhook.mdreferences/deploy-gcp-cloud-function.mdreferences/deploy-gcp-cloud-run.mdreferences/deploy-modal.mdreferences/sqlmesh-init.mdreferences/dlt-transformations.mdreferences/crawl4ai-dlt-summary.mdreferences/type-safe-pipeline.md
6. Cross-references
- The
dltskill is consumed by:data-engineeragent (the primary user) - The
dltskill collaborates with:bamlskill (type-safe pipelines),dagsterskill (@dlt_assetswrapping),sqlmeshskill (sqlmesh init -t dlt),lancedbskill (vector destination),cogneeskill (knowledge-graph destination),motherduckskill (MotherDuck destination) - The
dltskill feeds into:explore-data(analysis_plan.md) andbuild-notebook(marimo notebooks) for downstream visualisation
7. Examples
See ./examples/ for upstream dlt reference
British-Isles Education pipeline use case (BIEP, post-v4)
The dlt skill is the data-load backbone for the British-Isles
Education pipeline (openspec/changes/lc6-biep/). It drives:
- 7 per-subject BAML extraction stages (syllabus / exam paper
layout / marking scheme / cross-linguistic / syllabus diagram)
across the 6 LC subjects (Mathematics, Chemistry, Geography,
Gaeilge, English, Computer Science) — 42 lc5/lc6 Dagster assets
total in
orchestration/defs/2_materials/. gov.iecirculars — thegovernment_circularsresource mirrors the 7th v1 CocoIndex App (government_circulars), landing pages fromgov.ie/.../circulars/...intocianfhoghlaim.education.ie.gov_circulars_archive(one of the 4 canonical MotherDuck Dives).- Dual-language fan-out — every resource is partitioned by
language(en/ga) so the Gaeilge syllabus runs in parallel with English via the same@dlt_assetswrapper. - BAML → dlt columns — the
lc5/lc6extraction schemas (e.g.ExtractCurriculumSyllabus,ExtractMarkingSchemeGuideline) are theprimary_keysource-of-truth for themergewrite disposition.
Cross-references:
baml/SKILL.md— the BAML extraction functionsdagster/SKILL.md— the@dlt_assetswrappers + lc6 schedulemotherduck/SKILL.md— the 4 Dives (lc_syllabus_topics,lc_exam_difficulty,lc_marking_complexity,gov_circulars_archive)cocoindex/SKILL.md— the 7 v1 Apps (6 LC subjects +government_circulars)marimo/SKILL.md— the 6 per-subject marimo notebookschange-detection/SKILL.md— the NCCA / SEC /gov.iesitemap-hash sensors
2026-06 updates (from the upstream-package-monitoring openspec change)
-
dltHub Pro launched 2026-04-14. The Pro tier adds 9,700+ known source contexts that DLT can pull from in one call. The Cianfhoghlaim dev plan is tracked by
openspec/changes/dlt-pro-source-registry/. -
Cortex Code (Snowflake's AI assistant, launched ~9 weeks before dltHub Pro) integrates directly with the dlt Pro source registry.
-
ADE-Bench (the AI data-engineer benchmark) reported 65% task success on Snowflake via Cortex Code vs 58% for Claude Code. The paper's key finding: "without the workbench, the agent leaked credentials" — directly validates Cianfhoghlaim's strict-secret-hydration mandate (see
docs/secrets/secrets_management_plan.mdfor the Infisical + Locket + mise three-way contract). -
dlthub upstream monitor —
dlthub_blog.ymlinbonneagar/firecrawl/monitors/upstream_packages/is the Firecrawl monitor that detects new source-context additions, ADE-Bench results, and Cortex Code integration updates via the LLM-judge--goalfilter. The n8n workflowbonneagar/stacks/n8n/workflows/upstream-blog-monitor.jsonwrites the payload tos3://cianfhoghlaim-upstream-webhooks/dlthub/...jsonland triggers the Dagster assetupstream_blog_monitor_ingest. notebooks: -
data_engineering_dlt_small-data-sf-2025_elvis.ipynb— Dremio "small data" workshop (SF 2025): dlt pipelines for small files, REST API ingestion patterns, and DuckDB destination examples.
v7 flattening migration notes (added 2026-07-19)
Per openspec/changes/2026-07-14-fix-foundation-v7-flattening-and-baml-drift-v1:
- DLT sources are now at
cianfhoghlaim.dlt.british_isles.<jurisdiction>.education.<source> - The canonical CLI entry point is
python -m cianfhoghlaim.dlt.cli run-pipeline(NOTpython -m cianfhoghlaim.dlt.run_pipelinewhich was the pre-v7 name) - The new BIEP v3 jurisdiction pipelines live at:
dlt_sources/british_isles/ireland/education/ireland_jurisdiction_pipeline.pydlt_sources/british_isles/england/education/england_jurisdiction_pipeline.pydlt_sources/british_isles/sct_wls_ni/education/sct_wls_ni_jurisdiction_pipeline.pydlt_sources/british_isles/crown_dependencies/education/crown_dependencies_jurisdiction_pipeline.py
- Each pipeline uses
dlt.common.curriculum_registry.SubjectRegistryfor canonical subject metadata - The destination is configured via
dlt.common.destinations_cianfhoghlaim.get_dlt_destination(use_ducklake=True) - The 4 destinations: local DuckDB, MotherDuck BYOB, Garage S3 + Lakekeeper, R2 + Lakekeeper
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |