motherduckdb/agent-skills / plugins/motherduck-skills-claude/skills/motherduck-build-data-pipeline
motherduck-build-data-pipeline
UnverifiedBuild ingestion-to-serving pipelines on MotherDuck, including stage boundaries, transformations, and publication. Use for any recurring ingestion from an API, SaaS app, or database into MotherDuck.
Build a Data Pipeline with MotherDuck
Start Here: Is a MotherDuck Server Active?
Use an active remote MotherDuck MCP server or local MotherDuck server to inspect the in-scope database, schema, grain, keys, and relevant metrics. Reuse known context and narrow discovery to the requested work; do not scan the whole workspace by default. Let the actual data model shape the result.
Resolve the target from the request or active context. Ask only if ambiguity materially affects the result. Without a server, use supplied schema and explicit assumptions for planning; do not imply live validation.
Pipeline Defaults
- batch over streaming
- raw landing before curation
- explicit raw -> staging -> analytics boundaries
- bulk ingest paths over row-by-row writes: stage files and load with
read_parquet/read_json_autoor oneINSERT ... SELECT; neverexecutemany()or per-rowINSERTloops against MotherDuck - idempotent stage rebuilds or append contracts before scheduled automation
- verify the MotherDuck-supported DuckDB client version before recommending upstream-only write, checkpoint, or lakehouse features
- native MotherDuck storage unless DuckLake is explicitly required
- MotherDuck CLI for Flight source and large file-shaped output when the agent has a shell; MCP for chat-only operation
- a
flightsGuide for reusable scheduling, naming, secret, and ingestion conventions when the organization has them - Flights sized to their Python work (
F4when the job mostly sends SQL) and instrumented for the Flight Explainer so each run shows step status and the failing step - AI enrichment (classification with
prompt_jev(), extraction withprompt()) as an incremental transform step that writes results to a table, never recomputed at read time
Workflow
- Inspect the available MotherDuck server or supplied source and target context, including the current target data model.
- Define raw, staging, and analytics boundaries.
- Ingest raw data.
- Deduplicate, type, and promote into staging.
- Materialize analytics-ready outputs.
- Validate counts, freshness, uniqueness, and business metrics before publishing downstream assets.
- When durable context is part of delivery, capture stable business definitions and operating caveats in referenced Guides; keep executable transformation logic in source control.
Match execution to the request: answer, review, or planning work returns the requested pipeline artifacts; build or change work creates the requested in-scope files and warehouse objects and validates them. Ask before destructive actions, unrelated external writes, or a material expansion of scope.
When this skill produces a native DuckDB (md:) connection, watermark it with custom_user_agent=agent-skills/3.2.0(harness-<harness>;llm-<llm>). If metadata is missing, fall back to harness-unknown and llm-unknown.
Output
For a full engagement, cover the following as relevant to the request:
- the stage layout
- the ingestion method
- the transformation sequence
- the serving tables or views
- the validation checks
For explicit structured JSON requests, read the output contract. Otherwise use the format that fits the requested deliverable.
References
Read only the sections relevant to the task; these are guidance, not a mandatory itinerary.
references/dlt-dbt-motherduck-project/-- fully runnable MotherDuck reference project usingdlt,dbt-duckdb, and validation queriesreferences/PIPELINE_IMPLEMENTATION_GUIDE.md-- stage design, transformation sequencing, and ingestion-to-serving examples
Examples
Read the execution reference only to run the bundled examples or reproduce their validation.
Related Skills
Load related skills only for missing capabilities; reuse established context.
motherduck-connect-- choose the right connection pathmotherduck-load-data-- ingestion mechanicsmotherduck-model-data-- shape the analytics layermotherduck-query-- write transformations and validationsmotherduck-share-data-- publish curated outputsmotherduck-ducklake-- only when open-table-format storage is a real requirementmotherduck-create-flight-- schedule and run the pipeline on MotherDuck compute
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |