Skip to content

JPeetz/agent-skills / skills/data/data-pipeline-engineering

data-pipeline-engineering

Verified

Use this skill when designing, building, or debugging data pipelines — ETL/ELT workflows, data orchestration (Airflow/Dagster/Prefect), streaming (Kafka/Spark Streaming), schema drift handling, pipeline observability, backfill strategies, and data warehouse integration (BigQuery/Snowflake/Redshift). Covers batch and real-time patterns, idempotency, partitioning, incremental loads, and monitoring.

FileKindSize
CHANGELOG.mdmarkdown3.4 KB
LICENSEother1.0 KB
SKILL.mdskill md26.0 KB
evals/evals.jsonconfig5.1 KB
references/sources.mdmarkdown6.2 KB
scripts/data_freshness_check.pyscript9.1 KB
scripts/pipeline_lint.pyscript12.9 KB

evals/evals.json

View on GitHub
[
  {
    "id": "dpe-001",
    "description": "Design an hourly incremental ELT pipeline with idempotency and backfill",
    "input": "We have a Postgres orders table (about 200M rows) and need it in BigQuery for dashboards that must never be more than 30 minutes stale. It currently does a full reload every night and takes 2 hours. Design the replacement and give me the DAG.",
    "shouldTrigger": true,
    "expect": "Agent identifies full refresh as the root cost problem, chooses partition-scoped incremental load with a half-open watermark window read from the target, specifies MERGE on order_id with an updated_at guard plus dedupe before merge, sets daily/hourly partitioning on event_date with clustering, requires a partition filter and maximum_bytes_billed, specifies freshness/volume/uniqueness checks that block publish, and gives a partition-scoped backfill plan with throttling and a live-table swap. Notes catchup=False, max_active_runs=1, and deriving the partition from the logical date rather than now().",
    "assertions": [
      "recommends incremental over full refresh, with the reason stated in cost/scan terms",
      "watermark is read from the target and the window is half-open and bounded by the run's logical timestamp",
      "MERGE or partition-overwrite is specified with a concrete business key, not 'INSERT INTO ... SELECT'",
      "partitioning and clustering columns are named, and a partition filter is enforced on the large table",
      "quality checks are named and at least freshness and volume are marked blocking",
      "backfill is partition-scoped, throttled, and isolated from the live table",
      "does not use now()/CURRENT_TIMESTAMP for the partition bound"
    ]
  },
  {
    "id": "dpe-002",
    "description": "Debug duplicate and missing rows in a Kafka streaming sink",
    "input": "Our Kafka -> Spark Structured Streaming job writes clickstream events into Snowflake. After a weekend restart the dashboard shows event counts about 1.8x normal, and we're also seeing a few hundred messages sitting in the DLQ with 'column count mismatch'. Where do we start?",
    "shouldTrigger": true,
    "expect": "Agent separates the two incidents: (1) duplicates caused by non-idempotent sink plus offset/checkpoint mismatch on restart — recommends dedupe on a deterministic key (topic/partition/offset, or event_id) with MERGE into the target, exactly-once with checkpointing aligned to the sink, and idempotent producer settings; (2) schema drift — the mismatch is a drift event, so inspect the registry compatibility mode and the raw payload, add a tolerant reader with a quarantined _extra JSON column, replay the DLQ after fixing rather than dropping it. Also asks whether the 1.8x is duplicates or double-subscribed consumers, and recommends adding a uniqueness check plus volume check so the next occurrence pages before a human notices the dashboard.",
    "assertions": [
      "does not treat duplicates and the DLQ mismatch as one bug",
      "names a deterministic dedupe key and an idempotent sink write (MERGE/upsert), not just 'restart with exactly-once'",
      "explains the checkpoint/offset vs sink-commit gap as the duplicate source",
      "classifies column-count mismatch as schema drift and proposes a tolerant reader plus quarantine rather than dropping messages",
      "proposes replaying the DLQ after the fix and adding uniqueness/volume checks to detect recurrence",
      "considers the mundane explanation (double consumer / duplicated source) before assuming deep pathology"
    ]
  },
  {
    "id": "dpe-003",
    "description": "Backfill 18 months of history without breaking the live pipeline",
    "input": "We changed our revenue attribution logic and need to recompute 18 months of the orders_fact table, plus everything downstream. The table is live and the BI team uses it all day. How do we do this safely?",
    "shouldTrigger": true,
    "expect": "Agent scopes the work by partition and writes the partition list to a resumable manifest, proves idempotency by rerunning one known-good partition and comparing checksums, then backfills into an isolated table (or a separate warehouse/compute pool) with throttled concurrency, verifies, and swaps atomically. Cascades downstream dependents explicitly (asset graph or dbt state:modified+ selection) rather than assuming, tags runs as run_type=backfill so freshness alerts don't misfire, and schedules off-peak with lower priority. Warns against in-place UPDATE of production history and against 'rerun everything' as the first move.",
    "assertions": [
      "enumerates partitions explicitly and makes the run resumable",
      "verifies idempotency on a single good partition before starting the bulk run",
      "writes to an isolated target and swaps atomically instead of mutating the live table in place",
      "throttles concurrency and uses a separate queue/warehouse so production isn't starved",
      "names the downstream dependents and how the cascade is triggered",
      "tags backfill runs so monitoring distinguishes them from scheduled runs",
      "does not propose in-place UPDATE/DELETE of production history as the mechanism"
    ]
  }
]
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall