Skip to content

borghei/Claude-Skills / engineering/senior-data-engineer

senior-data-engineer

Verified

Data engineering for batch and streaming pipelines with Airflow, dbt, Spark, and Kafka. Use when designing data architectures, building pipelines, adding data-quality checks, optimizing ETL/ELT, or troubleshooting pipeline failures.

Senior Data Engineer

Generate pipeline configurations (Airflow, Prefect, Dagster), validate data quality with profiling and anomaly detection, and optimize SQL/Spark performance with actionable recommendations.

Core Capabilities

  • Pipeline generation — Airflow/Prefect/Dagster DAG code for batch and incremental loads, with DAG validation.
  • Data quality — schema validation, profiling, anomaly detection, data contracts, and Great Expectations suite generation.
  • ETL/ELT optimization — SQL and Spark analysis, partition strategy, and query cost estimation per warehouse.
  • Architecture decisions — batch vs streaming and warehouse vs lakehouse trade-off frameworks.
  • Reliability patterns — incremental watermarks, dead letter queues, freshness checks, and schema-drift detection.

When to Use

  • Designing a data architecture or choosing batch vs streaming / warehouse vs lakehouse.
  • Building or generating Airflow/Spark/dbt pipelines.
  • Adding data-quality checks or data contracts.
  • Optimizing slow ETL/ELT queries or troubleshooting pipeline failures.

Clarify First

Before generating pipelines, settle the inputs below. When one is missing or vague and would change the result, ask about it; when it wouldn't, use a sensible default and say which one you used.

  • Orchestrator: Airflow / Prefect / Dagster (--type; changes the generated DAG code).
  • Source, destination & load mode: systems involved and batch vs incremental (--source/--destination/--mode; shapes the pipeline).
  • Data-quality expectations: the schema and contracts to enforce (drives the Great Expectations suite generation).

Ask about the two or three inputs that matter most for this request, not the whole list. If the user wants a result straight away, produce it and state your assumptions at the top.

Quick Start

# Generate an Airflow DAG for incremental PostgreSQL -> Snowflake. Targets Airflow 3 (`airflow.sdk`,
# current 3.4) with import fallbacks for 2.7+; --output creates missing parent folders
python scripts/pipeline_orchestrator.py generate \
  --type airflow --source postgres --destination snowflake \
  --tables orders,customers --mode incremental --schedule "0 5 * * *" --output dags/orders_etl.py

# Validate data against a schema (sample schema and data ship in assets/)
python scripts/data_quality_validator.py validate assets/sample_orders_flawed.csv \
  --schema assets/sample_schema.json --detect-anomalies

# Draft a schema from existing data, then tighten it by hand
python scripts/data_quality_validator.py schema assets/sample_orders_clean.csv --output schema.json

# Review a SQL query (static analysis; read the warehouse's query profile for ground truth)
python scripts/etl_performance_optimizer.py analyze-sql assets/sample_query.sql --warehouse snowflake

Schema file format (--schema): a JSON object with name, version, optional primary_key (list of columns), row_count_min/row_count_max, and columns, each with name, type (string, integer, float, boolean, date, datetime, email, uuid), and optional nullable, unique, min_value, max_value, min_length, max_length, pattern (regex) and allowed_values. See assets/sample_schema.json. A schema generated by the schema subcommand only records types and observed ranges, so validating the same data against it finds nothing by construction; add the real business rules before using it as a gate. The report's check counts list failed checks, so a clean run shows 0.

Exit codes: 0 ok; 1 validate/contract found error-severity failures, or pipeline_orchestrator.py validate found issues; 2 bad input (missing file, invalid JSON or schema). assets/sample_orders_clean.csv validates clean (exit 0); assets/sample_orders_flawed.csv has a duplicate key, a null and a malformed email, a negative amount, an unknown status and a bad date (exit 1).

Tools

ToolSubcommandsPurpose
pipeline_orchestrator.pygenerate, validate, templateGenerate Airflow/Prefect/Dagster pipeline code, validate DAGs
data_quality_validator.pyvalidate, profile, generate-suite, contract, schemaSchema validation, profiling, anomaly detection, Great Expectations
etl_performance_optimizer.pyanalyze-sql, analyze-spark, optimize-partition, estimate-cost, templateSQL/Spark optimization, partition strategy, cost estimation

--json is available on data_quality_validator.py validate, profile and contract and on the four etl_performance_optimizer.py analysis subcommands (analyze-sql, analyze-spark, optimize-partition, estimate-cost). pipeline_orchestrator.py has no --json flag: generate and template print pipeline code, and validate always prints its result as JSON. Every subcommand except pipeline_orchestrator.py validate takes --output to write to a file.

References

Load the reference that matches the task — keep this file lean and pull detail on demand:

Scope & Limitations

In Scope:

  • Pipeline code generation for Airflow, Prefect and Dagster with pipeline_orchestrator.py (generate, validate, template), including incremental and full load modes.
  • Data quality checks on CSV, JSON or JSONL files with data_quality_validator.py: schema validation, profiling, anomaly detection, data contracts, schema inference and Great Expectations suite generation.
  • SQL and Spark analysis with etl_performance_optimizer.py: query recommendations, Spark job metrics review, partition strategy and query cost estimates for BigQuery, Snowflake, Redshift and Databricks.
  • Batch versus streaming and warehouse versus lakehouse decision frameworks, worked pipelines and troubleshooting in references/.

Out of Scope:

  • Running pipelines or connecting to sources, warehouses, Kafka or Spark clusters. The scripts work on local files.
  • Snowflake-specific query tuning and warehouse sizing (see engineering/snowflake-development/).
  • An organization-wide data quality program (see engineering/data-quality-auditor/).
  • Model training and feature pipelines for ML (see engineering/senior-ml-engineer/).

Important Caveats:

  • Generated DAG and flow code is a scaffold. It imports Airflow, Prefect, Dagster or pandas, which are not installed by the skill, and it needs real connections, credentials and testing before use.
  • SQL analysis is pattern-based on query text. It does not read execution plans or table statistics unless a stats file is provided.
  • Cost estimates use per-credit and per-TB prices built into the script. Warehouse pricing changes and varies by contract and region, so estimates are for comparison, not budgeting.
  • The YAML used for configs and contracts is read with a small stdlib parser, so complex YAML may not load as expected.

Anti-Patterns

Validating Against an Inferred Schema

Mistake: Generating a schema from today's data with schema, then gating tomorrow's loads on it. Why it happens: It is one command and the first run passes. Instead: An inferred schema encodes whatever the data already is, including its errors. Add the business rules (keys, allowed values, ranges) by hand and review them with the data owner.

Non-Idempotent Loads

Mistake: Appending each run's rows without a merge key, so retries and backfills duplicate data. Why it happens: Appends are simpler and work on the happy path. Instead: Load by partition or MERGE on a business key so re-running any date gives the same result; test by running the same date twice.

Relative Start Dates in DAGs

Mistake: start_date=days_ago(1) or datetime.now() in a DAG. Why it happens: Older tutorials used it. Instead: A fixed, timezone-aware start_date with catchup set deliberately. Relative dates shift every parse, break backfills, and the helper is gone in Airflow 3.

Files

FilePurpose
scripts/pipeline_orchestrator.pyAirflow / Prefect / Dagster code generation and DAG validation
scripts/data_quality_validator.pySchema validation, profiling, anomalies, contracts, schema inference, GE suites
scripts/etl_performance_optimizer.pySQL and Spark analysis, partition strategy, cost estimates
assets/sample_schema.jsonSchema with keys, types, ranges and allowed values
assets/sample_orders_clean.csvData that passes the schema (exit 0)
assets/sample_orders_flawed.csvData with six rule violations (exit 1)
assets/sample_query.sqlQuery for analyze-sql (SELECT *, function on a filter column, ORDER BY without LIMIT)
references/*.mdWorked pipelines, decisions and troubleshooting, architecture, modeling, DataOps
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 319kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall