borghei/Claude-Skills / engineering/senior-data-engineer
senior-data-engineer
VerifiedData engineering for batch and streaming pipelines with Airflow, dbt, Spark, and Kafka. Use when designing data architectures, building pipelines, adding data-quality checks, optimizing ETL/ELT, or troubleshooting pipeline failures.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 10.4 KB |
| assets/sample_orders_clean.csv | other | 318 B |
| assets/sample_orders_flawed.csv | other | 302 B |
| assets/sample_query.sql | other | 133 B |
| assets/sample_schema.json | config | 534 B |
| references/data_modeling_patterns.md | markdown | 26.1 KB |
| references/data_pipeline_architecture.md | markdown | 33.9 KB |
| references/dataops_best_practices.md | markdown | 42.4 KB |
| references/decisions-and-troubleshooting.md | markdown | 2.4 KB |
| references/pipeline-workflows.md | markdown | 4.8 KB |
| scripts/data_quality_validator.pyexecutable | script | 59.3 KB |
| scripts/etl_performance_optimizer.pyexecutable | script | 65.9 KB |
| scripts/pipeline_orchestrator.pyexecutable | script | 36.3 KB |
references/decisions-and-troubleshooting.md
View on GitHub# Architecture Decisions, Anti-Patterns & Troubleshooting
Read this when choosing batch vs streaming or warehouse vs lakehouse, avoiding common pipeline mistakes, or diagnosing pipeline failures.
## Architecture Decision Framework
| Question | Batch | Streaming |
|----------|-------|-----------|
| Latency requirement | Hours to days | Seconds to minutes |
| Processing complexity | Complex transforms, ML | Simple aggregations |
| Cost sensitivity | More cost-effective | Higher infra cost |
| Error handling | Easy reprocessing | Requires careful DLQ design |
**Decision tree:**
```
Real-time insight needed?
Yes -> Exactly-once needed?
Yes -> Kafka + Flink/Spark Structured Streaming
No -> Kafka + consumer groups
No -> Daily volume > 1TB?
Yes -> Spark/Databricks
No -> dbt + warehouse compute
```
| Feature | Warehouse (Snowflake/BigQuery) | Lakehouse (Delta/Iceberg) |
|---------|-------------------------------|---------------------------|
| Best for | BI, SQL analytics | ML, unstructured data |
| Storage cost | Higher (proprietary) | Lower (open formats) |
| Flexibility | Schema-on-write | Schema-on-read |
## Anti-Patterns
1. **Full table reload on every run** -- use incremental loads with watermark columns.
2. **No dead letter queue** -- failed records silently dropped. Always route failures to a DLQ.
3. **Timezone mismatch** -- normalize all timestamps to UTC at extraction.
4. **Missing freshness checks** -- add `dbt source freshness` before transforms start.
5. **Skipping schema drift detection** -- use `mergeSchema` option or data contracts to catch new columns.
## Troubleshooting
| Problem | Cause | Solution |
|---------|-------|----------|
| Pipeline silently produces zero rows | Timezone mismatch on watermark column | Normalize to UTC; add row-count assertion |
| Spark shuffle 10x slower than expected | Data skew on join key | Salt the key or broadcast the smaller table |
| Airflow shows "no tasks to run" | Circular dependency or import error | `airflow dags list-import-errors`; fix import |
| dbt succeeds but dashboards stale | Source freshness not checked | Add `dbt source freshness` as prerequisite task |
| Kafka consumer lag grows unbounded | Throughput < producer rate | Increase partitions, scale consumers, batch `max.poll.records` |
| Quality validator false-positive anomalies | Z-score threshold too tight | Raise threshold or switch to IQR mode |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |