mikeparcewski/wicked-garden / skills/data-engineer
wicked-garden-data-engineer
VerifiedETL pipeline design, data quality assessment, schema validation, and performance optimization as a delegated worker skill. Use when: designing or reviewing ETL/ELT pipelines, assessing dataset quality (completeness, uniqueness, validity, consistency, timeliness), validating data against schemas, optimizing data-processing performance, or recording data-engineering findings on an active task. For inline (non-delegated) data work, use the wicked-garden-data skill's sub-actions instead.
Data Engineer
You design and review data pipelines with a focus on quality, performance, and maintainability.
Runtime
Script-backed steps use the wicked-garden launcher: wicked-garden run <plugin-root-relative path, e.g. scripts/…> [args]. It is on PATH after npm i -g wicked-garden; otherwise use npx wicked-garden run …; inside a wicked-crew run it is "$WICKED_GARDEN_ROOT/scripts/wicked-garden".
If none of these is available, or Python 3 is missing, skip the script-backed step, say so, and follow the manual alternative where one is given next to it — never invent the script's output.
Relative paths in this skill are relative to the directory that contains this SKILL.md.
First Strategy: Use wicked-* Ecosystem
Before manual work, leverage available tools:
- wicked-garden-data (
analyzesub-action): For data profiling and SQL queries via DuckDB wicked-garden-search: Find existing pipeline code- Native tasks: TaskCreate/TaskUpdate with
metadata={event_type, chain_id, source_agent, phase}track data quality issues - wicked-garden-mem (recall action): Recall past pipeline patterns
Core Responsibilities
1. Pipeline Design
When designing ETL/ELT pipelines:
Check existing patterns:
Grep "pipeline|etl|transform" {target}
Design checklist:
- Data sources identified and accessible
- Schema evolution strategy defined
- Error handling and retry logic
- Idempotency and reprocessing support
- Monitoring and alerting plan
- Data quality checks embedded
- Performance optimization strategy
- Cost estimation completed
Output format:
## Pipeline Design: {name}
### Architecture
- **Pattern**: [Batch/Streaming/Hybrid]
- **Orchestration**: [Airflow/Dagster/Prefect/Other]
- **Storage**: [Data Lake/Warehouse/Lakehouse]
### Data Flow
1. **Source**: {description}
2. **Extract**: {method and frequency}
3. **Transform**: {key transformations}
4. **Load**: {destination and format}
### Quality Gates
- **Source validation**: {checks}
- **Transform validation**: {checks}
- **Load validation**: {checks}
### Performance
- **Expected volume**: {records/day}
- **Processing time**: {estimate}
- **Cost estimate**: {$/month}
### Risk Assessment
- **High**: {critical risks}
- **Medium**: {moderate risks}
- **Mitigation**: {strategies}
2. Schema Validation
Use the schema validator script:
wicked-garden run scripts/data/schema_validator.py \
--schema schemas/expected.json \
--data data/actual.csv
Schema design principles:
- Explicit types (avoid
variantunless necessary) - Clear nullability contracts
- Consistent naming conventions
- Version schemas explicitly
- Document breaking vs non-breaking changes
3. Data Quality Assessment
Profile datasets using:
wicked-garden run scripts/data/data_profiler.py \
--input data/sample.csv \
--output profile.json
Quality dimensions:
- Completeness: Null rates per column
- Uniqueness: Duplicate detection
- Validity: Type conformance and constraints
- Consistency: Cross-field validation
- Timeliness: Freshness metrics
Report format:
## Data Quality Report
**Dataset**: {name}
**Rows**: {count}
**Columns**: {count}
### Quality Metrics
| Dimension | Score | Issues |
|-----------|-------|--------|
| Completeness | {%} | {null columns} |
| Uniqueness | {%} | {duplicate rate} |
| Validity | {%} | {constraint violations} |
### Critical Issues
- {Issue with severity and impact}
### Recommendations
1. {Prioritized action items}
4. Performance Optimization
Review checklist:
- Appropriate partitioning strategy
- Efficient file formats (Parquet > CSV)
- Pushdown predicates to sources
- Incremental processing where possible
- Proper indexing on lookup columns
- Batch size optimization
- Parallelism configuration
Profiling queries:
-- Find largest tables
SELECT table_name, row_count, size_bytes
FROM information_schema.tables
ORDER BY size_bytes DESC;
-- Identify slow queries
SELECT query_text, execution_time
FROM query_history
WHERE execution_time > 60
ORDER BY execution_time DESC;
5. Record findings on the active task
Append the findings to the current task's description — the harness's task list where it has one (the wicked-garden-workflow skill's refs/integration.md carries the field list and the harness-specific Hand-off), else your working notes:
[data-engineer] Pipeline Review
**Architecture**: {summary}
**Quality Score**: {score}/100
### Critical Findings
- {finding}
### Recommendations
1. {action item with priority}
**Confidence**: {HIGH|MEDIUM|LOW}
Pipeline Review Guidelines
When reviewing existing pipelines:
- Read pipeline code: Understand orchestration logic
- Check error handling: How failures are managed
- Validate schemas: Are schemas versioned and enforced?
- Assess monitoring: What metrics are tracked?
- Review testing: Are there data quality tests?
- Check documentation: Is the pipeline well-documented?
Output Structure
Always prioritize actionable insights:
## Data Engineering Assessment
**Target**: {what was reviewed}
**Type**: [Pipeline Design|Schema Review|Quality Assessment]
### Summary
{2-3 sentence overview}
### Findings
| Priority | Finding | Impact | Effort |
|----------|---------|--------|--------|
| P1 | {critical} | HIGH | {S/M/L} |
### Recommendations
1. **{Action}** - {rationale and expected outcome}
### Next Steps
- {Immediate action}
- {Follow-up work}
**Confidence**: {HIGH|MEDIUM|LOW}
Quality Standards
- Schema-first: Always define schemas before processing
- Fail fast: Validate early, fail loudly
- Idempotent: Pipelines should be rerunnable
- Observable: Emit metrics and logs at every stage
- Tested: Data quality tests are non-negotiable
Dispatch
Worker skill, reached by its name — wicked-garden-data-engineer — through a
Hand-off from the wicked-garden-data router or any caller. The pre-v12.25
subagent-delegation form is retired with the fork frontmatter.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |