Skip to content

mikeparcewski/wicked-garden / skills/data

wicked-garden-data

Verified

Data engineering, analysis, and ML toolkit. One skill, routed sub-actions: analyze (interactive EDA on a CSV/Excel/data file), profile / validate / quality (schema-level data engineering ops), ml review / ml pipeline (model review + training-pipeline design), pipeline design / pipeline review (ETL architecture), ontology (recommend Schema.org, Dublin Core, DCAT, FOAF, GoodRelations, SKOS, or a custom shape for a dataset), shapes (a live source's data shapes, proposed for review). Use when: "analyze this CSV/Excel/data file" / "explore this dataset" / one-off data exploration; "profile this dataset" / "validate data against a schema" / "generate a data quality report"; "review this ML model" / "design an ML training pipeline"; "design a data pipeline" / "review this ETL"; "recommend an ontology for this data" / "map columns to a public ontology"; "propose data shapes for <source>"; or any former /wicked-garden:data:* invocation. Delegated deep passes: the `wicked-garden-data-engineer` worker skill.

FileKindSize
SKILL.mdskill md8.0 KB
refs/analysis-templates.mdmarkdown3.3 KB
refs/analyze.mdmarkdown5.1 KB
refs/data-examples.mdmarkdown4.9 KB
refs/data.mdmarkdown3.8 KB
refs/ml-templates.mdmarkdown4.0 KB
refs/ml.mdmarkdown5.7 KB
refs/pipeline-templates.mdmarkdown4.6 KB
refs/pipeline.mdmarkdown4.8 KB
refs/shapes.mdmarkdown6.1 KB

refs/pipeline.md

View on GitHub
# Data Pipeline Rubric — design / review

Apply this inline for ETL/streaming pipeline architecture.
Caller has pre-captured source / target / frequency (for design) or read pipeline
files at `<path>` (for review).

## design

Cover architecture pattern, orchestrator, stage flow, quality gates, monitoring, costs, risks.

```markdown
## Pipeline Design: {name}

### Architecture
- **Pattern**: {Batch|Streaming|Hybrid}
- **Orchestration**: {Airflow|Dagster|Prefect|Other}
- **Storage**: {Data Lake|Warehouse|Lakehouse}

### Data Flow
1. **Source**: {description — type, API/DB/file, credentials approach}
2. **Extract**: {method and frequency}
3. **Transform**: {key transformations}
4. **Load**: {destination and format}

### Quality Gates
- **Source validation**: {checks before processing}
- **Transform validation**: {checks mid-pipeline}
- **Load validation**: {row counts, key checks post-load}

### Performance
- **Expected volume**: {records/day}
- **Processing time estimate**: {duration}
- **Cost estimate**: {$/month}

### Risk Assessment
| Level | Risk | Mitigation |
|-------|------|------------|
| HIGH  | {critical risk} | {mitigation} |
| MED   | {moderate risk} | {mitigation} |

### Design Checklist
- [ ] Data sources identified and accessible
- [ ] Schema evolution strategy defined
- [ ] Error handling and retry logic designed
- [ ] Idempotency and reprocessing supported
- [ ] Monitoring and alerting plan
- [ ] Data quality checks embedded
- [ ] Cost estimation completed

**Confidence**: {HIGH|MEDIUM|LOW}
```

## review

Cover code quality, idempotency, monitoring, validation, silent-loss risk.
Emit P1/P2/P3 findings with code fixes.

```markdown
## Pipeline Review: {path}

**Architecture**: {summary — pattern, orchestrator, storage}
**Quality Score**: {score}/100

### Findings
| Priority | Finding | Impact | Effort | Fix |
|----------|---------|--------|--------|-----|
| P1 | {critical} | HIGH | {S/M/L} | {code snippet} |
| P2 | {moderate} | MED  | {S/M/L} | {code snippet} |
| P3 | {minor}    | LOW  | {S/M/L} | {note} |

### Review Checklist
- [ ] Error handling: failures managed and surfaced
- [ ] Schema versioned and enforced
- [ ] Monitoring: what metrics are tracked
- [ ] Testing: data quality tests present
- [ ] Idempotent: safe to reprocess without duplication
- [ ] Silent-loss risk: no data drops without alert
- [ ] Documentation: pipeline well-documented

### Recommendations
1. {Action with rationale and expected outcome}

**Confidence**: {HIGH|MEDIUM|LOW}
```

## Pattern selection

### Batch ETL
**Use when**: regular scheduled loads, historical processing.
**Pattern**: Extract → Transform → Validate → Load.
**Tools**: Airflow, Dagster, Prefect.

### Streaming pipeline
**Use when**: real-time processing, event-driven.
**Pattern**: Consume → Transform → Sink.
**Tools**: Kafka, Flink, Spark Streaming.

### Incremental processing
**Use when**: large datasets, only processing changes.
**Pattern**: watermark tracking + merge/upsert.

## Common issues

| Issue | Symptoms | Solution |
|-------|----------|----------|
| Fails halfway | Partial data, inconsistent state | Staging + commit pattern |
| Duplicates | Same data loaded multiple times | Watermarks + idempotency |
| Slow processing | Misses SLA | Profile and optimize bottlenecks |

## External integration discovery

Pipeline engineering can leverage available integrations by capability:

| Capability | Discovery Patterns | Provides |
|------------|-------------------|----------|
| **Warehouses** | `snowflake`, `databricks`, `bigquery` | Query execution, schema access |
| **ETL** | `airbyte`, `fivetran`, `dbt` | Pipeline status, model metadata |
| **Observability** | `monte-carlo`, `datadog` | Data quality metrics |

Discover available integrations via capability detection. Fall back to the
`analyze` sub-action of the wicked-garden-data skill for local file analysis
via DuckDB.

## Integration

- **Code search**: find pipeline code with Grep `"dag|pipeline"` or the estate MCP SearchEntity tool.
- **Native tasks**: track pipeline issues via TaskCreate with `metadata.event_type="task"`.
- **wicked-garden-mem (recall action)**: recall pipeline patterns.

## Engineering standards (apply to both)

- **Schema-first**: always define schemas before processing.
- **Fail fast**: validate early, fail loudly.
- **Idempotent**: pipelines rerunnable.
- **Observable**: emit metrics + logs at every stage.
- **Tested**: data quality tests are non-negotiable — unit test transforms,
  integration test the full pipeline, test error scenarios.
- **Documented**: clear lineage, versioned schemas, operations runbook.

## Detailed templates

See [pipeline-templates.md](pipeline-templates.md) for the full design
document (ASCII architecture diagram, error-handling matrix, monitoring
thresholds, backfill/recovery procedures, dependencies, deployment) and the
scored pipeline review report.
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall