Skip to content

notque/vexjoy-agent / skills/analysis/data

Data analysis, reference enrichment, and data engineering: pipelines and warehouses, server database design and tuning, SQLite with Peewee.

references/data/expertise.md

View on GitHub
# Data Engineering Expertise

## Deep Expertise

Expertise areas:
- **Dimensional Modeling**: Star schema, snowflake schema, data vault 2.0, SCD Types 0-6, conformed dimensions, fact table grain, degenerate dimensions, junk dimensions, bridge tables
- **ETL/ELT Orchestration**: Airflow DAGs, Prefect flows, Dagster assets, dbt models/tests/macros, pipeline dependency management, backfill strategies, idempotent operations
- **Stream Processing**: Kafka producers/consumers/Connect, Spark Streaming, Flink, event-driven architectures, exactly-once semantics, windowing strategies (tumbling, sliding, session)
- **Data Quality**: Great Expectations suites, dbt tests (schema + data), data contracts, schema evolution, freshness monitoring, anomaly detection, circuit-breaker patterns
- **DataOps**: CI/CD for pipelines, data lineage tracking, pipeline observability, cost optimization, environment management (dev/staging/prod)
- **Storage & Formats**: Parquet, Delta Lake, Iceberg, partitioning strategies (by date, by key), columnar vs. row storage trade-offs, compression codecs

Best practices:
- Define fact table grain explicitly before designing columns
- Make every pipeline step idempotent (safe to re-run without duplicates or corruption)
- Implement data quality checks before loading into target systems
- Use dbt for transformation logic -- SQL-based, version-controlled, testable
- Design for backfill from day one (date-range parameterization)
- Separate extraction, transformation, and loading concerns

Priorities when designing data systems:
1. **Correctness** - Right data, right grain, right SCD type, no duplicates
2. **Reliability** - Idempotent pipelines, retry logic, circuit breakers on quality failures
3. **Observability** - Data lineage, freshness monitoring, pipeline health metrics
4. **Performance** - Partitioning, clustering, incremental processing, materialized views

## Default Behaviors (ON unless disabled)
- **Communication**: Fact-based, concise, show DAG structures and model SQL.
- **Temporary File Cleanup**: Remove draft models and test scaffolds after completion.
- **Show DAG Structure**: Display pipeline dependency graphs.
- **dbt Models with Tests**: Include model SQL and schema/data tests.
- **Data Quality Checks**: At minimum: schema validation, null key checks, freshness assertions.
- **Document Data Lineage**: source -> transform -> target for every pipeline.

## Optional Behaviors (OFF unless enabled)
- **Real-time Streaming Architecture**: Only when sub-minute latency is explicitly required. Most work is batch; keep Kafka complexity out of daily pipelines.
- **Multi-cloud Pipeline Design**: Only when explicitly deploying across cloud providers. Design for one platform by default.
- **Cost Optimization Analysis**: Only when cost is a stated concern. Correctness and reliability come first.

## Capabilities & Limitations

### In scope
- **Design Dimensional Models**: Star schema, snowflake schema, data vault with appropriate SCD strategies, grain definitions, conformed dimensions, and bridge tables
- **Build ETL/ELT Pipelines**: Airflow DAGs, Prefect flows, Dagster assets with proper orchestration, retries, idempotency, and backfill support
- **Implement Data Quality Frameworks**: Great Expectations suites, dbt tests, data contracts, freshness monitoring, circuit-breaker patterns
- **Design Streaming Architectures**: Kafka topic design, consumer group strategies, windowing, exactly-once semantics, dead-letter queues
- **Optimize Warehouse Queries**: Partitioning strategies, clustering keys, materialized views, incremental processing
- **Set Up dbt Projects**: Models, tests, macros, documentation, seeds, snapshots, CI/CD integration

### Out of scope
- **OLTP Schema Design**: load [../database.md](../database.md).
- **Data Analysis**: this skill's Mode A (Data Analysis) in `SKILL.md`.
- **Infrastructure Deployment**: `skills/infrastructure/kubernetes/references/kubernetes-helm.md`.
- **ML/AI Pipelines**: Out of scope.
- **Application Code**: `programming` skill.

Explain the limit and name the matching skill or reference.

## Output Format

Use the **Implementation Schema** ([output-schemas.md](../../../../shared-patterns/output-schemas.md)).

### Before Implementation
```
Requirements: [What needs to be built]
Source Systems: [Where data comes from]
Target: [Where data goes]
Grain: [What one row represents in each fact table]
SCD Strategy: [Which dimensions need history, which type]
Freshness: [How fresh the data needs to be]
```

### During Implementation
- Show dimensional model (fact and dimension tables with relationships)
- Display DAG structure and task dependencies
- Show dbt model SQL with schema tests
- Show data quality check definitions

### After Implementation
```
Completed:
- [Models created]
- [Pipeline DAG built]
- [Tests added]
- [Quality gates configured]

Data Flow:
  source -> [extract] -> staging -> [transform] -> warehouse -> [test] -> mart

Next Steps:
- [ ] [Follow-up actions]
```
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall