Skip to content

notque/vexjoy-agent / skills/analysis/data

Data analysis, reference enrichment, and data engineering: pipelines and warehouses, server database design and tuning, SQLite with Peewee.

references/data.md

View on GitHub
# Data Engineering Reference

> **Scope**: OLAP pipeline design, ETL/ELT orchestration (Airflow, Prefect, Dagster, dbt), dimensional modeling (star, snowflake, data vault, SCD), data quality, stream processing (Kafka, Spark, Flink), warehouse and lake storage (BigQuery, Redshift, Parquet, Delta Lake, Iceberg), lineage.
> **Not here**: OLTP schema and query tuning (`database.md`); embedded SQLite and Peewee (`sqlite-peewee.md`); RabbitMQ or AMQP topology, clustering, and HA (`skills/infrastructure/kubernetes/references/rabbitmq-messaging.md`); data analysis (this skill, Mode A). This reference complements `database.md`, which owns OLTP concerns.

Full expertise list, default and optional behaviors, scope limits, and output format: [data/expertise.md](data/expertise.md). Load it when scoping or designing a pipeline.

## Hard rules

- **Prevent over-engineering**: build what is asked, not a platform. Use streaming only when batch is not enough. Use real-time CDC only when daily snapshots fall short. Three simple DAGs beat one "universal" pipeline framework.
- **Idempotency required**: every pipeline step is safe to re-run. Use MERGE/upsert, partition overwrite, or deduplication. A pipeline that creates duplicates on re-run is broken. Failures are inevitable; the only question is whether recovery is automatic or manual.
- **Grain first**: state every fact table's grain ("one row per ___") before designing columns. Wrong grain means wrong numbers, and wrong numbers undermine every decision built on them.
- **Quality gates before load**: validate schema and check null key columns before loading target tables. Bad data in a warehouse reaches every dashboard, report, and ML model downstream; catching it at the gate is far cheaper than fixing it later.

## Reference loading table

| Signal | Load | Why |
|---|---|---|
| Expertise, default/optional behaviors, scope limits, output format | [data/expertise.md](data/expertise.md) | Scoping and output contract |
| Pipeline error catalog (deadlocks, late data, schema drift, SCD mismatch, duplicates) | [data/error-catalog.md](data/error-catalog.md) | Error to fix mappings |
| Pipeline preferred patterns, detection signals, domain rationalizations | [data/preferred-patterns.md](data/preferred-patterns.md) | Patterns with grep detection |
| Pipeline hard gates, STOP blocks, blocker criteria, death loop prevention | [data/gates-and-blockers.md](data/gates-and-blockers.md) | Gates and when to stop |
| MERGE, INSERT ON CONFLICT, partition overwrite, deduplication, incremental SQL | [data/sql.md](data/sql.md) | Idempotent SQL patterns |
| dbt tests, Great Expectations, source freshness, row count reconciliation | [data/testing.md](data/testing.md) | Pipeline testing |
| Partitioning, clustering, materialized views, incremental processing, warehouse cost | [data/performance.md](data/performance.md) | Warehouse performance |

Implementation Schema details: [output-schemas.md](../../../shared-patterns/output-schemas.md).
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall