lidge-jun/codexclaw / plugins/codexclaw/skills/dev-data
cxc-dev-data
VerifiedUse for analytics and data pipelines. Triggers: ETL, ELT, data quality, SQL optimization, schema drift, backfill, 데이터 파이프라인, 데이터 품질, 백필.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 2.6 KB |
| agents/openai.yaml | config | 161 B |
| references/01-ingestion-pipelines.md | markdown | 4.8 KB |
| references/02-quality-evolution.md | markdown | 4.3 KB |
| references/03-analysis-engines.md | markdown | 5.2 KB |
| references/04-governance-queries.md | markdown | 4.5 KB |
| references/governance.md | markdown | 3.3 KB |
| references/ml-pipeline.md | markdown | 8.4 KB |
| references/streaming.md | markdown | 3.1 KB |
| references/tools.md | markdown | 3.0 KB |
references/02-quality-evolution.md
View on GitHub## Pre-Flight Checklist
Before delivering:
- [ ] Input contract defined: source, schema, expected columns/types, and owner
- [ ] Pipeline is idempotent and restartable from the last successful checkpoint
- [ ] Data-quality checks cover nulls, uniqueness, ranges, freshness, and row counts
- [ ] Volume and latency justify the chosen engine: pandas, Polars, DuckDB, SQL warehouse, Spark/Flink
- [ ] Invalid records have a dead-letter/quarantine path with enough context to debug
- [ ] PII/governance classification is complete or delegated to `dev-security`/[§7](04-governance-queries.md#7-data-governance--pii)
- [ ] Output format and downstream contract are explicit
---
## 4. Data Quality
### Validation Checks
Run these after every pipeline step, not just at the end:
| Check | What It Validates | Example |
|-------|-------------------|---------|
| **Not null** | Required fields have values | `WHERE order_id IS NULL` → 0 rows |
| **Unique** | No duplicates on key columns | `COUNT(*) = COUNT(DISTINCT id)` |
| **Range** | Numeric values within bounds | `amount BETWEEN 0 AND 1,000,000` |
| **Categorical** | Values in allowed set | `status IN ('pending', 'active', 'closed')` |
| **Freshness** | Data is recent enough | `MAX(updated_at) > NOW() - INTERVAL '24 hours'` |
| **Row count** | No unexpected data loss or explosion | Within ±10% of previous run |
| **Referential** | Foreign keys point to existing records | `customer_id EXISTS IN customers` |
### Quality Tool Integration
Use a **layered quality strategy** — different tools at different pipeline stages:
| Stage | Tool | Purpose |
|-------|------|---------|
| **Ingest** | Great Expectations | Validate raw data against expectations before staging |
| **Transform** | dbt tests | Assert model-level quality (not_null, unique, relationships, custom SQL) |
| **Production** | Soda / Monte Carlo | Real-time monitoring, anomaly detection, SLA enforcement |
Validate data dimensions: completeness, uniqueness, range, format, referential integrity, freshness.
**Rule:** Run validation on every pipeline step — skipping "because the data looks fine" leads to silent downstream corruption.
### Data Contracts
For datasets shared between teams, define a contract:
A data contract must include:
- **name**, **owner**, **version**
- **schema**: column name, type, nullability, uniqueness, allowed values
- **SLA**: freshness threshold, minimum completeness percentage
- **consumers**: list of downstream teams/systems
Changes to a contracted schema require **versioning and consumer notification**.
### Migration & Backfill Sequencing
**Rule (DATA-MIGRATION-01):** Treat schema changes and data backfills as separate steps. Production evolution uses expand → backfill → dual read/write when needed → contract; require a dry run, idempotency proof, and reconciliation counts before declaring the migration complete.
---
## Data Change Review Checklist (DATA-REVIEW-01, DEFAULT)
When reviewing or implementing changes that affect data pipelines, schemas,
or data stores, check these domain-specific concerns:
### Schema Changes
- [ ] Is the change backward-compatible? (additive fields, optional columns)
- [ ] Are existing consumers updated or tolerant of the new schema?
- [ ] Is there a migration path for existing data?
- [ ] Are destructive changes (DROP, RENAME, type narrowing) reversible?
- [ ] Is the schema change tested with representative production-scale data?
### Pipeline Changes
- [ ] Are late/out-of-order events handled correctly?
- [ ] Is the pipeline idempotent for replays?
- [ ] Are timezone/DST transitions handled (especially for daily aggregations)?
- [ ] Is numeric precision preserved across transforms (float → decimal)?
- [ ] Are nondeterministic transforms (sampling, shuffling) reproducible with seeds?
### Quality Gates
- [ ] Is there a before/after reconciliation report (row counts, checksums)?
- [ ] Are null/missing value rates within expected bounds?
- [ ] Are downstream consumers notified of schema or semantic changes?
- [ ] Is the blast radius documented (which dashboards, models, exports break)?
### Backfill Safety
- [ ] Is the backfill cost estimated (compute, I/O, lock duration)?
- [ ] Is there a rollback plan for partial backfill failure?
- [ ] Are concurrent writes handled during backfill?
- [ ] Is the backfill window documented and approved?
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |