borghei/Claude-Skills / engineering/data-quality-auditor
data-quality-auditor
VerifiedAudit data quality across pipelines, warehouses, and stores. Use when designing a DQ program, defining DQ dimensions, building rule-based checks, detecting schema drift, monitoring freshness SLAs, or responding to a DQ incident.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 11.5 KB |
| assets/baseline_schema.json | config | 463 B |
| assets/checks.yaml | config | 883 B |
| assets/current_schema_drifted.json | config | 540 B |
| assets/sample_orders.json | config | 588 B |
| assets/sample_orders_flawed.json | config | 576 B |
| references/data-quality-dimensions.md | markdown | 13.5 KB |
| references/dq-anti-patterns.md | markdown | 1.3 KB |
| references/dq-check-catalog.md | markdown | 12.5 KB |
| references/dq-incident-response.md | markdown | 12.5 KB |
| references/dq-maturity-model.md | markdown | 1.3 KB |
| references/dq-workflows.md | markdown | 3.0 KB |
| scripts/dq_check_runner.py | script | 16.0 KB |
| scripts/freshness_monitor.py | script | 5.6 KB |
| scripts/schema_drift_detector.py | script | 9.7 KB |
references/dq-workflows.md
View on GitHub# End-to-End DQ Workflows
Read this when executing a concrete DQ task: adding DQ to a new dataset, detecting/responding to schema drift, monitoring freshness SLAs, auditing existing pipelines, or hardening after an incident.
## Workflow: Add DQ to a new dataset
1. **Profile** the data — export a sample to CSV/JSON, then `scripts/dq_check_runner.py --profile --data mytable.json` reports row count, null rate, distinct count, a few sample values and min/max/mean for numeric columns.
2. **Pick check thresholds** based on the profile and product knowledge.
3. **Write the checks** — as a `checks.yaml` for `dq_check_runner.py --data mytable.json --checks checks.yaml` (see `assets/checks.yaml`), or in your tool of choice (Great Expectations / dbt tests / Soda / custom SQL).
4. **Wire into pipeline** — checks run on every load. Failures fail the pipeline (with alerting), don't silently produce bad data downstream.
5. **Document the DQ contract** in the data catalog: what does this table guarantee?
## Workflow: Detect and respond to schema drift
1. **Snapshot baseline schema** — `scripts/schema_drift_detector.py --snapshot mytable.json --output baseline.json` infers one from exported rows; or export `information_schema.columns` as `{"columns": [{"name", "type", "nullable", "ordinal"}]}`.
2. **Schedule the detector** — `scripts/schema_drift_detector.py --baseline baseline.json --current current.json --fail-on warning` before every consumer pipeline (or hourly); exit 1 means drift at or above the gate.
3. **On drift detection** — alert + block pipeline; investigate whether the change was intentional (upstream renamed a column) or accidental (data corruption).
4. **If intentional**: update consumer pipelines + baseline. If accidental: roll back upstream or fix.
## Workflow: Monitor freshness SLAs
1. **Define SLA per table** — e.g., `events_table` must be updated within 1 hour; `daily_revenue` within 24 hours.
2. **Wire freshness checks** — `scripts/freshness_monitor.py --data events.json --column event_time --max-age-min 60 --table-name events_table` (exit 1 on breach).
3. **Alert on SLA breach** — via pager/Slack.
4. **Triage** — pipeline failure? upstream delay? logic bug?
## Workflow: Audit existing pipelines
1. **Inventory** all production tables (typically from data catalog).
2. **Score per table** — for each, what % of dimensions have at least one check? There is no catalog scanner here: list each table's existing checks against the six dimensions by hand (or from your dbt/GX manifest), then run `dq_check_runner.py` on a sample of each table to confirm the checks pass today.
3. **Prioritize** — start with customer-facing + Sev1-impact tables.
4. **Schedule remediation** — add missing checks per priority.
## Workflow: Post-incident DQ improvement
1. After a DQ incident, ask: would an automated check have caught this?
2. If yes: add it. The check becomes regression prevention.
3. Document the incident → check mapping. Over time, DQ check inventory reflects organizational pain history.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |