aiopshwang/data-analysis-ml-agent-skills / skills/auditing-data-and-ground-truth
auditing-data-and-ground-truth
VerifiedAudit datasets, joins, labels, and ground truth before analysis or modeling. Use when data meaning, row grain, time semantics, source-of-truth reliability, or label construction may invalidate conclusions; not for general model evaluation after the evidence base is already trusted.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 3.7 KB |
| agents/openai.yaml | config | 250 B |
| assets/data-and-ground-truth-audit.md | markdown | 2.5 KB |
| references/audit-playbook.md | markdown | 4.0 KB |
references/audit-playbook.md
View on GitHub# Data and ground-truth audit playbook
Select checks according to the intended decision and failure risk. Do not run a large catalog without explaining what each check can falsify.
## Grain and identity
- Compare total rows, distinct candidate keys, and distinct real-world entities.
- Test key uniqueness within the time or source boundaries where uniqueness is claimed.
- Sample duplicates and classify exact replay, legitimate repeated event, correction, or unresolved collision.
- Check whether aggregations change the target population or create unequal observation periods.
High-risk signal: the modeled row does not map one-to-one to the decision unit.
## Joins and coverage
For each join, record expected cardinality and measure:
- unmatched keys on both sides;
- duplicate keys per side;
- row counts before and after;
- entity counts before and after;
- multiplication-factor distribution;
- whether missing matches cluster by source, time, or outcome.
High-risk signal: join survival or multiplication depends on the target or a protected or operationally important slice.
## Missingness and encoded absence
Separate unavailable, not applicable, not collected, failed extraction, suppressed, and true zero. Compare missingness by source, time, target, and population slice. Inspect sentinel values and default-filled fields before converting types.
High-risk signal: missingness is downstream of the outcome or a processing path unavailable in production.
## Time and historical truth
Distinguish:
- event time: when reality occurred;
- effective time: when a value became valid;
- record time: when the system stored it;
- availability time: when the model or analyst could retrieve it;
- update time: when a correction or backfill occurred.
Check negative durations, impossible sequences, time-zone changes, late arrivals, backfills, snapshot reconstruction, and use of current attributes for historical cases.
High-risk signal: a historically evaluated row contains a corrected or enriched value that was unavailable then.
## Ground-truth provenance
Trace label construction as a chain:
`real-world event -> source record -> extraction -> rule or annotation -> adjudication -> encoded label`
At each link, identify owner, timestamp, version, exclusions, and failure behavior. Measure coverage and unresolved labels. For human labels, measure agreement on a shared sample and examine disagreements by class and slice; agreement alone does not establish correctness. For delayed outcomes, establish a maturity cutoff and treat immature examples separately.
High-risk signals include using treatment or follow-up as proof of the original condition, treating missing follow-up as a negative, circularly validating a label against a derivative of itself, and allowing unresolved cases into a forced binary target.
## Independent reconciliation
Within the authorized access boundary, choose samples that can reveal systematic failure: random, edge cases, positives, negatives, missing labels, high-loss cases, and relevant sources or time periods. Compare against an approved operational record, original document, or independent reviewer that does not inherit the same transformation path. If that evidence requires new access or external sharing, specify the needed reconciliation and obtain authority before retrieving or transmitting records.
Record the sampling method, reviewer, evidence accessed, mismatch taxonomy, and estimated affected scope. Keep examples privacy-safe in public artifacts.
## Severity and readiness
Classify a finding by consequence, not cosmetic size:
- **critical:** can reverse the target, split, metric, or primary conclusion;
- **major:** materially biases a population, time period, or operational slice;
- **moderate:** limits interpretation or reproducibility but has a bounded workaround;
- **minor:** does not change the conclusion and is straightforward to correct.
An unresolved critical or major finding normally means `not ready` or `unknown`. A condition is acceptable only when it is explicit, enforced, and reflected in the claimed scope.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |