Robertjam954/multimodal_rag_application / .claude/skills/data-analysis-hygiene
data-analysis-hygiene
UnverifiedData-handling rules for this research repo. Use when loading datasets, merging/joining tables, running EDA or Table 1, building features, or handling any patient-level / PHI data.
Data Analysis Hygiene
When to Use
- Loading, merging, or transforming datasets
- Before any EDA, Table 1, or modeling step
- Anytime patient-level or PHI-adjacent data is involved
Rules
- Missingness audit first. Before EDA / Table 1 / modeling on any finalized dataset, produce a missingness audit (per-column % missing, patterns). The analyst decides the imputation strategy from it - do not silently impute.
- Skip existing outputs. If an extraction / feature / metric output already exists on disk, integrate from that cache instead of re-running the pipeline.
- Step-by-step merges. For multi-step joins, print per-step key-intersection counts and row-count diagnostics after each join. Take user-provided file paths literally.
- PHI safety. Never print, log, commit, or send patient-level data to any external service. Treat everything under
data/as sensitive. - iCloud-evicted files. Files under
~/Documentsmay be dataless (iCloud-evicted); materialize withdd if=<file> of=/dev/null bs=1mbefore a pandas read fails.
Missingness Audit Snippet
def missingness(df):
m = df.isna().mean().sort_values(ascending=False)
return (m[m > 0] * 100).round(1).rename("pct_missing").to_frame()
Integration
- Pair with [[publication-figures]] for any plots produced from audited data.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution. | Stars: 114k | Updated | Claude CodeCodex | Verified | Install |
| add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation. | Stars: 114k | Updated | Claude CodeCodex | Verified | Install |
| diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope. | Stars: 114k | Updated | Claude CodeCodex | Verified | Install |
| fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks. | Stars: 114k | Updated | Claude CodeCodex | Verified | Install |