ilandahan/AID / skills/role-data-scientist
role-data-scientist
UnverifiedData Scientist role within the AID methodology. Activate when working on: EDA, feasibility assessment, experiment planning, ML architecture, model development & validation, or production deployment & monitoring. Stack-agnostic — ask the user for their stack at the start of any new project.
| File | Kind | Size |
|---|---|---|
| SKILL.extended.md | markdown | 18.2 KB |
| SKILL.md | skill md | 10.7 KB |
| references/bias-audit-checklist.md | markdown | 6.1 KB |
| references/ml-pipeline-checklist.md | markdown | 5.6 KB |
| references/model-card-template.md | markdown | 5.4 KB |
| references/monitoring-setup-guide.md | markdown | 7.3 KB |
| references/prompt-testing-patterns.md | markdown | 6.6 KB |
| references/rag-architecture-guide.md | markdown | 6.4 KB |
references/bias-audit-checklist.md
View on GitHub# Bias Audit Checklist
Pre-deployment fairness audit for ML models.
**Use in**: Phase 6 — Deploy to Production & Monitoring
> **Iron Rule #7: No model without bias audit**
---
## When to Run This Audit
- Before every production deployment
- After retraining on new data
- After significant changes to features or preprocessing
- On a scheduled quarterly basis in production
---
## Step 1 — Protected Attributes Inventory
- [ ] List all protected attributes relevant to your domain and jurisdiction
- [ ] Identify which attributes are in the training data (directly or as proxies)
- [ ] Document proxy variables — features that correlate with protected attributes
**Common protected attributes and their proxies**:
| Attribute | Proxies to Watch |
|-----------|-----------------|
| Race / Ethnicity | Name, zip code, language, neighborhood |
| Gender | Name, occupation, job title |
| Age | Graduation year, years of experience |
| Disability | Medical codes, accommodation flags |
| Religion | Name, dietary preferences, holidays |
| Socioeconomic status | Zip code, education level, device type |
| Geographic location | IP address, language, timezone |
---
## Step 2 — Data Bias Assessment
- [ ] Check training data distribution across protected groups
- [ ] Compare group sizes vs. population representation — flag underrepresented groups
- [ ] Check for historical bias in labels (e.g., past hiring decisions encoding past discrimination)
- [ ] Check for sampling bias in data collection method
- [ ] Document known data gaps or limitations
**Distribution check**:
| Group | Training % | Population % | Gap | Action Required |
|-------|-----------|-------------|-----|-----------------|
| Group A | | | | |
| Group B | | | | |
| Group C | | | | |
---
## Step 3 — Fairness Metric Selection
No single metric captures all fairness concerns. Select metrics appropriate to your use case and document why.
| Metric | Definition | When to Use |
|--------|-----------|-------------|
| **Demographic Parity** | P(positive prediction) equal across groups | When selection rates should be equal |
| **Equal Opportunity** | True positive rate equal across groups | When equal benefit is the priority |
| **Equalized Odds** | TPR and FPR equal across groups | When both error types should be equal |
| **Predictive Parity** | Precision equal across groups | When prediction trust should be equal |
| **Calibration** | P(Y=1 | score=s) equal across groups | When scores should mean the same thing per group |
**Domain guide**:
- Hiring / lending → Demographic parity + equal opportunity
- Healthcare → Equal opportunity + calibration
- Criminal justice / risk scoring → Equalized odds + calibration
- Content recommendation → Demographic parity
---
## Step 4 — Slice-Based Evaluation
> Aggregate metrics hide disparities. Always evaluate on slices.
- [ ] Evaluate model performance on each protected group separately
- [ ] Evaluate intersectional groups (e.g., age AND gender, not just each alone)
- [ ] Flag any group whose metric differs by more than **10%** from the overall metric
- [ ] Document per-slice results in the model card
**Evaluation table**:
| Slice | Accuracy | TPR (Recall) | FPR | Precision | N Samples | Flag |
|-------|----------|-------------|-----|-----------|-----------|------|
| Overall | | | | | | |
| Group A | | | | | | |
| Group B | | | | | | |
| A ∩ Group X | | | | | | |
| B ∩ Group X | | | | | | |
**Flag if**: Any group metric differs > 10% from overall metric.
---
## Step 5 — Mitigation (If Bias Detected)
| Stage | Strategy | How |
|-------|----------|-----|
| **Pre-processing** | Resampling | Oversample underrepresented groups |
| **Pre-processing** | Reweighting | Higher loss weight for underrepresented groups |
| **In-processing** | Fairness constraints | Add fairness term to loss function |
| **In-processing** | Adversarial debiasing | Train adversary to predict protected attribute |
| **Post-processing** | Threshold adjustment | Different decision thresholds per group |
| **Post-processing** | Calibration | Calibrate scores independently per group |
Document which strategy was applied and its measured effect on both fairness and overall performance.
---
## Step 6 — Documentation
- [ ] All protected attributes considered are listed
- [ ] Fairness metrics selected with rationale documented
- [ ] Per-slice evaluation results recorded
- [ ] Any bias detected and mitigation applied are documented
- [ ] Limitations of this audit are noted (attributes not tested, data gaps)
- [ ] Results entered in the model card (`model-card-template.md` → Bias & Fairness section)
---
## Step 7 — Sign-Off
| Role | Responsibility | Signed | Date |
|------|---------------|--------|------|
| Data Scientist | Conducted audit and documented results | | |
| Tech Lead | Reviewed methodology and results | | |
| Product Manager | Accepted risk level and mitigation approach | | |
| Legal / Compliance | Confirmed regulatory compliance | | |
---
## Ongoing Monitoring
Bias audit is not one-time — bias can drift as data changes.
- [ ] Configure fairness metrics in the production monitoring dashboard
- [ ] Schedule quarterly bias reviews
- [ ] Re-audit when training data changes significantly
- [ ] Re-audit after every retraining run
- [ ] Track fairness metrics over time for degradation
**Reference**: `monitoring-setup-guide.md` — add fairness metrics to Model Performance signals
---
## Common Pitfalls
| Pitfall | Prevention |
|---------|------------|
| Only checking aggregate metrics | Always evaluate on per-group slices |
| Ignoring intersectional groups | Test combinations, not just individual attributes |
| One-time audit only | Schedule recurring audits |
| No proxy variable check | Actively search for proxy correlations before modeling |
| Choosing one fairness metric | Use multiple metrics — document trade-offs honestly |
| Audit without documentation | Results must appear in model card |
---
**Cross-references**:
- `SKILL.md` — Phase 6 behaviors and Iron Rule #7
- `model-card-template.md` — Where audit results are recorded
- `ml-pipeline-checklist.md` — Phase 6 deployment checklist
- `monitoring-setup-guide.md` — Ongoing fairness monitoring
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |