foryourhealth111-pixel/Vibe-Skills / bundled/skills/scientific-data-preprocessing
scientific-data-preprocessing
Verified⚠️ CRITICAL USER EXPERIENCE-BASED SKILL - ALWAYS CONSULT BEFORE DATA PREPROCESSING ⚠️ Prevents catastrophic errors (88.9% error rate in V1.0 case study) through multi-level feature analysis, data leakage detection, and semantic validation. MANDATORY for: data preprocessing, feature engineering, standardization, normalization, interpolation, missing value handling, feature selection, or ANY data transformation task. Covers grouped time-series, cross-sectional, panel data. Detects: time travel leakage, causal inversion, ID misuse, semantic-numeric fallacies, distribution blindness. User's hard-won lessons from real project failures.
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 8.1 KB |
| SKILL.md | skill md | 19.5 KB |
| UPDATE_LOG_V1.1.md | markdown | 12.8 KB |
| references/ai-common-pitfalls.md | markdown | 22.7 KB |
| references/decision-trees.md | markdown | 13.9 KB |
| references/error-case-studies.md | markdown | 12.4 KB |
| references/index.md | markdown | 1.8 KB |
| references/validation-checklist.md | markdown | 13.3 KB |
references/validation-checklist.md
View on GitHub# Validation Checklist for Scientific Data Preprocessing
Use this checklist before and after preprocessing to ensure data integrity.
---
## Pre-Processing Validation
### ✅ 1. Data Structure Understanding
- [ ] Confirmed data has natural groupings (match_id, patient_id, session_id, etc.)
- [ ] Identified group column name: `________________`
- [ ] Number of groups: `________________`
- [ ] Average observations per group: `________________`
- [ ] Data is time-ordered: Yes / No
- [ ] Time column name (if applicable): `________________`
### ✅ 2. Feature Classification
For each numeric feature, classify:
| Feature | Type | Action | Notes |
|---------|------|--------|-------|
| _______ | Binary/Categorical/Continuous | Keep/One-hot/Standardize | |
| _______ | Binary/Categorical/Continuous | Keep/One-hot/Standardize | |
| _______ | Binary/Categorical/Continuous | Keep/One-hot/Standardize | |
**Classification code:**
```python
for col in df.select_dtypes(include=['int64', 'float64']).columns:
n_unique = df[col].nunique()
print(f"{col}: {n_unique} unique values")
if n_unique == 2:
print(f" → Binary: Keep as-is")
elif n_unique <= 10:
print(f" → Categorical or ordinal: One-hot encode or keep")
else:
print(f" → Continuous: Standardize")
```
### ✅ 3. Data Quality Checks
- [ ] **Physical range validation**: All continuous features within reasonable bounds
```python
# Example
assert df['speed_mph'].max() < 170, "Speed exceeds world record"
assert df['distance_meters'].min() >= 0, "Negative distance"
```
- [ ] **Unit consistency**: Values in expected units (not km/h when expecting mph, etc.)
```python
# Check median is in expected range
print(f"Speed median: {df['speed_mph'].median()}") # Should be ~110-120 for tennis
```
- [ ] **No extreme outliers**: Check for data errors (>5σ from mean)
```python
for col in continuous_features:
mean = df[col].mean()
std = df[col].std()
extreme = df[abs(df[col] - mean) > 5 * std]
if len(extreme) > 0:
print(f"⚠️ {col}: {len(extreme)} extreme outliers")
```
- [ ] **Missing value pattern**: Is missingness random or systematic?
```python
# Check by group
missing_by_group = df.groupby('group_col').apply(
lambda g: g.isna().sum() / len(g) * 100
)
print(missing_by_group.describe())
```
### ✅ 4. Analysis Goal Definition
- [ ] Goal is **within-group (relative)** comparison → Use within-group standardization
- Example: "Was this point intense FOR THIS MATCH?"
- [ ] Goal is **cross-group (absolute)** comparison → Use global standardization
- Example: "Which matches had the most intense points overall?"
- [ ] Goal is **momentum/trend detection** → Use sliding window features
- Window size: `________________`
---
## During Processing Validation
### ✅ 5. Interpolation Validation
- [ ] **Interpolation scope**: Within-group only (not cross-group)
```python
# Verify: interpolated values within observed group range
for group_id in df['group_col'].unique():
group_data = df[df['group_col'] == group_id]
obs_min = group_data['feature'].quantile(0.01)
obs_max = group_data['feature'].quantile(0.99)
# Check interpolated values
suspicious = group_data[
(group_data['feature'] < obs_min * 0.8) |
(group_data['feature'] > obs_max * 1.2)
]
assert len(suspicious) == 0, f"Group {group_id}: suspicious interpolation"
```
- [ ] **No extrapolation**: Interpolated values between observed points, not beyond
### ✅ 6. Standardization Validation
- [ ] **Within-group standardization** (if applicable):
```python
# Each group should have mean≈0, std≈1
group_stats = df.groupby('group_col')['feature_std'].agg(['mean', 'std'])
assert (group_stats['mean'].abs() < 0.1).all(), "Means not centered per group"
assert (group_stats['std'].between(0.9, 1.1)).all(), "Stds not scaled per group"
```
- [ ] **Global standardization** (if applicable):
```python
# Overall should have mean≈0, std≈1
assert abs(df['feature_std'].mean()) < 0.01, "Mean not centered globally"
assert abs(df['feature_std'].std() - 1.0) < 0.01, "Std not scaled globally"
```
### ✅ 7. Feature Type Preservation
- [ ] **Binary features unchanged**:
```python
for col in binary_features:
assert df[col].isin([0, 1]).all(), f"{col} not binary!"
```
- [ ] **Categorical features one-hot encoded**:
```python
# Check for new one-hot columns
original_cats = ['server', 'serve_number']
for cat in original_cats:
onehot_cols = [c for c in df.columns if c.startswith(f"{cat}_")]
assert len(onehot_cols) > 0, f"{cat} not one-hot encoded"
# Exactly one should be 1 per row
assert (df[onehot_cols].sum(axis=1) == 1).all(), f"{cat} one-hot sum != 1"
```
---
## Post-Processing Validation
### ✅ 8. Output Data Quality
- [ ] **No missing values** (after interpolation):
```python
assert df[continuous_features].isna().sum().sum() == 0, "Still have missing values"
```
- [ ] **Column count**:
- Original columns: `________________`
- New columns added: `________________`
- Expected total: `________________`
- Actual total: `________________`
- [ ] **Row count unchanged**:
```python
assert len(df) == original_row_count, "Rows were dropped!"
```
### ✅ 9. Feature Distribution Sanity
- [ ] **Continuous features have reasonable distributions**:
```python
for col in continuous_features:
std_col = f"{col}_std"
if std_col in df.columns:
# Standardized should be roughly normal(-3, 3)
assert df[std_col].min() > -10, f"{col}: extreme negative z-score"
assert df[std_col].max() < 10, f"{col}: extreme positive z-score"
# Should have values across range
assert df[std_col].std() > 0.5, f"{col}: standardized but no variance"
```
- [ ] **Binary features still binary**:
```python
for col in binary_features:
unique_vals = df[col].unique()
assert set(unique_vals).issubset({0, 1}), f"{col} not binary anymore!"
```
### ✅ 10. Feature Engineering Validation
- [ ] **Sliding window features vary** (not monotonic):
```python
for col in sliding_window_features:
# Should have substantial variation
std = df.groupby('group_col')[col].std().mean()
assert std > 0.1, f"{col}: sliding window not varying"
```
- [ ] **Cumulative features monotonic** (if any):
```python
for col in cumulative_features:
# Diffs should be non-negative
diffs = df.groupby('group_col')[col].diff()
assert (diffs[diffs.notna()] >= 0).all(), f"{col}: cumulative not monotonic"
```
### ✅ 11. Group Boundary Validation
- [ ] **No cross-group leakage**:
```python
# For sliding windows, first N values in each group should use min_periods
for group_id in df['group_col'].unique():
group_data = df[df['group_col'] == group_id]
# First value should exist (min_periods=1)
first_val = group_data.iloc[0]['rolling_feature']
assert not pd.isna(first_val), f"Group {group_id}: first rolling value is NaN"
# First value should NOT equal last value from previous group
# (would indicate cross-group contamination)
```
---
## Common Error Detection
### ⚠️ Error Pattern 1: Binary Variable Transformation
```python
# Detect if binary features were incorrectly transformed
for col in ['is_ace', 'is_winner', 'is_error']:
if col in df.columns:
unique_vals = df[col].unique()
if not set(unique_vals).issubset({0, 1}):
print(f"❌ ERROR: {col} was transformed!")
print(f" Values: {unique_vals}")
print(f" Expected: [0, 1]")
```
### ⚠️ Error Pattern 2: Cross-Group Standardization
```python
# Detect if standardization was done cross-group
for std_col in [c for c in df.columns if c.endswith('_std')]:
group_means = df.groupby('group_col')[std_col].mean()
if (group_means.abs() < 0.1).all():
print(f"✅ {std_col}: Within-group standardization")
else:
print(f"❌ {std_col}: Cross-group standardization detected!")
print(f" Group means range: [{group_means.min():.2f}, {group_means.max():.2f}]")
print(f" Expected: All ≈ 0")
```
### ⚠️ Error Pattern 3: Cross-Group Interpolation
```python
# Detect suspicious interpolated values (outside group's observed range)
for group_id in df['group_col'].unique():
group_data = df[df['group_col'] == group_id]
for col in continuous_features:
# Original observed range
obs_min = group_data[col].quantile(0.05)
obs_max = group_data[col].quantile(0.95)
# Check for values well outside range
suspicious = group_data[
(group_data[col] < obs_min * 0.7) |
(group_data[col] > obs_max * 1.3)
]
if len(suspicious) > 0:
print(f"⚠️ Group {group_id}, {col}:")
print(f" Observed range: [{obs_min:.2f}, {obs_max:.2f}]")
print(f" Suspicious values: {suspicious[col].values}")
```
---
## Final Sign-Off Checklist
Before using preprocessed data:
- [ ] All binary features are still 0/1
- [ ] All categorical features are one-hot encoded
- [ ] Continuous features standardized appropriately (within-group or global)
- [ ] No missing values remain (or documented and acceptable)
- [ ] No cross-group contamination detected
- [ ] Sliding window features show variation (not monotonic)
- [ ] Data quality checks passed (no extreme outliers, physical ranges OK)
- [ ] Row count unchanged from original
- [ ] All processing steps documented
- [ ] Sample validation: Manually check a few groups end-to-end
---
## Automated Validation Script
Save and run this script:
```python
def validate_preprocessing(df_original, df_processed, group_col, config):
"""
Automated validation of preprocessing pipeline
Args:
df_original: Original raw data
df_processed: Processed data
group_col: Group column name
config: Dict with keys:
- binary_features: List of binary feature names
- categorical_features: List of categorical feature names
- continuous_features: List of continuous feature names
- standardization_scope: 'within_group' or 'global'
"""
print("=" * 60)
print("PREPROCESSING VALIDATION REPORT")
print("=" * 60)
# 1. Row count
assert len(df_original) == len(df_processed), "❌ Row count changed!"
print("✅ Row count preserved:", len(df_processed))
# 2. Binary features
print("\n--- Binary Features ---")
for col in config['binary_features']:
if col in df_processed.columns:
vals = df_processed[col].unique()
if set(vals).issubset({0, 1}):
print(f"✅ {col}: Still binary")
else:
print(f"❌ {col}: NOT BINARY! Values: {vals}")
# 3. Categorical features (one-hot)
print("\n--- Categorical Features ---")
for col in config['categorical_features']:
onehot_cols = [c for c in df_processed.columns if c.startswith(f"{col}_")]
if len(onehot_cols) > 0:
# Check sum
row_sums = df_processed[onehot_cols].sum(axis=1)
if (row_sums == 1).all():
print(f"✅ {col}: One-hot encoded ({len(onehot_cols)} categories)")
else:
print(f"❌ {col}: One-hot sum != 1")
else:
print(f"⚠️ {col}: No one-hot columns found")
# 4. Continuous features (standardization)
print("\n--- Continuous Features ---")
for col in config['continuous_features']:
std_col = f"{col}_std"
if std_col in df_processed.columns:
if config['standardization_scope'] == 'within_group':
# Check within-group
group_stats = df_processed.groupby(group_col)[std_col].agg(['mean', 'std'])
if (group_stats['mean'].abs() < 0.1).all() and \
(group_stats['std'].between(0.9, 1.1)).all():
print(f"✅ {col}: Within-group standardization correct")
else:
print(f"❌ {col}: Within-group standardization WRONG")
print(f" Group means: {group_stats['mean'].describe()}")
elif config['standardization_scope'] == 'global':
# Check global
mean = df_processed[std_col].mean()
std = df_processed[std_col].std()
if abs(mean) < 0.01 and abs(std - 1.0) < 0.01:
print(f"✅ {col}: Global standardization correct")
else:
print(f"❌ {col}: Global standardization WRONG (mean={mean:.3f}, std={std:.3f})")
# 5. Missing values
print("\n--- Missing Values ---")
missing_count = df_processed[config['continuous_features']].isna().sum().sum()
if missing_count == 0:
print(f"✅ No missing values")
else:
print(f"❌ {missing_count} missing values remain")
print("\n" + "=" * 60)
print("VALIDATION COMPLETE")
print("=" * 60)
# Usage
config = {
'binary_features': ['is_ace', 'is_winner'],
'categorical_features': ['server', 'serve_number'],
'continuous_features': ['distance_run', 'speed_mph'],
'standardization_scope': 'within_group' # or 'global'
}
validate_preprocessing(df_original, df_processed, 'match_id', config)
```
---
*Source: Validation best practices from scientific data preprocessing*
*Last updated: 2026-01-18*
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |