foryourhealth111-pixel/Vibe-Skills / bundled/skills/LQF_Machine_Learning_Expert_Guide
LQF_Machine_Learning_Expert_Guide
VerifiedLQF Machine Learning Expert Guide - Routed skill for ML/Statistical Modeling with Critical Discussion Mode. Triggers on: machine learning, modeling, prediction, training, classification, regression, clustering, deep learning, neural network, model evaluation, feature engineering, hyperparameter tuning, overfitting, underfitting, baseline, ablation study, critique my approach, review my model, is this a good idea, should I use, what's wrong with, evaluate my solution, challenge my assumptions, discuss my approach Engages in critical discussion with minimum 3 rounds of iterative refinement. Challenges both user proposals and own suggestions with fact-based critique. Demands evidence and baselines before accepting solutions.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 30.4 KB |
| references/expert-thinking.md | markdown | 12.7 KB |
| references/index.md | markdown | 2.7 KB |
| references/workflow-guide.md | markdown | 16.0 KB |
references/expert-thinking.md
View on GitHub# Expert Thinking Frameworks
This document provides a deep dive into the philosophical and conceptual differences between novice and expert machine learning practitioners.
## Part 1: The Fundamental Mindset Shift
### Core Differences: Novice vs Expert
| Dimension | Novice (The Alchemist) | Expert (The Architect) | Essential Difference |
|-----------|------------------------|------------------------|---------------------|
| **Core Focus** | **Tools & Metrics** - Wielding SOTA models like hammers, chasing leaderboard scores | **Essence & Assumptions** - Understanding data generation mechanisms, matching model assumptions to problem structure | Form vs Substance |
| **Thinking Path** | **Linear** - Find data → Import library → Tune parameters → Chase high accuracy | **Systemic** - Define problem → Establish baseline → Validate hypotheses → Iterate → Stress test | Process vs Closed Loop |
| **Data Perspective** | **Fuel** - More is better, feed it to the model uncritically, assume data is objective truth | **Ore** - Contains noise and bias, must understand generation process and collection flaws | Passive Acceptance vs Active Scrutiny |
| **Model Perspective** | **Black Box Worship** - Bigger and more complex is better (blindly using Transformers) | **Occam's Razor** - Prefer simple, interpretable models unless complexity brings proven gains | Stacking vs Fitting |
### The Three Core Thinking Transformations
To evolve from novice to expert, you must internalize these three mental models:
## A. Baseline Thinking - Combat Blind Confidence
**Novice Tendency:** Jump directly to building complex models
**Expert Practice:** Always ask the counterfactual - "What if we didn't use machine learning at all?"
### The Baseline Hierarchy
1. **Dummy Baseline (Statistical Blind Guess)**
- Classification: Predict the most frequent class
- Regression: Predict the mean value
- **Purpose:** If your complex model can't beat this, your features are useless
2. **Simple Heuristic (Domain Logic)**
- Time series: "Tomorrow's value = Today's value"
- Recommendation: "Recommend what user bought last time"
- **Purpose:** Captures domain knowledge without ML complexity
3. **Value Logic**
```
Lift = (Model Performance - Baseline Performance) / Baseline Performance
```
- Only when Lift is significantly > 0 does the complex model have business value
- If Lift < 10%, question whether the engineering cost is justified
### Baseline Thinking in Practice
```python
# Expert always establishes multiple baselines
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
import numpy as np
# Baseline 1: Dummy (statistical)
dummy = DummyClassifier(strategy='stratified')
dummy.fit(X_train, y_train)
dummy_score = dummy.score(X_test, y_test)
# Baseline 2: Simple model
simple = LogisticRegression(max_iter=1000)
simple.fit(X_train, y_train)
simple_score = simple.score(X_test, y_test)
# Baseline 3: Domain heuristic (example: predict based on single most important feature)
most_important_feature = X_train.columns[0] # Assume known from domain
heuristic_pred = (X_test[most_important_feature] > X_test[most_important_feature].median()).astype(int)
heuristic_score = (heuristic_pred == y_test).mean()
# Now build complex model
complex = RandomForestClassifier(n_estimators=100)
complex.fit(X_train, y_train)
complex_score = complex.score(X_test, y_test)
# Calculate lifts
print(f"Dummy baseline: {dummy_score:.3f}")
print(f"Simple model: {simple_score:.3f}")
print(f"Heuristic: {heuristic_score:.3f}")
print(f"Complex model: {complex_score:.3f}")
print(f"Lift over dummy: {(complex_score - dummy_score) / dummy_score * 100:.1f}%")
print(f"Lift over simple: {(complex_score - simple_score) / simple_score * 100:.1f}%")
# Decision: If lift over simple < 5%, use simple model (interpretability wins)
```
## B. Ablation Study - Combat Black Box Mysticism
**Novice Tendency:** "Adding this module improved performance, so it must be good"
**Expert Practice:** "Prove it's necessary by removing it"
### The Subtraction Principle
Don't prove that adding a component helps. Prove that **removing** it hurts.
**Why?** Because performance gains might come from:
- Increased parameter count (more capacity to overfit)
- Random initialization luck
- Hyperparameter interactions
### Ablation Study Protocol
```python
# Example: Testing if attention mechanism is necessary
# Full model with attention
class ModelWithAttention:
def __init__(self):
self.attention = AttentionLayer()
self.dense = DenseLayer()
def forward(self, x):
attended = self.attention(x)
return self.dense(attended)
# Ablated model (attention removed)
class ModelWithoutAttention:
def __init__(self):
self.dense = DenseLayer()
def forward(self, x):
return self.dense(x)
# Train both
full_model = ModelWithAttention()
full_score = train_and_evaluate(full_model) # 0.850
ablated_model = ModelWithoutAttention()
ablated_score = train_and_evaluate(ablated_model) # 0.848
# Analysis
performance_drop = (full_score - ablated_score) / full_score * 100
print(f"Performance drop without attention: {performance_drop:.2f}%")
# Decision rule: If drop < 2%, REMOVE the component (Occam's Razor)
if performance_drop < 2.0:
print("Attention adds complexity without sufficient benefit → REMOVE")
else:
print("Attention is justified → KEEP")
```
### Systematic Ablation
For a model with multiple components, ablate each one:
```python
components = ['attention', 'batch_norm', 'dropout', 'residual_connections']
baseline_score = 0.850
for component in components:
ablated_score = train_without_component(component)
drop = (baseline_score - ablated_score) / baseline_score * 100
print(f"Without {component}: {ablated_score:.3f} (drop: {drop:.1f}%)")
if drop < 1.0:
print(f" → {component} is NOT necessary, consider removing")
```
## C. Adversarial Thinking - Combat Confirmation Bias
**Novice Tendency:** Try to prove the model is "right" - celebrate high accuracy
**Expert Practice:** Try to prove the model is "wrong" - hunt for failure modes
### Error Analysis Over Metrics
**Novice:** "95% accuracy! Ship it!"
**Expert:** "What's happening in that 5% of errors? Let me examine every single one."
```python
# Expert error analysis workflow
from sklearn.metrics import confusion_matrix
import pandas as pd
# Get predictions
y_pred = model.predict(X_test)
# Find error cases
error_mask = (y_pred != y_test)
error_indices = np.where(error_mask)[0]
# Create error analysis dataframe
error_df = pd.DataFrame({
'true_label': y_test[error_indices],
'predicted_label': y_pred[error_indices],
'confidence': model.predict_proba(X_test[error_indices]).max(axis=1)
})
# Add original features for inspection
error_df = pd.concat([error_df, X_test.iloc[error_indices].reset_index(drop=True)], axis=1)
# Analyze patterns
print("Error Analysis:")
print(f"Total errors: {len(error_df)}")
print(f"High-confidence errors (>0.8): {(error_df['confidence'] > 0.8).sum()}")
print("\nError distribution by true label:")
print(error_df['true_label'].value_counts())
# Manual inspection of first 20 errors
print("\nFirst 20 error cases:")
print(error_df.head(20))
# Look for patterns:
# - Are certain classes systematically confused?
# - Are errors concentrated in specific feature ranges?
# - Are there mislabeled examples?
# - Are there missing features that would help?
```
### Stress Testing - Adversarial Inputs
```python
# Test 1: Missing values
X_test_missing = X_test.copy()
X_test_missing.iloc[:, 0] = np.nan # Set first feature to missing
stress_pred_1 = model.predict(X_test_missing)
print(f"Accuracy with missing values: {(stress_pred_1 == y_test).mean():.3f}")
# Test 2: Extreme values
X_test_extreme = X_test.copy()
X_test_extreme[:] = X_test.max() # Set all features to maximum
stress_pred_2 = model.predict(X_test_extreme)
print(f"Predictions with extreme values: {np.unique(stress_pred_2, return_counts=True)}")
# Test 3: Out-of-distribution
X_test_ood = X_test + np.random.normal(0, 5, X_test.shape) # Add large noise
stress_pred_3 = model.predict(X_test_ood)
print(f"Accuracy with OOD data: {(stress_pred_3 == y_test).mean():.3f}")
# Expert question: Does the model degrade gracefully or catastrophically?
```
## Part 2: The Four Mathematical Dimensions of ML
Experts understand ML from multiple mathematical perspectives:
### 1. Function Approximation (Optimization View)
**Definition:** Find optimal function f* in hypothesis space H that minimizes loss
```
f* = argmin_{f ∈ H} [ L(f(X), y) + λR(f) ]
```
**Expert Insight:** ML is not "perfect memorization" but **Bias-Variance Tradeoff**
- High bias (underfitting): Model too simple, can't learn patterns
- High variance (overfitting): Model too complex, memorizes noise
- Regularization term λR(f) encodes preference for simplicity
### 2. Probability Density Estimation (Statistical View)
**Definition:** Estimate true distribution P(X, y) from finite samples
**Expert Insight:** Embrace uncertainty, not determinism
- I.I.D. assumption: Training and test data come from same distribution
- Beware **Concept Drift**: Distribution changes over time
- Output confidence intervals P(y|X), not just point predictions ŷ
```python
# Expert outputs probability distributions
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier()
model.fit(X_train, y_train)
# Novice: Only point prediction
y_pred = model.predict(X_test)
# Expert: Probability distribution + confidence
y_proba = model.predict_proba(X_test)
confidence = y_proba.max(axis=1)
# Flag low-confidence predictions for human review
low_confidence_mask = confidence < 0.7
print(f"Low confidence predictions: {low_confidence_mask.sum()} / {len(y_test)}")
```
### 3. Data Compression (Information Theory View)
**Definition:** Learning = Compression. Good model = Minimum Description Length (MDL)
**Expert Insight:**
- Deep learning uses **Information Bottleneck**: Each layer discards noise (high-frequency info), keeps signal
- Overfitting = Compression failure: Model becomes a "hard drive copy" of data
### 4. Manifold Learning (Geometric View)
**Definition:** High-dimensional data (e.g., images) lies on low-dimensional manifolds. Deep learning untangles these manifolds through nonlinear transformations.
**Expert Insight:**
- Feature engineering = Manual space transformation
- Deep learning = Automatic space transformation
- Goal: Make data linearly separable in transformed space
## Part 3: Traditional ML vs Deep Learning
Experts know when to use each approach:
| Dimension | Traditional ML | Deep Learning |
|-----------|---------------|---------------|
| **Best For** | Tabular data (finance, healthcare, Kaggle) | Perceptual data (images, audio, text) |
| **Work Focus** | Feature engineering (70% of time) | Architecture design |
| **Interpretability** | White box (SHAP values explain decisions) | Black box (hard to explain) |
| **Data Requirements** | Works with small datasets (100s-1000s) | Needs large datasets (10,000s+) |
| **Metaphor** | Cooking: Human preps ingredients, machine cooks | Farming: Human provides seeds/environment, machine grows end-to-end |
### Traditional ML Expert Techniques
```python
# Feature explosion: Create interactions for linear models
from sklearn.preprocessing import PolynomialFeatures
# Linear models can't learn interactions automatically
poly = PolynomialFeatures(degree=2, interaction_only=True)
X_interactions = poly.fit_transform(X_train)
# Feature selection: Remove noise
from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import LassoCV
# L1 regularization for feature selection
lasso = LassoCV()
selector = SelectFromModel(lasso)
X_selected = selector.fit_transform(X_train, y_train)
print(f"Original features: {X_train.shape[1]}")
print(f"Selected features: {X_selected.shape[1]}")
```
### Deep Learning Expert Perspective
**Hierarchical Abstraction:** Edges → Parts → Objects
```python
# Expert understands each layer's role
# Layer 1: Low-level features (edges, textures)
# Layer 2: Mid-level features (parts, patterns)
# Layer 3: High-level features (objects, concepts)
# Expert designs architecture based on data structure
# - Images: CNN (translation invariance)
# - Sequences: RNN/Transformer (temporal dependencies)
# - Graphs: GNN (topological structure)
```
## Summary: The Expert Mindset
**Novice Goal:** "Get a high score"
**Expert Goal:** "Build an interpretable, robust system that respects causal laws"
The journey from novice to expert is a shift from:
- **Luck** → **Science**
- **Complexity worship** → **Essence pursuit**
- **Confirmation** → **Falsification**
Master these three thinking modes (Baseline, Ablation, Adversarial) and you'll think like an expert.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |