Skip to content

DAAF-Contribution-Community/daaf / .claude/skills/scikit-learn

scikit-learn

Verified

Machine learning: clustering, PCA/t-SNE/UMAP, classification, prediction regression (Ridge/Lasso/ensemble), cross-validation, Pipelines. For unsupervised analysis, classification, or prediction. For econometric regression use pyfixest/statsmodels. R equivalent: tidymodels (use when execution language is R).

references/classification.md

View on GitHub
# Classification

Supervised classification with scikit-learn: LogisticRegression, DecisionTreeClassifier, RandomForestClassifier, GradientBoostingClassifier, HistGradientBoostingClassifier, SVC, and KNeighborsClassifier. For model evaluation metrics (accuracy, F1, ROC-AUC, cross-validation), see `evaluation-supervised.md`.

## Classifier Comparison

| Classifier | Interpretable? | Handles Nonlinear? | Scales to Large N? | Key Strength |
|------------|---------------|--------------------|--------------------|--------------|
| LogisticRegression | Yes (penalized coefficients) | No (linear boundary) | Yes | Baseline; regularization |
| DecisionTree | Yes (tree viz) | Yes | Moderate | Interpretable rules |
| RandomForest | Moderate (importances) | Yes | Yes | Robust; low tuning |
| GradientBoosting | Moderate (importances) | Yes | Moderate | High accuracy |
| HistGradientBoosting | Moderate (importances) | Yes | Yes (fast) | Large data; native NaN |
| SVC | No (kernel) | Yes (kernel) | No (slow >10K) | Small data; complex boundaries |
| KNeighbors | No | Yes | No (slow at predict) | Simple; no training |

## LogisticRegression

Linear classifier with probabilistic output. The standard baseline for classification.

```python
from sklearn.linear_model import LogisticRegression

model = LogisticRegression(
    C=1.0,                   # Inverse regularization strength (smaller = stronger)
    l1_ratio=0.0,            # Penalty type (replaces penalty=, deprecated in 1.8): 0 = L2, 1 = L1, between = elastic net
    solver="lbfgs",          # Solver: 'lbfgs' (default; L2 or none), 'saga' (for L1/elastic net)
    max_iter=1000,           # Increase if convergence warning
    class_weight=None,       # 'balanced' for imbalanced classes
    random_state=42
)
model.fit(X_train, y_train)

# --- Predictions ---
y_pred = model.predict(X_test)
y_proba = model.predict_proba(X_test)  # Class probabilities

# --- Coefficients (shrunk toward zero by the default penalty; see note below) ---
print(f"Coefficients: {model.coef_}")
print(f"Intercept: {model.intercept_}")
```

**Default coefficients are penalized.** With the defaults (`C=1.0`, L2 penalty), `coef_` is shrunk toward zero, and how much depends on each feature's scale. These are not the maximum-likelihood estimates that `statsmodels` or Stata `logit` report, and scikit-learn gives no standard errors. For prediction, tune `C` by cross-validation. For unpenalized estimates, use a very large `C`:

```python
model_mle = LogisticRegression(C=1e10, max_iter=1000).fit(X_train, y_train)   # effectively unpenalized
```

In 1.8.0 both documented unpenalized spellings warn: `penalty=None` raises a FutureWarning (deprecated in 1.8, to be removed in 1.10), and `C=np.inf` raises a UserWarning ("Setting penalty=None will ignore the C and l1_ratio parameters") that scikit-learn issue #33871 reports as spurious. With the default `tol=1e-4` the estimates can differ from `statsmodels` in the third significant digit; add `tol=1e-8` when they must match. For coefficient inference (SEs, p-values), use the `statsmodels` skill.

### Solver Selection

| Solver | Penalties Supported | Multiclass | Notes |
|--------|-------------------|------------|-------|
| `lbfgs` | l2, None | Yes (multinomial) | Default; good all-around |
| `saga` | l1, l2, elasticnet, None | Yes | Needed for l1/elasticnet |
| `liblinear` | l1, l2 | No (binary only; wrap in `OneVsRestClassifier`) | Fast for small datasets |

## DecisionTreeClassifier

Single decision tree. Highly interpretable but prone to overfitting.

```python
from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier(
    max_depth=5,                 # Limit tree depth (prevents overfitting)
    min_samples_leaf=10,         # Minimum samples in leaf node
    min_samples_split=20,        # Minimum samples to split a node
    class_weight=None,           # 'balanced' for imbalanced classes
    random_state=42
)
model.fit(X_train, y_train)

# --- Feature importance ---
importances = model.feature_importances_
for name, imp in sorted(zip(feature_names, importances), key=lambda x: -x[1]):
    print(f"{name}: {imp:.4f}")
```

## RandomForestClassifier

Ensemble of decision trees with bagging. Robust, low tuning overhead, good default choice.

```python
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(
    n_estimators=100,            # Number of trees (more = better, diminishing returns)
    max_depth=None,              # None = trees grow until pure (or min_samples_leaf)
    max_features="sqrt",         # Features per split: 'sqrt', 'log2', int, float
    min_samples_leaf=5,          # Minimum samples in leaf
    class_weight=None,           # 'balanced' for imbalanced classes
    n_jobs=-1,                   # Use all CPU cores
    random_state=42
)
model.fit(X_train, y_train)

# --- Predictions with probability ---
y_pred = model.predict(X_test)
y_proba = model.predict_proba(X_test)

# --- Feature importance ---
importances = model.feature_importances_

# --- Out-of-bag score (free validation estimate) ---
model_oob = RandomForestClassifier(n_estimators=100, oob_score=True, random_state=42)
model_oob.fit(X_train, y_train)
print(f"OOB accuracy: {model_oob.oob_score_:.3f}")
```

## GradientBoostingClassifier

Sequential ensemble that builds trees to correct previous errors. Often highest accuracy.

```python
from sklearn.ensemble import GradientBoostingClassifier

model = GradientBoostingClassifier(
    n_estimators=200,            # Number of boosting stages
    learning_rate=0.1,           # Shrinkage (lower = more stages needed, better generalization)
    max_depth=3,                 # Depth per tree (typically 3-8 for boosting)
    min_samples_leaf=10,
    subsample=0.8,               # Fraction of samples per tree (stochastic boosting)
    random_state=42
)
model.fit(X_train, y_train)
```

## HistGradientBoostingClassifier

Histogram-based gradient boosting. Much faster than GradientBoostingClassifier on large datasets. Handles missing values natively.

```python
from sklearn.ensemble import HistGradientBoostingClassifier

model = HistGradientBoostingClassifier(
    max_iter=200,                # Number of boosting iterations
    learning_rate=0.1,
    max_depth=None,              # None = unlimited (controlled by max_leaf_nodes)
    max_leaf_nodes=31,           # Max leaves per tree
    min_samples_leaf=20,
    random_state=42
)
model.fit(X_train, y_train)     # Handles NaN values natively (no imputation needed)
```

### GradientBoosting vs HistGradientBoosting

| Feature | GradientBoosting | HistGradientBoosting |
|---------|-----------------|---------------------|
| Speed | Slower | Much faster (histogram binning) |
| Large datasets | Slow on >10K rows | Handles millions of rows |
| Missing values | Requires imputation | Handles NaN natively |
| Categorical features | Requires encoding | Native (`categorical_features="from_dtype"` default since 1.6) |
| API | `n_estimators`, `max_depth` | `max_iter`, `max_leaf_nodes` |

## SVC (Support Vector Classifier)

Finds the maximum-margin decision boundary. Effective on small-to-medium datasets with complex boundaries.

```python
from sklearn.svm import SVC

model = SVC(
    kernel="rbf",                # 'linear', 'rbf', 'poly', 'sigmoid'
    C=1.0,                       # Regularization (larger = less regularization)
    gamma="scale",               # Kernel coefficient: 'scale', 'auto', or float
    probability=True,            # Enable predict_proba (slower training)
    class_weight=None,           # 'balanced' for imbalanced classes
    random_state=42
)
model.fit(X_train_scaled, y_train)   # Scale features first (SVM is distance-based)
y_pred = model.predict(X_test_scaled)
```

## KNeighborsClassifier

Classifies based on majority vote of k nearest neighbors. No training phase.

```python
from sklearn.neighbors import KNeighborsClassifier

model = KNeighborsClassifier(
    n_neighbors=5,               # Number of neighbors
    weights="uniform",           # 'uniform' or 'distance' (inverse-distance weighting)
    metric="minkowski",          # Distance metric
    p=2,                         # Minkowski p (2=Euclidean, 1=Manhattan)
    n_jobs=-1
)
model.fit(X_train_scaled, y_train)   # Scale features first (distance-based)
y_pred = model.predict(X_test_scaled)
```

## Common Patterns

### Handling Class Imbalance

```python
# --- Option 1: class_weight parameter ---
model = RandomForestClassifier(class_weight="balanced", random_state=42)

# --- Option 2: class_weight with custom weights ---
model = LogisticRegression(class_weight={0: 1, 1: 10}, random_state=42)

# --- Option 3: Adjust decision threshold ---
y_proba = model.predict_proba(X_test)[:, 1]
y_pred_custom = (y_proba >= 0.3).astype(int)  # Lower threshold for minority class
```

### Multi-Class Classification

```python
# --- Most classifiers handle multi-class natively ---
model = LogisticRegression(max_iter=1000)   # multinomial by default; multi_class= was removed (TypeError in 1.8)
model.fit(X_train, y_train)  # y has >2 classes

# --- To force one-vs-rest (SVC is natively one-vs-one; liblinear LogisticRegression is binary-only) ---
from sklearn.multiclass import OneVsRestClassifier
model = OneVsRestClassifier(SVC(kernel="rbf", probability=True))
```

### Quick Model Comparison

```python
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

classifiers = {
    "LogisticRegression": LogisticRegression(max_iter=1000),
    "RandomForest": RandomForestClassifier(n_estimators=100, random_state=42),
    "HistGradientBoosting": HistGradientBoostingClassifier(random_state=42),
}

for name, clf in classifiers.items():
    pipe = make_pipeline(StandardScaler(), clf)
    scores = cross_val_score(pipe, X, y, cv=5, scoring="accuracy")
    print(f"{name}: {scores.mean():.3f} +/- {scores.std():.3f}")
```

## Beyond HistGradientBoosting: LightGBM and XGBoost

For use cases where scikit-learn's HistGradientBoostingClassifier is not sufficient, LightGBM and XGBoost provide additional capabilities. Both follow the scikit-learn estimator API (`fit`/`predict`/`predict_proba`).

```python
# LightGBM (available in DAAF Dockerfile)
import lightgbm as lgb
model = lgb.LGBMClassifier(
    n_estimators=100,
    learning_rate=0.1,
    num_leaves=31,
    random_state=42,
)
model.fit(X_train, y_train)

# XGBoost (not pre-installed; runtime installs blocked — see CLAUDE.md § Runtime Package Installation)
import xgboost as xgb
model = xgb.XGBClassifier(
    n_estimators=100,
    learning_rate=0.1,
    max_depth=6,
    random_state=42,
)
model.fit(X_train, y_train)
```

### Comparison

| Feature | HistGradientBoosting | LightGBM | XGBoost |
|---------|---------------------|----------|---------|
| In Dockerfile | Yes | Yes | No (optional) |
| SHAP TreeExplainer | Native (fast) | Native (fast) | Native (fast) |
| Custom loss functions | No | Yes | Yes |
| Categorical features | Native | Native | Native support via `enable_categorical=True` (1.5+); otherwise requires encoding |
| GPU support | No | Yes | Yes |

> **Guidance:** For most DAAF use cases, HistGradientBoosting is sufficient. Use XGBoost for custom loss functions.
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 321kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall