Skip to content

DAAF-Contribution-Community/daaf / .claude/skills/scikit-learn

scikit-learn

Verified

Machine learning: clustering, PCA/t-SNE/UMAP, classification, prediction regression (Ridge/Lasso/ensemble), cross-validation, Pipelines. For unsupervised analysis, classification, or prediction. For econometric regression use pyfixest/statsmodels. R equivalent: tidymodels (use when execution language is R).

references/manifold.md

View on GitHub
# Manifold Learning

t-SNE, UMAP (via umap-learn), Isomap, LocallyLinearEmbedding, MDS, and SpectralEmbedding for nonlinear dimension reduction and visualization. For methodology guidance (interpreting embeddings, parameter sensitivity, visualization-only warnings), see `exploratory-unsupervised.md` in the `data-scientist` skill; its References section gives full citations for works cited here.

**Key principle:** These methods are primarily **visualization tools**, not analysis tools. Cluster sizes, between-cluster distances, and apparent structure in 2D embeddings should not be interpreted as real properties of the high-dimensional data without formal validation.

## Method Comparison

| Method | Preserves | Speed | Key Parameter | Output Dims |
|--------|-----------|-------|---------------|-------------|
| t-SNE | Local neighborhoods | Slow (>10K rows) | `perplexity` (5-50) | 2-3 (Barnes-Hut default) |
| UMAP | Local + some global | Fast | `n_neighbors`, `min_dist` | Any |
| Isomap | Geodesic distances | Moderate | `n_neighbors` | Any |
| LLE | Local linear structure | Moderate | `n_neighbors` | Any |
| MDS | Pairwise distances | Slow | `metric` (True/False) | Any |
| SpectralEmbedding | Graph structure | Moderate | `n_neighbors`, `affinity` | Any |

## t-SNE

t-Distributed Stochastic Neighbor Embedding. Excellent for 2D visualization of high-dimensional data. Preserves local neighborhood structure.

```python
from sklearn.manifold import TSNE

# --- Fit t-SNE ---
tsne = TSNE(
    n_components=2,          # Almost always 2 (or 3)
    perplexity=30,           # Balance local/global structure (5-50)
    max_iter=1000,           # Iterations (increase if not converged)
    learning_rate="auto",    # Auto-scales with dataset size (default in 1.2+)
    init="pca",              # Initialize with PCA (more stable than random)
    random_state=42
)
X_embedded = tsne.fit_transform(X_scaled)

# --- Check convergence ---
print(f"KL divergence: {tsne.kl_divergence_:.4f}")
# Lower is better; if decreasing with more iterations, increase max_iter
```

### Perplexity Sensitivity

Perplexity controls the effective number of nearest neighbors. Different values can produce qualitatively different plots.

```python
# --- Run multiple perplexity values to assess sensitivity ---
for perp in [5, 15, 30, 50]:
    tsne = TSNE(n_components=2, perplexity=perp, random_state=42)
    embedding = tsne.fit_transform(X_scaled)
    print(f"Perplexity {perp}: KL divergence = {tsne.kl_divergence_:.4f}")
    # Compare plots across perplexity values
```

### What t-SNE Cannot Tell You

- Cluster sizes in the plot are not meaningful (algorithm equalizes apparent density)
- Distances between well-separated clusters may not be meaningful (only within-cluster neighborhoods are preserved)
- Apparent clusters in t-SNE can emerge from random noise at low perplexity
- Different random seeds produce different layouts

## UMAP (via umap-learn)

Uniform Manifold Approximation and Projection. Often preferred over t-SNE: faster, arguably preserves more global structure (McInnes, Healy & Melville, 2018), and supports arbitrary output dimensions.

> `umap-learn` (0.5.11) is **pre-installed** in the DAAF container — no install
> needed; just `import umap`. Runtime installs are blocked in any case (see
> CLAUDE.md § Runtime Package Installation).

```python
import umap

# --- Fit UMAP ---
reducer = umap.UMAP(
    n_components=2,          # Output dimensions (can be >3 for general reduction)
    n_neighbors=15,          # Local neighborhood size (docs: 2-100; 5-50 common)
    min_dist=0.1,            # Minimum distance between points in embedding (set relative to spread)
    metric="euclidean",      # Distance metric
    random_state=42
)
X_embedded = reducer.fit_transform(X_scaled)

# --- Transform new data (unlike t-SNE, UMAP supports this) ---
X_new_embedded = reducer.transform(X_new_scaled)
```

### Key Parameters

| Parameter | Effect of Increasing | Typical Range |
|-----------|---------------------|---------------|
| `n_neighbors` | More global structure preserved; less fine detail | 2-100 (umap-learn docs); 5-50 common in practice |
| `min_dist` | Points spread out more; less tight clusters | Rule of thumb; set relative to `spread` |

### UMAP vs t-SNE

| Feature | UMAP | t-SNE |
|---------|------|-------|
| Speed | Much faster (especially >10K rows) | Slow on large data |
| Global structure | Arguably better preserved, partly owing to spectral initialization (McInnes et al., 2018) | Not reliably preserved; distances between well-separated clusters may mean nothing (Wattenberg, Viégas & Johnson, 2016) |
| Reproducibility | Differs between runs; more stable than t-SNE under subsampling (McInnes et al., 2018) | Layout can change across seeds; most examples keep the same global shape, but some data sets differ markedly (Wattenberg et al., 2016) |
| Transform new data | Yes (`reducer.transform()`) | No (must re-fit) |
| Output dimensions | Any number | Practically limited to 2-3 |
| Implementation | `umap-learn` (separate library) | `sklearn.manifold.TSNE` |

## Isomap

Estimates geodesic distances (shortest path along data manifold) and applies classical MDS.

```python
from sklearn.manifold import Isomap

# --- Fit Isomap ---
isomap = Isomap(
    n_components=2,
    n_neighbors=10           # Size of local neighborhood
)
X_embedded = isomap.fit_transform(X_scaled)

# --- Reconstruction error ---
print(f"Reconstruction error: {isomap.reconstruction_error():.4f}")

# --- Transform new data ---
X_new_embedded = isomap.transform(X_new_scaled)
```

## LocallyLinearEmbedding (LLE)

Preserves local linear relationships between neighbors.

```python
from sklearn.manifold import LocallyLinearEmbedding

# --- Fit LLE ---
lle = LocallyLinearEmbedding(
    n_components=2,
    n_neighbors=10,
    method="standard",       # 'standard', 'modified', 'hessian', 'ltsa'
    random_state=42
)
X_embedded = lle.fit_transform(X_scaled)
```

## MDS

Multidimensional Scaling. Preserves pairwise distances in the low-dimensional embedding.

```python
from sklearn.manifold import MDS

# --- Metric MDS (preserves absolute distances) ---
mds = MDS(
    n_components=2,
    metric=True,             # True = metric MDS, False = non-metric
    n_init=4,                # Number of random initializations
    max_iter=300,
    random_state=42
)
X_embedded = mds.fit_transform(X_scaled)

# --- Stress (lower = better fit) ---
print(f"Stress: {mds.stress_:.2f}")
```

## SpectralEmbedding

Graph-based embedding using the Laplacian eigenvectors of the affinity graph.

```python
from sklearn.manifold import SpectralEmbedding

# --- Fit SpectralEmbedding ---
se = SpectralEmbedding(
    n_components=2,
    affinity="nearest_neighbors",  # or 'rbf'
    n_neighbors=10,
    random_state=42
)
X_embedded = se.fit_transform(X_scaled)
```

## Best Practice: Run Multiple Methods and Parameters

```python
# --- Compare embeddings to assess robustness ---
# If structure appears consistently across methods and parameters,
# it is more likely to reflect genuine data structure.
# If structure changes dramatically, interpret with extreme caution.
```
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall