DANG-ai/SKILLER / skilllearnbench/skills/human_authored/dbscan-parameter-tuning/parallel-processing
parallel-processing
VerifiedParallel processing with joblib for grid search and batch computations. Use when speeding up computationally intensive tasks across multiple CPU cores.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 2.0 KB |
SKILL.md
View on GitHub---
name: parallel-processing
description: Parallel processing with joblib for grid search and batch computations. Use when speeding up computationally intensive tasks across multiple CPU cores.
---
# Parallel Processing with joblib
Speed up computationally intensive tasks by distributing work across multiple CPU cores.
## Basic Usage
```python
from joblib import Parallel, delayed
def process_item(x):
"""Process a single item."""
return x ** 2
# Sequential
results = [process_item(x) for x in range(100)]
# Parallel (uses all available cores)
results = Parallel(n_jobs=-1)(
delayed(process_item)(x) for x in range(100)
)
```
## Key Parameters
- **n_jobs**: `-1` for all cores, `1` for sequential, or specific number
- **verbose**: `0` (silent), `10` (progress), `50` (detailed)
- **backend**: `'loky'` (CPU-bound, default) or `'threading'` (I/O-bound)
## Grid Search Example
```python
from joblib import Parallel, delayed
from itertools import product
def evaluate_params(param_a, param_b):
"""Evaluate one parameter combination."""
score = expensive_computation(param_a, param_b)
return {'param_a': param_a, 'param_b': param_b, 'score': score}
# Define parameter grid
params = list(product([0.1, 0.5, 1.0], [10, 20, 30]))
# Parallel grid search
results = Parallel(n_jobs=-1, verbose=10)(
delayed(evaluate_params)(a, b) for a, b in params
)
# Filter results
results = [r for r in results if r is not None]
best = max(results, key=lambda x: x['score'])
```
## Pre-computing Shared Data
When all tasks need the same data, pre-compute it once:
```python
# Pre-compute once
shared_data = load_data()
def process_with_shared(params, data):
return compute(params, data)
# Pass shared data to each task
results = Parallel(n_jobs=-1)(
delayed(process_with_shared)(p, shared_data)
for p in param_list
)
```
## Performance Tips
- Only worth it for tasks taking >0.1s per item (overhead cost)
- Watch memory usage - each worker gets a copy of data
- Use `verbose=10` to monitor progress
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |