DANG-ai/SKILLER / skilllearnbench/skills/human_authored/dbscan-parameter-tuning/parallel-processing
parallel-processing
VerifiedParallel processing with joblib for grid search and batch computations. Use when speeding up computationally intensive tasks across multiple CPU cores.
Parallel Processing with joblib
Speed up computationally intensive tasks by distributing work across multiple CPU cores.
Basic Usage
from joblib import Parallel, delayed
def process_item(x):
"""Process a single item."""
return x ** 2
# Sequential
results = [process_item(x) for x in range(100)]
# Parallel (uses all available cores)
results = Parallel(n_jobs=-1)(
delayed(process_item)(x) for x in range(100)
)
Key Parameters
- n_jobs:
-1for all cores,1for sequential, or specific number - verbose:
0(silent),10(progress),50(detailed) - backend:
'loky'(CPU-bound, default) or'threading'(I/O-bound)
Grid Search Example
from joblib import Parallel, delayed
from itertools import product
def evaluate_params(param_a, param_b):
"""Evaluate one parameter combination."""
score = expensive_computation(param_a, param_b)
return {'param_a': param_a, 'param_b': param_b, 'score': score}
# Define parameter grid
params = list(product([0.1, 0.5, 1.0], [10, 20, 30]))
# Parallel grid search
results = Parallel(n_jobs=-1, verbose=10)(
delayed(evaluate_params)(a, b) for a, b in params
)
# Filter results
results = [r for r in results if r is not None]
best = max(results, key=lambda x: x['score'])
Pre-computing Shared Data
When all tasks need the same data, pre-compute it once:
# Pre-compute once
shared_data = load_data()
def process_with_shared(params, data):
return compute(params, data)
# Pass shared data to each task
results = Parallel(n_jobs=-1)(
delayed(process_with_shared)(p, shared_data)
for p in param_list
)
Performance Tips
- Only worth it for tasks taking >0.1s per item (overhead cost)
- Watch memory usage - each worker gets a copy of data
- Use
verbose=10to monitor progress
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |