a5c-ai/babysitter / library/specializations/data-engineering-analytics/skills/apache-spark-optimizer
Apache Spark Optimizer
VerifiedAnalyzes and optimizes Apache Spark jobs for performance, cost, and resource utilization
README.md
View on GitHub# Apache Spark Optimizer
Skill ID: SK-DEA-001
## Purpose
Analyzes and optimizes Apache Spark jobs for performance, cost, and resource utilization. Provides recommendations for partition strategies, shuffle reduction, memory tuning, and Catalyst optimizer hints.
## Key Features
- Execution plan analysis
- Partition optimization
- Shuffle reduction
- Memory and executor tuning
- Data skew mitigation
- Broadcast join optimization
- Caching strategies
## Applicable Processes
- ETL/ELT Pipeline
- Streaming Pipeline
- Feature Store Setup
- Pipeline Migration
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |