Skip to content

a5c-ai/babysitter / library/specializations/data-engineering-analytics/skills/apache-spark-optimizer

Apache Spark Optimizer

Verified

Analyzes and optimizes Apache Spark jobs for performance, cost, and resource utilization

FileKindSize
README.mdmarkdown566 B
SKILL.mdskill md2.3 KB
# Apache Spark Optimizer

Skill ID: SK-DEA-001

## Purpose

Analyzes and optimizes Apache Spark jobs for performance, cost, and resource utilization. Provides recommendations for partition strategies, shuffle reduction, memory tuning, and Catalyst optimizer hints.

## Key Features

- Execution plan analysis
- Partition optimization
- Shuffle reduction
- Memory and executor tuning
- Data skew mitigation
- Broadcast join optimization
- Caching strategies

## Applicable Processes

- ETL/ELT Pipeline
- Streaming Pipeline
- Feature Store Setup
- Pipeline Migration
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall