Skip to content

DAAF-Contribution-Community/daaf / .claude/skills/polars

polars

Verified

High-performance data manipulation with lazy/eager execution, expressions, I/O (CSV, Parquet, JSON), aggregations, joins, string/datetime ops, and pandas interop. Covers performance optimization patterns and common anti-patterns. DAAF's default DataFrame library — all pipeline code uses Polars, not pandas. Use for any DataFrame operation, reading/writing Parquet files, or migrating existing pandas code to Polars. R equivalent: tidyverse (use when execution language is R).

references/performance.md

View on GitHub
# Performance Optimization

## Lazy Evaluation

### Why Use Lazy Mode

Lazy evaluation is Polars' killer feature for performance:

1. **Query Optimization**: Polars rewrites your query for efficiency
2. **Predicate Pushdown**: Filters are pushed to file read operations
3. **Projection Pushdown**: Only required columns are loaded
4. **Common Subexpression Elimination**: Avoids redundant computation
5. **Streaming**: Enables processing larger-than-memory datasets

### Lazy vs Eager Comparison

```python
# Eager (reads entire file, then filters)
df = pl.read_csv("huge.csv")           # Load all data
df = df.filter(pl.col("status") == "active")  # Then filter

# Lazy (optimized - reads only what's needed)
df = (
    pl.scan_csv("huge.csv")            # Just creates plan
    .filter(pl.col("status") == "active")  # Adds filter to plan
    .collect()                         # Executes optimized plan
)
```

### Inspecting Query Plans

```python
lf = (
    pl.scan_parquet("data.parquet")
    .filter(pl.col("date") > pl.date(2024, 1, 1))
    .select("id", "name", "value")
    .group_by("name")
    .agg(pl.col("value").sum())
)

# View logical plan (what you wrote)
print(lf.explain())

# View optimized plan (what Polars will do)
print(lf.explain(optimized=True))
```

## Best Practices

### Use scan_ Instead of read_

```python
# Prefer this
lf = pl.scan_csv("data.csv")
lf = pl.scan_parquet("data.parquet")
lf = pl.scan_ndjson("data.ndjson")

# Over this
df = pl.read_csv("data.csv")
df = pl.read_parquet("data.parquet")
```

### Filter Early

```python
# Good - filter pushed to scan
result = (
    pl.scan_parquet("data.parquet")
    .filter(pl.col("year") == 2024)      # First
    .select("id", "name", "value")
    .group_by("name")
    .agg(pl.col("value").sum())
    .collect()
)

# Less efficient - processes more data than needed
result = (
    pl.scan_parquet("data.parquet")
    .group_by("name")
    .agg(pl.col("value").sum())
    .filter(pl.col("year") == 2024)      # Too late!
    .collect()
)
```

### Select Only Needed Columns

```python
# Good - only reads 3 columns from file
result = (
    pl.scan_parquet("wide_table.parquet")
    .select("id", "name", "value")
    .collect()
)

# Bad - reads all columns then discards
df = pl.read_parquet("wide_table.parquet")
result = df.select("id", "name", "value")
```

### Avoid Multiple collect() Calls

```python
# Bad - scans file twice
lf = pl.scan_parquet("data.parquet")
sum_result = lf.select(pl.col("value").sum()).collect()
mean_result = lf.select(pl.col("value").mean()).collect()

# Good - single scan
lf = pl.scan_parquet("data.parquet")
results = lf.select(
    pl.col("value").sum().alias("sum"),
    pl.col("value").mean().alias("mean")
).collect()

# Or use collect_all for separate LazyFrames
lf = pl.scan_parquet("data.parquet")
lf1 = lf.filter(pl.col("type") == "A").select(pl.col("value").sum())
lf2 = lf.filter(pl.col("type") == "B").select(pl.col("value").sum())
results = pl.collect_all([lf1, lf2])  # Optimized together
```

## Anti-Patterns to Avoid

### Row-by-Row Iteration

```python
# BAD - Extremely slow
result = []
for row in df.iter_rows(named=True):
    result.append(row["value"] * 2)

# GOOD - Vectorized
df.with_columns(
    (pl.col("value") * 2).alias("doubled")
)
```

### Using map_elements (Python UDFs)

```python
# BAD - Python overhead for each element
df.with_columns(
    pl.col("value").map_elements(lambda x: x * 2, return_dtype=pl.Int64)
)

# GOOD - Native Polars expression
df.with_columns(
    (pl.col("value") * 2).alias("result")
)

# If you must use Python functions, prefer map_batches
df.with_columns(
    pl.col("value").map_batches(lambda s: s * 2)  # Works on entire Series
)
```

### String Concatenation in Loop

```python
# BAD
result = df.select(pl.col("a"))
for col in ["b", "c", "d"]:
    result = result.with_columns(pl.col(col))

# GOOD
result = df.select("a", "b", "c", "d")
```

### Creating DataFrames in Loops

```python
# BAD - Creates many small DataFrames
dfs = []
for file in files:
    dfs.append(pl.read_parquet(file))
combined = pl.concat(dfs)

# GOOD - Single scan with glob
combined = pl.scan_parquet("data/*.parquet").collect()
```

## Memory Optimization

### Use Appropriate Data Types

```python
# Check current memory usage
print(df.estimated_size())  # Bytes

# Downcast numeric types
df = df.with_columns(
    pl.col("small_int").cast(pl.Int16),      # Instead of Int64
    pl.col("tiny_int").cast(pl.Int8),
    pl.col("unsigned").cast(pl.UInt32),
    pl.col("small_float").cast(pl.Float32),  # Instead of Float64
)
```

### Use Categorical for Repeated Strings

```python
# High cardinality strings = lots of memory
df = pl.DataFrame({"category": ["A", "B", "A", "C", "B"] * 1000000})
print(df.estimated_size())  # Large

# Categorical = much smaller
df = df.with_columns(pl.col("category").cast(pl.Categorical))
print(df.estimated_size())  # Much smaller
```

### Streaming for Large Files

```python
# Process larger-than-memory files
result = (
    pl.scan_parquet("huge_file.parquet")
    .filter(pl.col("status") == "active")
    .group_by("category")
    .agg(pl.col("value").sum())
    .collect(engine="streaming")  # Processes in chunks
)

# Write directly without collecting
(
    pl.scan_parquet("input.parquet")
    .filter(pl.col("date") > pl.date(2024, 1, 1))
    .sink_parquet("output.parquet")  # Streams to file
)
```

### Release Memory

```python
# Delete when done
del large_df

# Or use context manager for temporary data
def process():
    df = pl.read_parquet("data.parquet")
    result = df.group_by("key").agg(pl.col("value").sum())
    return result  # df goes out of scope

summary = process()  # Large df is released
```

## Parallel Processing

Polars automatically parallelizes operations across CPU cores.

### Check Thread Count

```python
import polars as pl
print(pl.thread_pool_size())
```

### Control Parallelism

```python
# Set at runtime (before any operations)
import os
os.environ["POLARS_MAX_THREADS"] = "4"

# Or in code
pl.Config.set_streaming_chunk_size(100_000)
```

### Operations That Parallelize Well

- Aggregations (sum, mean, etc.)
- Filtering
- Joins
- Group by operations
- Window functions

## Benchmarking

### Profile Your Code

```python
import time

start = time.perf_counter()
result = (
    pl.scan_parquet("data.parquet")
    .filter(pl.col("x") > 0)
    .group_by("category")
    .agg(pl.col("value").sum())
    .collect()
)
elapsed = time.perf_counter() - start
print(f"Elapsed: {elapsed:.3f}s")
```

### Compare Approaches

```python
import timeit

# Approach 1: Eager
def eager_approach():
    df = pl.read_parquet("data.parquet")
    return df.filter(pl.col("x") > 0).group_by("y").agg(pl.col("z").sum())

# Approach 2: Lazy
def lazy_approach():
    return (
        pl.scan_parquet("data.parquet")
        .filter(pl.col("x") > 0)
        .group_by("y")
        .agg(pl.col("z").sum())
        .collect()
    )

print("Eager:", timeit.timeit(eager_approach, number=10))
print("Lazy:", timeit.timeit(lazy_approach, number=10))
```

## File Format Performance

### Use Parquet

Parquet is significantly faster than CSV:

```python
# CSV - slow, no type preservation
df = pl.read_csv("data.csv")           # Slow
df = pl.scan_csv("data.csv").collect() # Better but still slow

# Parquet - fast, typed, compressed
df = pl.scan_parquet("data.parquet").collect()  # Fast!
```

### Parquet Compression

```python
# Write with compression
df.write_parquet("data.parquet", compression="zstd")  # Good balance
df.write_parquet("data.parquet", compression="snappy")  # Faster decompress
df.write_parquet("data.parquet", compression="lz4")     # Fastest

# Compression levels (zstd)
df.write_parquet("data.parquet", compression="zstd", compression_level=3)  # Default
df.write_parquet("data.parquet", compression="zstd", compression_level=19) # High compression (max is 22)
```

## Common Performance Issues

| Issue | Cause | Solution |
|-------|-------|----------|
| Slow on large CSV | Eager read | Use `scan_csv` + lazy |
| Memory error | Loading entire file | Use streaming or lazy |
| Slow filters | After aggregation | Move filters before grouping |
| Slow string ops | `map_elements` | Use native `.str` methods |
| Multiple scans | Multiple `collect()` | Use `collect_all` or single pipeline |
| Type conversion | Wrong inference | Specify schema explicitly |
| Slow joins | Large DataFrames | Use lazy mode, filter first |

## Checklist for Optimal Performance

1. Use `scan_*` instead of `read_*`
2. Use Parquet format when possible
3. Filter early in the pipeline
4. Select only needed columns
5. Avoid `map_elements`/Python UDFs
6. Use appropriate data types (Int32 vs Int64, Categorical)
7. Use single `collect()` or `collect_all()`
8. Enable streaming for large data: `collect(engine="streaming")`
9. Use native expressions instead of iteration
10. Profile and benchmark your code
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall