Skip to content

DAAF-Contribution-Community/daaf / .claude/skills/polars

polars

Verified

High-performance data manipulation with lazy/eager execution, expressions, I/O (CSV, Parquet, JSON), aggregations, joins, string/datetime ops, and pandas interop. Covers performance optimization patterns and common anti-patterns. DAAF's default DataFrame library — all pipeline code uses Polars, not pandas. Use for any DataFrame operation, reading/writing Parquet files, or migrating existing pandas code to Polars. R equivalent: tidyverse (use when execution language is R).

references/quickstart.md

View on GitHub
# Quickstart

## Installation

No installation needed in DAAF: `polars` (1.39.3) is pre-installed in the
container as the default DataFrame library. Runtime installs are blocked in DAAF
(`pip install`/`uv add` are refused both at the command line and inside executed
scripts — see CLAUDE.md § Runtime Package Installation). If an optional extra
(`polars[timezone]`, `polars[connectorx]`, `polars[fsspec]`, etc.) is genuinely
required, escalate to the user to add it to the Dockerfile (user additions block
near the end) and rebuild.

### Verify Installation

```python
import polars as pl
print(pl.__version__)  # Should be 1.x
```

## Core Concepts

### Lazy vs Eager Execution

Polars has two execution modes:

**Eager** - Executes immediately (like Pandas):
```python
df = pl.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
result = df.filter(pl.col("a") > 1)  # Runs now
```

**Lazy** - Builds query plan, optimizes, then executes:
```python
lf = pl.LazyFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
# Or from file scan
lf = pl.scan_csv("data.csv")

result = (
    lf.filter(pl.col("a") > 1)
    .select("a", "b")
    .collect()  # Executes optimized plan here
)
```

**When to use which:**
- **Eager**: Small data, interactive exploration, quick tests
- **Lazy**: Large data, complex pipelines, production code (recommended)

### Why Lazy is Better for Large Data

1. **Query Optimization**: Polars rewrites your query for efficiency
2. **Predicate Pushdown**: Filters applied at read time (skip unneeded rows)
3. **Projection Pushdown**: Only reads columns you actually use
4. **Streaming**: Process larger-than-memory datasets

```python
# This reads ONLY the needed columns and rows
result = (
    pl.scan_parquet("huge_file.parquet")
    .filter(pl.col("date") > pl.date(2024, 1, 1))  # Pushed down to file read
    .select("id", "value")                          # Only these columns loaded
    .collect()
)
```

### Expressions

Expressions are the heart of Polars. They describe transformations:

```python
# Expressions are functions: Series → Series
pl.col("a")              # Reference column "a"
pl.col("a") + 1          # Add 1 to each value
pl.col("a").sum()        # Sum all values
pl.col("a").alias("b")   # Rename to "b"
```

Expressions are evaluated in **contexts**:
- `select()` - Choose/transform columns
- `with_columns()` - Add new columns
- `filter()` - Filter rows
- `group_by().agg()` - Aggregate by groups

## Creating DataFrames

### From Dictionary

```python
df = pl.DataFrame({
    "name": ["Alice", "Bob", "Charlie"],
    "age": [25, 30, 35],
    "score": [85.5, 90.0, 78.5]
})
```

### With Explicit Schema

```python
df = pl.DataFrame(
    {
        "id": [1, 2, 3],
        "value": ["a", "b", "c"]
    },
    schema={
        "id": pl.Int32,
        "value": pl.String
    }
)
```

### From Series

```python
s1 = pl.Series("a", [1, 2, 3])
s2 = pl.Series("b", [4, 5, 6])
df = pl.DataFrame([s1, s2])
```

### Empty DataFrame with Schema

```python
df = pl.DataFrame(
    schema={
        "id": pl.Int64,
        "name": pl.String,
        "created": pl.Datetime
    }
)
```

### LazyFrame (Lazy Mode)

```python
# From data
lf = pl.LazyFrame({"a": [1, 2, 3]})

# From files (preferred for large data)
lf = pl.scan_csv("data.csv")
lf = pl.scan_parquet("data.parquet")
lf = pl.scan_ndjson("data.ndjson")
```

## Basic Operations

### View Data

```python
df.head(5)          # First 5 rows
df.tail(5)          # Last 5 rows
df.sample(10)       # Random 10 rows
df.glimpse()        # Transposed view
df.describe()       # Statistics summary
df.schema           # Column names and types
df.columns          # List of column names
df.shape            # (rows, cols) tuple
len(df)             # Row count
```

### Select Columns

```python
df.select("name", "age")
df.select(pl.col("name"), pl.col("age"))
df.select(pl.all())                      # All columns
df.select(pl.exclude("id"))              # All except "id"
```

### Filter Rows

```python
df.filter(pl.col("age") > 25)
df.filter((pl.col("age") > 25) & (pl.col("score") > 80))
```

### Add Columns

```python
df.with_columns(
    (pl.col("age") + 1).alias("age_next_year"),
    pl.lit("active").alias("status")
)
```

### Sort

```python
df.sort("age")
df.sort("age", descending=True)
df.sort(["name", "age"], descending=[False, True])
```

## LazyFrame Operations

### Building a Query

```python
query = (
    pl.scan_csv("data.csv")
    .filter(pl.col("status") == "active")
    .select("id", "name", "value")
    .with_columns(
        (pl.col("value") * 1.1).alias("adjusted")
    )
    .sort("value", descending=True)
)
```

### Executing

```python
# Execute and get DataFrame
df = query.collect()

# Execute with streaming (for large data)
df = query.collect(engine="streaming")

# Write directly to file (streaming)
query.sink_parquet("output.parquet")
query.sink_csv("output.csv")
```

### Inspecting the Plan

```python
# Logical plan (what you wrote)
print(query.explain())

# Optimized plan (what Polars will do)
print(query.explain(optimized=True))
```

## Type System

### Common Types

| Polars Type | Python Equivalent | Notes |
|-------------|-------------------|-------|
| `pl.Int8/16/32/64` | `int` | Signed integers |
| `pl.UInt8/16/32/64` | `int` | Unsigned integers |
| `pl.Float32/64` | `float` | Floating point |
| `pl.String` | `str` | UTF-8 strings (alias: `pl.Utf8`) |
| `pl.Boolean` | `bool` | True/False |
| `pl.Date` | `datetime.date` | Calendar date |
| `pl.Datetime` | `datetime.datetime` | Timestamp |
| `pl.Duration` | `datetime.timedelta` | Time difference |
| `pl.Time` | `datetime.time` | Time of day |
| `pl.Categorical` | - | Categorical strings |
| `pl.Enum` | - | Fixed set of categories |
| `pl.List` | `list` | Variable-length lists |
| `pl.Array` | - | Fixed-length arrays |
| `pl.Struct` | `dict` | Named fields |
| `pl.Null` | `None` | Null type |

### Casting Types

```python
df.with_columns(
    pl.col("id").cast(pl.Int32),
    pl.col("price").cast(pl.Float64),
    pl.col("category").cast(pl.Categorical)
)
```

## Next Steps

- Learn the [expression system](./expressions.md) - the core of Polars
- Master [data I/O](./io-data.md) for loading files
- Understand [performance patterns](./performance.md) for efficient code
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall