Skip to content

DAAF-Contribution-Community/daaf / .claude/skills/polars

polars

Verified

High-performance data manipulation with lazy/eager execution, expressions, I/O (CSV, Parquet, JSON), aggregations, joins, string/datetime ops, and pandas interop. Covers performance optimization patterns and common anti-patterns. DAAF's default DataFrame library — all pipeline code uses Polars, not pandas. Use for any DataFrame operation, reading/writing Parquet files, or migrating existing pandas code to Polars. R equivalent: tidyverse (use when execution language is R).

references/strings-datetime-categorical.md

View on GitHub
# Strings, Datetime & Categorical

## String Operations

All string operations are accessed via the `.str` namespace.

### Basic String Operations

```python
df.with_columns(
    pl.col("text").str.to_uppercase().alias("upper"),
    pl.col("text").str.to_lowercase().alias("lower"),
    pl.col("text").str.to_titlecase().alias("title"),
    pl.col("text").str.len_chars().alias("char_count"),      # Character count
    pl.col("text").str.len_bytes().alias("byte_count"),      # Byte count (UTF-8)
)
```

### Whitespace Handling

```python
df.with_columns(
    pl.col("text").str.strip_chars(),               # Strip both ends
    pl.col("text").str.strip_chars_start(),         # Left strip
    pl.col("text").str.strip_chars_end(),           # Right strip
    pl.col("text").str.strip_chars(" \t\n"),        # Strip specific chars
)
```

### Slicing and Substrings

```python
df.with_columns(
    pl.col("text").str.slice(0, 5),        # First 5 characters
    pl.col("text").str.slice(-3),          # Last 3 characters
    pl.col("text").str.head(5),            # First 5 characters
    pl.col("text").str.tail(3),            # Last 3 characters
)
```

### Search and Match

```python
df.with_columns(
    pl.col("text").str.contains("pattern"),              # Contains substring
    pl.col("text").str.contains("pat.*n", literal=False),# Regex match
    pl.col("text").str.starts_with("prefix"),            # Starts with
    pl.col("text").str.ends_with("suffix"),              # Ends with
    pl.col("text").str.find("needle"),                   # Index of first match (null if not found)
    pl.col("text").str.count_matches("a"),               # Count occurrences
)

# Filter by string content
df.filter(pl.col("name").str.contains("Smith"))
df.filter(pl.col("email").str.ends_with("@gmail.com"))
```

### Replace

```python
df.with_columns(
    # Replace first occurrence
    pl.col("text").str.replace("old", "new"),
    
    # Replace all occurrences
    pl.col("text").str.replace_all("old", "new"),
    
    # Regex replace
    pl.col("text").str.replace_all(r"\d+", "NUM"),
    
    # Replace with captured groups
    pl.col("text").str.replace_all(r"(\w+)@(\w+)", r"\2:\1"),
)
```

### Split and Join

```python
df.with_columns(
    # Split to list
    pl.col("tags").str.split(","),                  # Returns List[String]
    pl.col("path").str.split("/"),
    
    # Split and extract
    pl.col("tags").str.split(",").list.first(),     # First element
    pl.col("tags").str.split(",").list.get(1),      # Second element
    pl.col("tags").str.split(",").list.len(),       # Count elements
)

# Join list to string
df.with_columns(
    pl.col("list_col").list.join(", "),            # Join with separator
)
```

### Extract with Regex

```python
df.with_columns(
    # Extract first match of pattern
    pl.col("text").str.extract(r"(\d+)", group_index=1),
    
    # Extract all matches
    pl.col("text").str.extract_all(r"\d+"),         # Returns list
    
    # Named groups
    pl.col("text").str.extract_groups(r"(?<name>\w+):(?<value>\d+)"),
)
```

### Padding and Alignment

```python
df.with_columns(
    pl.col("code").str.pad_start(5, "0"),       # "42" -> "00042"
    pl.col("code").str.pad_end(10, " "),        # Right pad with spaces
    pl.col("code").str.zfill(5),                # Zero-fill left
)
```

### Concatenation

```python
# Concatenate columns
df.with_columns(
    pl.concat_str([
        pl.col("first_name"),
        pl.lit(" "),
        pl.col("last_name")
    ]).alias("full_name")
)

# With separator
df.with_columns(
    pl.concat_str(["city", "state", "country"], separator=", ").alias("location")
)
```

## Datetime Operations

All datetime operations are accessed via the `.dt` namespace.

### Parsing Strings to Datetime

```python
df.with_columns(
    # Auto-detect format
    pl.col("date_str").str.to_datetime(),
    
    # Explicit format
    pl.col("date_str").str.strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S"),
    pl.col("date_str").str.strptime(pl.Date, "%Y-%m-%d"),
    pl.col("date_str").str.strptime(pl.Time, "%H:%M:%S"),
    
    # With timezone
    pl.col("date_str").str.strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S", time_zone="UTC"),
)
```

### Common Format Codes

| Code | Meaning | Example |
|------|---------|---------|
| `%Y` | 4-digit year | 2024 |
| `%y` | 2-digit year | 24 |
| `%m` | Month (01-12) | 07 |
| `%d` | Day (01-31) | 15 |
| `%H` | Hour 24h (00-23) | 14 |
| `%I` | Hour 12h (01-12) | 02 |
| `%M` | Minute (00-59) | 30 |
| `%S` | Second (00-59) | 45 |
| `%f` | Microseconds | 123456 |
| `%p` | AM/PM | PM |
| `%z` | UTC offset | +0000 |
| `%Z` | Timezone name | UTC |

### Extracting Components

```python
df.with_columns(
    pl.col("datetime").dt.year().alias("year"),
    pl.col("datetime").dt.month().alias("month"),
    pl.col("datetime").dt.day().alias("day"),
    pl.col("datetime").dt.hour().alias("hour"),
    pl.col("datetime").dt.minute().alias("minute"),
    pl.col("datetime").dt.second().alias("second"),
    pl.col("datetime").dt.microsecond().alias("microsecond"),
    pl.col("datetime").dt.nanosecond().alias("nanosecond"),
    
    # Day of week (1=Monday, 7=Sunday)
    pl.col("datetime").dt.weekday().alias("weekday"),
    
    # Week of year
    pl.col("datetime").dt.week().alias("week"),
    
    # Day of year (1-366)
    pl.col("datetime").dt.ordinal_day().alias("day_of_year"),
    
    # Quarter (1-4)
    pl.col("datetime").dt.quarter().alias("quarter"),
    
    # ISO year and week
    pl.col("datetime").dt.iso_year().alias("iso_year"),
)
```

### Formatting to String

```python
df.with_columns(
    pl.col("datetime").dt.strftime("%Y-%m-%d").alias("date_str"),
    pl.col("datetime").dt.strftime("%B %d, %Y").alias("formatted"),
    pl.col("datetime").dt.to_string("%Y-%m-%d %H:%M").alias("str"),
)
```

### Truncation (Rounding Down)

```python
df.with_columns(
    pl.col("datetime").dt.truncate("1h").alias("hour_start"),
    pl.col("datetime").dt.truncate("1d").alias("day_start"),
    pl.col("datetime").dt.truncate("1w").alias("week_start"),
    pl.col("datetime").dt.truncate("1mo").alias("month_start"),
)
```

### Date/Time Arithmetic

```python
df.with_columns(
    # Add duration
    (pl.col("date") + pl.duration(days=7)).alias("next_week"),
    (pl.col("datetime") + pl.duration(hours=2)).alias("plus_2h"),
    
    # Subtract dates (returns Duration)
    (pl.col("end_date") - pl.col("start_date")).alias("duration"),
    
    # Duration to days/hours
    (pl.col("end_date") - pl.col("start_date")).dt.total_days().alias("days"),
    (pl.col("end_date") - pl.col("start_date")).dt.total_hours().alias("hours"),
)

# Duration literals
pl.duration(days=1)
pl.duration(hours=2, minutes=30)
pl.duration(weeks=1)
```

### Timezone Handling

```python
df.with_columns(
    # Set timezone (localize)
    pl.col("datetime").dt.replace_time_zone("UTC"),
    
    # Convert between timezones
    pl.col("datetime").dt.convert_time_zone("America/New_York"),
    
    # Remove timezone
    pl.col("datetime").dt.replace_time_zone(None),
)
```

### Filtering by Date

```python
# Date comparisons
df.filter(pl.col("date") > pl.date(2024, 1, 1))
df.filter(pl.col("date").is_between(pl.date(2024, 1, 1), pl.date(2024, 12, 31)))

# Filter by component
df.filter(pl.col("datetime").dt.year() == 2024)
df.filter(pl.col("datetime").dt.month() == 7)
df.filter(pl.col("datetime").dt.weekday() < 5)  # Weekdays only
```

### Creating Dates/Datetimes

```python
# Date literals
pl.date(2024, 7, 15)

# Datetime literals
pl.datetime(2024, 7, 15, 14, 30, 0)

# From columns
df.with_columns(
    pl.datetime(
        pl.col("year"),
        pl.col("month"),
        pl.col("day")
    ).alias("date")
)

# Date range
pl.date_range(
    start=pl.date(2024, 1, 1),
    end=pl.date(2024, 12, 31),
    interval="1d",
    eager=True
)
```

## Categorical Data

Categorical type stores strings as integers for memory efficiency.

### Creating Categorical

```python
df.with_columns(
    pl.col("category").cast(pl.Categorical)
)

# Or on read
df = pl.read_csv("data.csv", schema_overrides={"category": pl.Categorical})
```

### Enum Type (Fixed Categories)

For known, fixed set of categories:

```python
# Define enum type
status_type = pl.Enum(["pending", "active", "completed", "cancelled"])

df.with_columns(
    pl.col("status").cast(status_type)
)

# Enum is stricter - errors on unknown values
```

### StringCache (No Longer Needed)

In current Polars 1.x (verified on 1.39.3), every `pl.Categorical` without explicit categories uses one global category mapping, so the same string gets the same code in every DataFrame ("Categoricals no longer have a local scope"). Categoricals built separately can be concatenated or joined directly; `pl.StringCache()` still runs but is not required.

```python
# No StringCache needed: both frames share the global categories
df1 = pl.DataFrame({"cat": ["a", "b"]}).with_columns(
    pl.col("cat").cast(pl.Categorical)
)
df2 = pl.DataFrame({"cat": ["b", "c"]}).with_columns(
    pl.col("cat").cast(pl.Categorical)
)

combined = pl.concat([df1, df2])
```

### Categorical Operations

```python
df.with_columns(
    # Get physical (integer) representation
    pl.col("category").to_physical().alias("cat_code"),
    
    # Get categories
    pl.col("category").cat.get_categories().alias("categories"),
)

# Sort by categorical order (not alphabetically)
df.sort("category")
```

### Converting Back to String

```python
df.with_columns(
    pl.col("category").cast(pl.String)
)
```

## Summary

### String Functions

| Operation | Code |
|-----------|------|
| Uppercase | `pl.col("s").str.to_uppercase()` |
| Contains | `pl.col("s").str.contains("x")` |
| Replace | `pl.col("s").str.replace_all("a", "b")` |
| Split | `pl.col("s").str.split(",")` |
| Extract | `pl.col("s").str.extract(r"(\d+)")` |
| Length | `pl.col("s").str.len_chars()` |
| Trim | `pl.col("s").str.strip_chars()` |
| Concat | `pl.concat_str(["a", "b"], separator=" ")` |

### Datetime Functions

| Operation | Code |
|-----------|------|
| Parse | `pl.col("s").str.strptime(pl.Datetime, "%Y-%m-%d")` |
| Year | `pl.col("dt").dt.year()` |
| Month | `pl.col("dt").dt.month()` |
| Weekday | `pl.col("dt").dt.weekday()` |
| Format | `pl.col("dt").dt.strftime("%Y-%m-%d")` |
| Truncate | `pl.col("dt").dt.truncate("1d")` |
| Add time | `pl.col("dt") + pl.duration(days=7)` |
| Diff | `(pl.col("end") - pl.col("start")).dt.total_days()` |

### Categorical

| Operation | Code |
|-----------|------|
| Cast to categorical | `pl.col("c").cast(pl.Categorical)` |
| Use enum | `pl.col("c").cast(pl.Enum(["a", "b"]))` |
| Get codes | `pl.col("c").to_physical()` |
| Combine across frames | `pl.concat([df1, df2])` (no StringCache needed) |
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall