DAAF-Contribution-Community/daaf / .claude/skills/polars
polars
VerifiedHigh-performance data manipulation with lazy/eager execution, expressions, I/O (CSV, Parquet, JSON), aggregations, joins, string/datetime ops, and pandas interop. Covers performance optimization patterns and common anti-patterns. DAAF's default DataFrame library — all pipeline code uses Polars, not pandas. Use for any DataFrame operation, reading/writing Parquet files, or migrating existing pandas code to Polars. R equivalent: tidyverse (use when execution language is R).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 12.4 KB |
| references/aggregations-grouping.md | markdown | 9.6 KB |
| references/dataframes-series.md | markdown | 8.6 KB |
| references/expressions.md | markdown | 8.7 KB |
| references/gotchas.md | markdown | 8.5 KB |
| references/interop.md | markdown | 8.8 KB |
| references/io-data.md | markdown | 9.5 KB |
| references/joins-concat.md | markdown | 7.7 KB |
| references/performance.md | markdown | 8.7 KB |
| references/quickstart.md | markdown | 6.1 KB |
| references/strings-datetime-categorical.md | markdown | 10.5 KB |
references/joins-concat.md
View on GitHub# Joins & Concatenation
## Join Types
### Inner Join (Default)
Returns only rows with matches in both DataFrames:
```python
df1.join(df2, on="key")
df1.join(df2, on="key", how="inner")
```
### Left Join
Returns all rows from left, matching rows from right (null if no match):
```python
df1.join(df2, on="key", how="left")
```
### Right Join
Returns all rows from right, matching rows from left:
```python
df1.join(df2, on="key", how="right")
```
### Full Outer Join
Returns all rows from both DataFrames:
```python
df1.join(df2, on="key", how="full")
# Note: how="outer" is deprecated (still runs with a DeprecationWarning); use how="full"
```
### Cross Join
Cartesian product (every row paired with every row):
```python
df1.join(df2, how="cross")
```
### Anti Join
Rows in left that have NO match in right:
```python
df1.join(df2, on="key", how="anti")
```
### Semi Join
Rows in left that HAVE a match in right (but doesn't include right columns):
```python
df1.join(df2, on="key", how="semi")
```
## Join Keys
### Single Key
```python
df1.join(df2, on="id")
```
### Multiple Keys
```python
df1.join(df2, on=["id", "date"])
```
### Different Column Names
```python
df1.join(df2, left_on="user_id", right_on="id")
df1.join(df2, left_on=["user_id", "date"], right_on=["id", "timestamp"])
```
### Join on Expression
```python
# Join on transformed keys
df1.join(
df2,
left_on=pl.col("date").dt.date(),
right_on=pl.col("timestamp").dt.date()
)
```
## Handling Duplicate Columns
When both DataFrames have columns with the same name (other than join keys):
```python
# Default: adds "_right" suffix
df1.join(df2, on="id")
# If both have "value" column, result has "value" and "value_right"
# Custom suffix
df1.join(df2, on="id", suffix="_from_df2")
# Coalesce join keys: one "id" column instead of "id" and "id_right".
# Non-key columns (value / value_right) are never merged; use pl.coalesce for that
df1.join(df2, on="id", how="full", coalesce=True)
```
## Null Keys
By default null keys never match.
```python
# Default: null keys do not match (nulls_equal=False)
df1.join(df2, on="key")
# Match null keys to each other (renamed from join_nulls in 1.24)
df1.join(df2, on="key", nulls_equal=True)
```
## Concatenation
### Vertical Concatenation (Stack Rows)
```python
# Two DataFrames
pl.concat([df1, df2])
# Multiple DataFrames
pl.concat([df1, df2, df3, df4])
# Different column order: reorder first (how="align" is a horizontal full join, not a stack)
pl.concat([df1, df2.select(df1.columns)])
# Diagonal (union of all columns, nulls for missing)
pl.concat([df1, df2], how="diagonal")
# Diagonal with relaxed types
pl.concat([df1, df2], how="diagonal_relaxed")
```
### Horizontal Concatenation (Stack Columns)
```python
pl.concat([df1, df2], how="horizontal")
# Requires same number of rows
```
### From List of DataFrames
```python
dfs = [pl.read_csv(f"data_{i}.csv") for i in range(10)]
combined = pl.concat(dfs)
```
## Pivot (Long to Wide)
Convert long format to wide format:
```python
# Long format:
# | date | product | sales |
# | 2024-01-01 | A | 100 |
# | 2024-01-01 | B | 150 |
# | 2024-01-02 | A | 120 |
df.pivot(
on="product", # Column to spread
index="date", # Keep as rows
values="sales" # Values to fill
)
# Result:
# | date | A | B |
# | 2024-01-01 | 100 | 150 |
# | 2024-01-02 | 120 | null|
```
### Pivot with Aggregation
```python
df.pivot(
on="product",
index="date",
values="sales",
aggregate_function="sum" # If duplicates exist
)
# Multiple aggregations
df.pivot(
on="product",
index="date",
values="sales",
aggregate_function="first" # first, last, sum, mean, len, etc.
)
```
### Multiple Index Columns
```python
df.pivot(
on="product",
index=["date", "region"],
values="sales"
)
```
### Multiple Value Columns
```python
df.pivot(
on="product",
index="date",
values=["sales", "quantity"]
)
```
## Unpivot / Melt (Wide to Long)
Convert wide format to long format:
```python
# Wide format:
# | date | product_A | product_B |
# | 2024-01-01 | 100 | 150 |
df.unpivot(
on=["product_A", "product_B"], # Columns to melt
index="date", # Keep as identifier
variable_name="product", # Name for column names
value_name="sales" # Name for values
)
# Result:
# | date | product | sales |
# | 2024-01-01 | product_A | 100 |
# | 2024-01-01 | product_B | 150 |
```
**Note**: `melt` was renamed to `unpivot` in Polars 1.0; `melt` still runs but emits a DeprecationWarning.
### Unpivot All Except Index
```python
df.unpivot(
index=["id", "date"], # Keep these
# All other columns are melted
)
```
### Select Columns to Melt
```python
# By name
df.unpivot(on=["col1", "col2", "col3"], index="id")
# By pattern
import polars.selectors as cs
df.unpivot(on=cs.starts_with("value_"), index="id")
```
## Common Patterns
### Self Join
```python
# Join DataFrame to itself (e.g., find pairs)
df.join(df, on="category", suffix="_other").filter(
pl.col("id") < pl.col("id_other")
)
```
### Lookup Table
```python
# Main data
orders = pl.DataFrame({
"order_id": [1, 2, 3],
"product_code": ["A", "B", "A"]
})
# Lookup table
products = pl.DataFrame({
"code": ["A", "B"],
"name": ["Apple", "Banana"]
})
# Add product names
orders.join(products, left_on="product_code", right_on="code", how="left")
```
### Update Values from Another DataFrame
```python
# Left join, then pl.coalesce to prefer the new value
df1.join(df2, on="id", how="left", suffix="_new").with_columns(
pl.coalesce(pl.col("value_new"), pl.col("value")).alias("value")
).drop("value_new")
```
### Conditional Join (Inequality)
```python
# Polars doesn't have direct inequality joins
# Use cross join + filter
df1.join(df2, how="cross").filter(
(pl.col("start") <= pl.col("date")) &
(pl.col("date") < pl.col("end"))
)
# Or use join_asof for time-based inequality
```
### As-Of Join (Time-Based)
Join on nearest time match:
```python
# df1 has events at various times
# df2 has reference values at specific times
# Match each event to the most recent reference
df1.sort("timestamp").join_asof(
df2.sort("timestamp"),
on="timestamp",
strategy="backward" # Use most recent value <= event time
)
# Strategies:
# "backward" - most recent value <= key
# "forward" - next value >= key
# "nearest" - closest value
# With tolerance
df1.join_asof(
df2,
on="timestamp",
tolerance="1h" # Match only within 1 hour
)
# With grouping
df1.join_asof(
df2,
on="timestamp",
by="symbol", # Match within same symbol
strategy="backward"
)
```
### Multiple DataFrames Join
```python
# Chain joins
result = (
df1
.join(df2, on="key1", how="left")
.join(df3, on="key2", how="left")
.join(df4, on="key3", how="left")
)
# Or reduce over list
from functools import reduce
dfs = [df1, df2, df3, df4]
result = reduce(lambda a, b: a.join(b, on="id", how="left"), dfs)
```
## Summary
| Operation | Code |
|-----------|------|
| Inner join | `df1.join(df2, on="key")` |
| Left join | `df1.join(df2, on="key", how="left")` |
| Full outer join | `df1.join(df2, on="key", how="full")` |
| Anti join | `df1.join(df2, on="key", how="anti")` |
| Semi join | `df1.join(df2, on="key", how="semi")` |
| Different keys | `df1.join(df2, left_on="a", right_on="b")` |
| Vertical concat | `pl.concat([df1, df2])` |
| Horizontal concat | `pl.concat([df1, df2], how="horizontal")` |
| Pivot (long→wide) | `df.pivot(on="col", index="id", values="val")` |
| Unpivot (wide→long) | `df.unpivot(on=["a", "b"], index="id")` |
| As-of join | `df1.join_asof(df2, on="time")` |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |