foryourhealth111-pixel/Vibe-Skills / bundled/skills/polars
polars
VerifiedFast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.2 KB |
| references/best_practices.md | markdown | 13.8 KB |
| references/core_concepts.md | markdown | 8.2 KB |
| references/io_guide.md | markdown | 10.9 KB |
| references/operations.md | markdown | 11.8 KB |
| references/pandas_migration.md | markdown | 12.1 KB |
| references/transformations.md | markdown | 10.9 KB |
references/pandas_migration.md
View on GitHub# Pandas to Polars Migration Guide
This guide helps you migrate from pandas to Polars with comprehensive operation mappings and key differences.
## Core Conceptual Differences
### 1. No Index System
**Pandas:** Uses row-based indexing system
```python
df.loc[0, "column"]
df.iloc[0:5]
df.set_index("id")
```
**Polars:** Uses integer positions only
```python
df[0, "column"] # Row position, column name
df[0:5] # Row slice
# No set_index equivalent - use group_by instead
```
### 2. Memory Format
**Pandas:** Row-oriented NumPy arrays
**Polars:** Columnar Apache Arrow format
**Implications:**
- Polars is faster for column operations
- Polars uses less memory
- Polars has better data sharing capabilities
### 3. Parallelization
**Pandas:** Primarily single-threaded (requires Dask for parallelism)
**Polars:** Parallel by default using Rust's concurrency
### 4. Lazy Evaluation
**Pandas:** Only eager evaluation
**Polars:** Both eager (DataFrame) and lazy (LazyFrame) with query optimization
### 5. Type Strictness
**Pandas:** Allows silent type conversions
**Polars:** Strict typing, explicit casts required
**Example:**
```python
# Pandas: Silently converts to float
pd_df["int_col"] = [1, 2, None, 4] # dtype: float64
# Polars: Keeps as integer with null
pl_df = pl.DataFrame({"int_col": [1, 2, None, 4]}) # dtype: Int64
```
## Operation Mappings
### Data Selection
| Operation | Pandas | Polars |
|-----------|--------|--------|
| Select column | `df["col"]` or `df.col` | `df.select("col")` or `df["col"]` |
| Select multiple | `df[["a", "b"]]` | `df.select("a", "b")` |
| Select by position | `df.iloc[:, 0:3]` | `df.select(pl.col(df.columns[0:3]))` |
| Select by condition | `df[df["age"] > 25]` | `df.filter(pl.col("age") > 25)` |
### Data Filtering
| Operation | Pandas | Polars |
|-----------|--------|--------|
| Single condition | `df[df["age"] > 25]` | `df.filter(pl.col("age") > 25)` |
| Multiple conditions | `df[(df["age"] > 25) & (df["city"] == "NY")]` | `df.filter(pl.col("age") > 25, pl.col("city") == "NY")` |
| Query method | `df.query("age > 25")` | `df.filter(pl.col("age") > 25)` |
| isin | `df[df["city"].isin(["NY", "LA"])]` | `df.filter(pl.col("city").is_in(["NY", "LA"]))` |
| isna | `df[df["value"].isna()]` | `df.filter(pl.col("value").is_null())` |
| notna | `df[df["value"].notna()]` | `df.filter(pl.col("value").is_not_null())` |
### Adding/Modifying Columns
| Operation | Pandas | Polars |
|-----------|--------|--------|
| Add column | `df["new"] = df["old"] * 2` | `df.with_columns(new=pl.col("old") * 2)` |
| Multiple columns | `df.assign(a=..., b=...)` | `df.with_columns(a=..., b=...)` |
| Conditional column | `np.where(condition, a, b)` | `pl.when(condition).then(a).otherwise(b)` |
**Important difference - Parallel execution:**
```python
# Pandas: Sequential (lambda sees previous results)
df.assign(
a=lambda df_: df_.value * 10,
b=lambda df_: df_.value * 100
)
# Polars: Parallel (all computed together)
df.with_columns(
a=pl.col("value") * 10,
b=pl.col("value") * 100
)
```
### Grouping and Aggregation
| Operation | Pandas | Polars |
|-----------|--------|--------|
| Group by | `df.groupby("col")` | `df.group_by("col")` |
| Agg single | `df.groupby("col")["val"].mean()` | `df.group_by("col").agg(pl.col("val").mean())` |
| Agg multiple | `df.groupby("col").agg({"val": ["mean", "sum"]})` | `df.group_by("col").agg(pl.col("val").mean(), pl.col("val").sum())` |
| Size | `df.groupby("col").size()` | `df.group_by("col").agg(pl.len())` |
| Count | `df.groupby("col").count()` | `df.group_by("col").agg(pl.col("*").count())` |
### Window Functions
| Operation | Pandas | Polars |
|-----------|--------|--------|
| Transform | `df.groupby("col").transform("mean")` | `df.with_columns(pl.col("val").mean().over("col"))` |
| Rank | `df.groupby("col")["val"].rank()` | `df.with_columns(pl.col("val").rank().over("col"))` |
| Shift | `df.groupby("col")["val"].shift(1)` | `df.with_columns(pl.col("val").shift(1).over("col"))` |
| Cumsum | `df.groupby("col")["val"].cumsum()` | `df.with_columns(pl.col("val").cum_sum().over("col"))` |
### Joins
| Operation | Pandas | Polars |
|-----------|--------|--------|
| Inner join | `df1.merge(df2, on="id")` | `df1.join(df2, on="id", how="inner")` |
| Left join | `df1.merge(df2, on="id", how="left")` | `df1.join(df2, on="id", how="left")` |
| Different keys | `df1.merge(df2, left_on="a", right_on="b")` | `df1.join(df2, left_on="a", right_on="b")` |
### Concatenation
| Operation | Pandas | Polars |
|-----------|--------|--------|
| Vertical | `pd.concat([df1, df2], axis=0)` | `pl.concat([df1, df2], how="vertical")` |
| Horizontal | `pd.concat([df1, df2], axis=1)` | `pl.concat([df1, df2], how="horizontal")` |
### Sorting
| Operation | Pandas | Polars |
|-----------|--------|--------|
| Sort by column | `df.sort_values("col")` | `df.sort("col")` |
| Descending | `df.sort_values("col", ascending=False)` | `df.sort("col", descending=True)` |
| Multiple columns | `df.sort_values(["a", "b"])` | `df.sort("a", "b")` |
### Reshaping
| Operation | Pandas | Polars |
|-----------|--------|--------|
| Pivot | `df.pivot(index="a", columns="b", values="c")` | `df.pivot(values="c", index="a", columns="b")` |
| Melt | `df.melt(id_vars="id")` | `df.unpivot(index="id")` |
### I/O Operations
| Operation | Pandas | Polars |
|-----------|--------|--------|
| Read CSV | `pd.read_csv("file.csv")` | `pl.read_csv("file.csv")` or `pl.scan_csv()` |
| Write CSV | `df.to_csv("file.csv")` | `df.write_csv("file.csv")` |
| Read Parquet | `pd.read_parquet("file.parquet")` | `pl.read_parquet("file.parquet")` |
| Write Parquet | `df.to_parquet("file.parquet")` | `df.write_parquet("file.parquet")` |
| Read Excel | `pd.read_excel("file.xlsx")` | `pl.read_excel("file.xlsx")` |
### String Operations
| Operation | Pandas | Polars |
|-----------|--------|--------|
| Upper | `df["col"].str.upper()` | `df.select(pl.col("col").str.to_uppercase())` |
| Lower | `df["col"].str.lower()` | `df.select(pl.col("col").str.to_lowercase())` |
| Contains | `df["col"].str.contains("pattern")` | `df.filter(pl.col("col").str.contains("pattern"))` |
| Replace | `df["col"].str.replace("old", "new")` | `df.select(pl.col("col").str.replace("old", "new"))` |
| Split | `df["col"].str.split(" ")` | `df.select(pl.col("col").str.split(" "))` |
### Datetime Operations
| Operation | Pandas | Polars |
|-----------|--------|--------|
| Parse dates | `pd.to_datetime(df["col"])` | `df.select(pl.col("col").str.strptime(pl.Date, "%Y-%m-%d"))` |
| Year | `df["date"].dt.year` | `df.select(pl.col("date").dt.year())` |
| Month | `df["date"].dt.month` | `df.select(pl.col("date").dt.month())` |
| Day | `df["date"].dt.day` | `df.select(pl.col("date").dt.day())` |
### Missing Data
| Operation | Pandas | Polars |
|-----------|--------|--------|
| Drop nulls | `df.dropna()` | `df.drop_nulls()` |
| Fill nulls | `df.fillna(0)` | `df.fill_null(0)` |
| Check null | `df["col"].isna()` | `df.select(pl.col("col").is_null())` |
| Forward fill | `df.fillna(method="ffill")` | `df.select(pl.col("col").fill_null(strategy="forward"))` |
### Other Operations
| Operation | Pandas | Polars |
|-----------|--------|--------|
| Unique values | `df["col"].unique()` | `df["col"].unique()` |
| Value counts | `df["col"].value_counts()` | `df["col"].value_counts()` |
| Describe | `df.describe()` | `df.describe()` |
| Sample | `df.sample(n=100)` | `df.sample(n=100)` |
| Head | `df.head()` | `df.head()` |
| Tail | `df.tail()` | `df.tail()` |
## Common Migration Patterns
### Pattern 1: Chained Operations
**Pandas:**
```python
result = (df
.assign(new_col=lambda x: x["old_col"] * 2)
.query("new_col > 10")
.groupby("category")
.agg({"value": "sum"})
.reset_index()
)
```
**Polars:**
```python
result = (df
.with_columns(new_col=pl.col("old_col") * 2)
.filter(pl.col("new_col") > 10)
.group_by("category")
.agg(pl.col("value").sum())
)
# No reset_index needed - Polars doesn't have index
```
### Pattern 2: Apply Functions
**Pandas:**
```python
# Avoid in Polars - breaks parallelization
df["result"] = df["value"].apply(lambda x: x * 2)
```
**Polars:**
```python
# Use expressions instead
df = df.with_columns(result=pl.col("value") * 2)
# If custom function needed
df = df.with_columns(
result=pl.col("value").map_elements(lambda x: x * 2, return_dtype=pl.Float64)
)
```
### Pattern 3: Conditional Column Creation
**Pandas:**
```python
df["category"] = np.where(
df["value"] > 100,
"high",
np.where(df["value"] > 50, "medium", "low")
)
```
**Polars:**
```python
df = df.with_columns(
category=pl.when(pl.col("value") > 100)
.then("high")
.when(pl.col("value") > 50)
.then("medium")
.otherwise("low")
)
```
### Pattern 4: Group Transform
**Pandas:**
```python
df["group_mean"] = df.groupby("category")["value"].transform("mean")
```
**Polars:**
```python
df = df.with_columns(
group_mean=pl.col("value").mean().over("category")
)
```
### Pattern 5: Multiple Aggregations
**Pandas:**
```python
result = df.groupby("category").agg({
"value": ["mean", "sum", "count"],
"price": ["min", "max"]
})
```
**Polars:**
```python
result = df.group_by("category").agg(
pl.col("value").mean().alias("value_mean"),
pl.col("value").sum().alias("value_sum"),
pl.col("value").count().alias("value_count"),
pl.col("price").min().alias("price_min"),
pl.col("price").max().alias("price_max")
)
```
## Performance Anti-Patterns to Avoid
### Anti-Pattern 1: Sequential Pipe Operations
**Bad (disables parallelization):**
```python
df = df.pipe(function1).pipe(function2).pipe(function3)
```
**Good (enables parallelization):**
```python
df = df.with_columns(
function1_result(),
function2_result(),
function3_result()
)
```
### Anti-Pattern 2: Python Functions in Hot Paths
**Bad:**
```python
df = df.with_columns(
result=pl.col("value").map_elements(lambda x: x * 2)
)
```
**Good:**
```python
df = df.with_columns(result=pl.col("value") * 2)
```
### Anti-Pattern 3: Using Eager Reading for Large Files
**Bad:**
```python
df = pl.read_csv("large_file.csv")
result = df.filter(pl.col("age") > 25).select("name", "age")
```
**Good:**
```python
lf = pl.scan_csv("large_file.csv")
result = lf.filter(pl.col("age") > 25).select("name", "age").collect()
```
### Anti-Pattern 4: Row Iteration
**Bad:**
```python
for row in df.iter_rows():
# Process row
pass
```
**Good:**
```python
# Use vectorized operations
df = df.with_columns(
# Vectorized computation
)
```
## Migration Checklist
When migrating from pandas to Polars:
1. **Remove index operations** - Use integer positions or group_by
2. **Replace apply/map with expressions** - Use Polars native operations
3. **Update column assignment** - Use `with_columns()` instead of direct assignment
4. **Change groupby.transform to .over()** - Window functions work differently
5. **Update string operations** - Use `.str.to_uppercase()` instead of `.str.upper()`
6. **Add explicit type casts** - Polars won't silently convert types
7. **Consider lazy evaluation** - Use `scan_*` instead of `read_*` for large data
8. **Update aggregation syntax** - More explicit in Polars
9. **Remove reset_index calls** - Not needed in Polars
10. **Update conditional logic** - Use `when().then().otherwise()` pattern
## Compatibility Layer
For gradual migration, you can use both libraries:
```python
import pandas as pd
import polars as pl
# Convert pandas to Polars
pl_df = pl.from_pandas(pd_df)
# Convert Polars to pandas
pd_df = pl_df.to_pandas()
# Use Arrow for zero-copy (when possible)
pl_df = pl.from_arrow(pd_df)
pd_df = pl_df.to_arrow().to_pandas()
```
## When to Stick with Pandas
Consider staying with pandas when:
- Working with time series requiring complex index operations
- Need extensive ecosystem support (some libraries only support pandas)
- Team lacks Rust/Polars expertise
- Data is small and performance isn't critical
- Using advanced pandas features without Polars equivalents
## When to Switch to Polars
Switch to Polars when:
- Performance is critical
- Working with large datasets (>1GB)
- Need lazy evaluation and query optimization
- Want better type safety
- Need parallel execution by default
- Starting a new project
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |