brycewang-stanford/Auto-Empirical-Research-Skills / skills/17-DAAF-Contribution-Community-daaf/dot-claude/skills/polars
polars
VerifiedPolars DataFrame library for high-performance data manipulation. Lazy/eager execution, expressions, I/O (CSV, Parquet, JSON), aggregations, joins, string/datetime ops, pandas interop. Use for Polars DataFrames or reading/writing Parquet files.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 11.8 KB |
| references/aggregations-grouping.md | markdown | 9.5 KB |
| references/dataframes-series.md | markdown | 8.6 KB |
| references/expressions.md | markdown | 8.6 KB |
| references/gotchas.md | markdown | 8.1 KB |
| references/interop.md | markdown | 8.4 KB |
| references/io-data.md | markdown | 8.7 KB |
| references/joins-concat.md | markdown | 7.5 KB |
| references/performance.md | markdown | 8.7 KB |
| references/quickstart.md | markdown | 6.1 KB |
| references/strings-datetime-categorical.md | markdown | 10.3 KB |
references/interop.md
View on GitHub# Interoperability
## Pandas Integration
### Polars to Pandas
```python
# Basic conversion
pandas_df = polars_df.to_pandas()
# With PyArrow backend (recommended - preserves types better)
pandas_df = polars_df.to_pandas(use_pyarrow_extension_array=True)
# Series to pandas Series
pandas_series = polars_series.to_pandas()
```
### Pandas to Polars
```python
import pandas as pd
import polars as pl
# From pandas DataFrame
polars_df = pl.from_pandas(pandas_df)
# From pandas Series
polars_series = pl.from_pandas(pandas_series)
# With schema override
polars_df = pl.from_pandas(pandas_df, schema_overrides={"id": pl.Int32})
```
### Type Mapping
| Pandas Type | Polars Type |
|-------------|-------------|
| `int64` | `pl.Int64` |
| `float64` | `pl.Float64` |
| `object` (strings) | `pl.String` |
| `bool` | `pl.Boolean` |
| `datetime64[ns]` | `pl.Datetime` |
| `timedelta64[ns]` | `pl.Duration` |
| `category` | `pl.Categorical` |
### Common Gotchas
```python
# Pandas uses NaN for missing values in numeric columns
# Polars uses null - conversion handles this automatically
# String columns with None in pandas become pl.String with null
# Make sure to handle null values appropriately
# Index is NOT converted - use reset_index() first if needed
pandas_df = pandas_df.reset_index()
polars_df = pl.from_pandas(pandas_df)
```
## NumPy Integration
### Polars to NumPy
```python
import numpy as np
# Series to numpy array
arr = polars_series.to_numpy()
# DataFrame to 2D numpy array
arr = polars_df.to_numpy()
# Specific column to numpy
arr = polars_df["column_name"].to_numpy()
# Allow copy (default) or zero-copy if possible
arr = polars_series.to_numpy(allow_copy=False) # Raises if copy needed
```
### NumPy to Polars
```python
# From numpy array
arr = np.array([1, 2, 3, 4, 5])
series = pl.Series("values", arr)
# 2D array to DataFrame
arr = np.array([[1, 2], [3, 4], [5, 6]])
df = pl.DataFrame(arr, schema=["a", "b"])
# From dict of numpy arrays
df = pl.DataFrame({
"a": np.array([1, 2, 3]),
"b": np.array([4.0, 5.0, 6.0])
})
```
### Using NumPy Functions
```python
# NumPy ufuncs work on Polars expressions
df.with_columns(
np.log(pl.col("value")).alias("log_value"),
np.sqrt(pl.col("value")).alias("sqrt_value"),
np.exp(pl.col("value")).alias("exp_value"),
)
# Note: This goes through Python/NumPy, prefer native Polars when available
# Polars native:
df.with_columns(
pl.col("value").log().alias("log_value"),
pl.col("value").sqrt().alias("sqrt_value"),
pl.col("value").exp().alias("exp_value"),
)
```
## PyArrow Integration
PyArrow provides zero-copy conversion when possible.
### Polars to Arrow
```python
import pyarrow as pa
# DataFrame to Arrow Table
arrow_table = polars_df.to_arrow()
# Series to Arrow Array
arrow_array = polars_series.to_arrow()
# Chunked (for large data)
arrow_table = polars_df.to_arrow() # Already chunked internally
```
### Arrow to Polars
```python
# From Arrow Table
arrow_table = pa.table({"a": [1, 2, 3], "b": ["x", "y", "z"]})
polars_df = pl.from_arrow(arrow_table)
# From Arrow Array
arrow_array = pa.array([1, 2, 3])
polars_series = pl.from_arrow(arrow_array, schema={"values": pl.Int64})
# From RecordBatch
polars_df = pl.from_arrow(record_batch)
```
### Zero-Copy Benefits
```python
# Arrow is Polars' native memory format
# Conversion is often zero-copy (no data copying)
arrow_table = polars_df.to_arrow() # Fast, shares memory
polars_df = pl.from_arrow(arrow_table) # Fast, shares memory
```
## DuckDB Integration
DuckDB and Polars can share data efficiently via Arrow.
### Query Polars DataFrame with DuckDB
```python
import duckdb
import polars as pl
df = pl.DataFrame({
"id": [1, 2, 3],
"name": ["Alice", "Bob", "Charlie"],
"value": [100, 200, 300]
})
# Method 1: Register and query
con = duckdb.connect()
con.register("my_table", df.to_arrow())
result = con.execute("SELECT * FROM my_table WHERE value > 100").arrow()
result_df = pl.from_arrow(result)
# Method 2: Direct query (DuckDB 0.8+)
result = duckdb.query("SELECT * FROM df WHERE value > 100").pl()
```
### Use DuckDB for Complex SQL
```python
# For complex SQL that's easier in SQL syntax
result = duckdb.query("""
WITH ranked AS (
SELECT *,
ROW_NUMBER() OVER (PARTITION BY category ORDER BY value DESC) as rn
FROM df
)
SELECT * FROM ranked WHERE rn <= 3
""").pl()
```
### Polars SQL Interface
Polars has built-in SQL support:
```python
# Query single DataFrame
result = df.sql("SELECT name, SUM(value) as total FROM self GROUP BY name")
# SQL Context for multiple tables
ctx = pl.SQLContext({"users": users_df, "orders": orders_df})
result = ctx.execute("""
SELECT u.name, COUNT(*) as order_count
FROM users u
JOIN orders o ON u.id = o.user_id
GROUP BY u.name
""").collect()
```
## Database Connections
### SQLAlchemy
```python
from sqlalchemy import create_engine
# Create engine
engine = create_engine("postgresql://user:pass@host:5432/database")
# Read with SQL
df = pl.read_database("SELECT * FROM users", engine)
# Write (via pandas for now)
df.to_pandas().to_sql("table_name", engine, if_exists="replace")
```
### ConnectorX (Recommended for Speed)
```python
# Requires: pip install connectorx
# Direct connection string
df = pl.read_database_uri(
"SELECT * FROM users WHERE active = true",
"postgresql://user:pass@host:5432/database"
)
# Supported databases:
# PostgreSQL, MySQL, SQLite, SQL Server, Oracle, Redshift, ClickHouse
```
### ADBC (Arrow Database Connectivity)
```python
# Modern, high-performance option
# pip install adbc-driver-postgresql
import adbc_driver_postgresql.dbapi as pg_dbapi
with pg_dbapi.connect("postgresql://user:pass@host/db") as conn:
df = pl.read_database("SELECT * FROM users", conn)
```
## Scikit-learn Integration
```python
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
# Polars for data prep
df = (
pl.scan_parquet("data.parquet")
.select(["feature1", "feature2", "feature3", "target"])
.drop_nulls()
.collect()
)
# Convert to numpy for sklearn
X = df.select(pl.exclude("target")).to_numpy()
y = df["target"].to_numpy()
# Train model
X_train, X_test, y_train, y_test = train_test_split(X, y)
model = RandomForestClassifier()
model.fit(X_train, y_train)
# Predictions back to Polars
predictions = model.predict(X_test)
result_df = pl.DataFrame({
"prediction": predictions,
"actual": y_test
})
```
## Visualization Libraries
### Matplotlib
```python
import matplotlib.pyplot as plt
# Convert column to numpy for plotting
plt.plot(df["x"].to_numpy(), df["y"].to_numpy())
plt.show()
# Or convert to pandas
df.to_pandas().plot(x="date", y="value")
```
### Altair
```python
import altair as alt
# Altair works with pandas
chart = alt.Chart(df.to_pandas()).mark_point().encode(
x="x:Q",
y="y:Q",
color="category:N"
)
```
### Plotly
```python
import plotly.express as px
# Plotly works with pandas
fig = px.scatter(df.to_pandas(), x="x", y="y", color="category")
fig.show()
```
## Common Patterns
### Mixed Pipeline
```python
import polars as pl
import pandas as pd
# Start with Polars for efficient data loading and transformation
df = (
pl.scan_parquet("data.parquet")
.filter(pl.col("date") > "2024-01-01")
.group_by("category")
.agg(pl.col("value").sum())
.collect()
)
# Convert to pandas for specific library that requires it
pandas_df = df.to_pandas()
# Use pandas-specific functionality
result = some_pandas_only_function(pandas_df)
# Convert back to Polars if needed
polars_result = pl.from_pandas(result)
```
### Efficient Data Pipeline
```python
# Best practice: Use Polars as much as possible
# Only convert at boundaries (I/O, specific libraries)
# Read with Polars (fast)
df = pl.scan_parquet("input.parquet")
# Transform with Polars (fast, parallel)
df = df.filter(...).with_columns(...).group_by(...).agg(...)
# Only convert when necessary
if need_sklearn:
X = df.select(features).to_numpy()
if need_specific_pandas_function:
pandas_df = df.to_pandas()
# Write with Polars (fast)
df.collect().write_parquet("output.parquet")
```
## Summary
| Source | To Polars | From Polars |
|--------|-----------|-------------|
| Pandas | `pl.from_pandas(df)` | `df.to_pandas()` |
| NumPy | `pl.Series(arr)` / `pl.DataFrame(arr)` | `df.to_numpy()` |
| PyArrow | `pl.from_arrow(table)` | `df.to_arrow()` |
| DuckDB | `duckdb.query(...).pl()` | `conn.register("t", df.to_arrow())` |
| Database | `pl.read_database(sql, conn)` | Via pandas or Arrow |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |