brycewang-stanford/Auto-Empirical-Research-Skills / skills/17-DAAF-Contribution-Community-daaf/dot-claude/skills/polars
polars
VerifiedPolars DataFrame library for high-performance data manipulation. Lazy/eager execution, expressions, I/O (CSV, Parquet, JSON), aggregations, joins, string/datetime ops, pandas interop. Use for Polars DataFrames or reading/writing Parquet files.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 11.8 KB |
| references/aggregations-grouping.md | markdown | 9.5 KB |
| references/dataframes-series.md | markdown | 8.6 KB |
| references/expressions.md | markdown | 8.6 KB |
| references/gotchas.md | markdown | 8.1 KB |
| references/interop.md | markdown | 8.4 KB |
| references/io-data.md | markdown | 8.7 KB |
| references/joins-concat.md | markdown | 7.5 KB |
| references/performance.md | markdown | 8.7 KB |
| references/quickstart.md | markdown | 6.1 KB |
| references/strings-datetime-categorical.md | markdown | 10.3 KB |
references/io-data.md
View on GitHub# Data I/O
## CSV Files
### Read CSV (Eager)
```python
df = pl.read_csv("data.csv")
# With options
df = pl.read_csv(
"data.csv",
separator=",", # Delimiter (default: ",")
has_header=True, # First row is header (default: True)
skip_rows=1, # Skip first N rows
n_rows=1000, # Only read first N rows
columns=["a", "b", "c"], # Only these columns
schema_overrides={"id": pl.Int32}, # Override types ("dtypes" renamed to "schema_overrides" in 0.20.31)
null_values=["NA", ""], # Treat as null
ignore_errors=True, # Skip bad rows
encoding="utf8", # Encoding (default: utf8)
)
```
### Scan CSV (Lazy - Recommended)
```python
lf = pl.scan_csv("data.csv")
# With options
lf = pl.scan_csv(
"data.csv",
has_header=True,
separator=",",
skip_rows=0,
n_rows=None, # None = all rows
schema_overrides={"id": pl.Int32},
null_values=["NA"],
infer_schema_length=10000, # Rows to infer types (default: 100)
low_memory=False, # Reduce memory at cost of speed
)
df = lf.collect()
```
### Write CSV
```python
df.write_csv("output.csv")
# With options
df.write_csv(
"output.csv",
separator=",",
include_header=True,
null_value="", # How to write nulls
datetime_format="%Y-%m-%d", # Format for datetime
float_precision=6, # Decimal places
)
```
### Streaming Write (Large Data)
```python
lf.sink_csv("output.csv")
```
## Parquet Files (Recommended)
Parquet is the preferred format: compressed, fast, type-preserving.
### Read Parquet (Eager)
```python
df = pl.read_parquet("data.parquet")
# Specific columns only
df = pl.read_parquet("data.parquet", columns=["a", "b"])
# Row groups
df = pl.read_parquet("data.parquet", n_rows=1000)
```
### Scan Parquet (Lazy - Best for Large Files)
```python
lf = pl.scan_parquet("data.parquet")
# Benefits: predicate/projection pushdown
result = (
pl.scan_parquet("huge.parquet")
.filter(pl.col("date") > "2024-01-01") # Only reads matching rows
.select("id", "value") # Only reads these columns
.collect()
)
```
### Write Parquet
```python
df.write_parquet("output.parquet")
# With compression
df.write_parquet(
"output.parquet",
compression="zstd", # "zstd", "snappy", "gzip", "lz4", "uncompressed"
compression_level=3, # 1-22 for zstd (higher = smaller, slower)
statistics=True, # Include column statistics
row_group_size=100_000, # Rows per group
)
```
### Streaming Write
```python
lf.sink_parquet("output.parquet", compression="zstd")
```
## JSON Files
### Read JSON
```python
# Standard JSON (array of objects)
df = pl.read_json("data.json")
```
### Read NDJSON (Newline-Delimited JSON)
```python
# Better for large files, streaming
df = pl.read_ndjson("data.ndjson")
# Lazy scan
lf = pl.scan_ndjson("data.ndjson")
```
### Write JSON/NDJSON
```python
df.write_json("output.json")
df.write_ndjson("output.ndjson")
# Streaming
lf.sink_ndjson("output.ndjson")
```
## Multiple Files (Glob Patterns)
### Read Multiple Files
```python
# All CSVs in directory
lf = pl.scan_csv("data/*.csv")
# Recursive glob
lf = pl.scan_parquet("data/**/*.parquet")
# With pattern matching
lf = pl.scan_csv("data/sales_202[34]*.csv")
```
### Add Source File Column
```python
lf = pl.scan_csv("data/*.csv", include_file_paths="source_file")
# Adds column with the file path for each row
```
### Hive Partitioned Data
```python
# For directory structure: data/year=2024/month=01/data.parquet
lf = pl.scan_parquet("data/**/*.parquet", hive_partitioning=True)
# Automatically creates year and month columns from path
```
## Excel Files
```python
# Requires: pip install xlsx2csv or pip install openpyxl
df = pl.read_excel("data.xlsx")
# Specific sheet
df = pl.read_excel("data.xlsx", sheet_name="Sheet2")
df = pl.read_excel("data.xlsx", sheet_id=1) # 1-indexed
# Read all sheets
sheets = pl.read_excel("data.xlsx", sheet_id=0) # Returns dict
# Write Excel
df.write_excel("output.xlsx")
```
## Database Connections
### SQLite
```python
# Read
df = pl.read_database(
"SELECT * FROM users WHERE active = 1",
"sqlite:///database.db"
)
# Or with connection object
import sqlite3
conn = sqlite3.connect("database.db")
df = pl.read_database("SELECT * FROM users", conn)
```
### PostgreSQL
```python
# Requires: pip install connectorx
df = pl.read_database(
"SELECT * FROM users",
"postgresql://user:pass@host:5432/database"
)
# With query parameters
df = pl.read_database_uri(
"SELECT * FROM users WHERE id = $1",
"postgresql://user:pass@host/db",
execute_options={"parameters": [42]}
)
```
### MySQL
```python
df = pl.read_database(
"SELECT * FROM users",
"mysql://user:pass@host:3306/database"
)
```
### Generic ADBC (Arrow Database Connectivity)
```python
# Modern, high-performance option
import adbc_driver_postgresql.dbapi
conn = adbc_driver_postgresql.dbapi.connect("postgresql://...")
df = pl.read_database("SELECT * FROM users", conn)
```
## Cloud Storage
### AWS S3
```python
# Requires: pip install polars[fsspec] and s3fs
# Read directly
df = pl.read_parquet("s3://bucket/path/data.parquet")
lf = pl.scan_parquet("s3://bucket/path/*.parquet")
# With credentials (via environment or explicit)
import s3fs
fs = s3fs.S3FileSystem(
key="ACCESS_KEY",
secret="SECRET_KEY",
endpoint_url="https://s3.region.amazonaws.com"
)
df = pl.read_parquet("s3://bucket/data.parquet", storage_options={"fs": fs})
```
### Google Cloud Storage
```python
# Requires: pip install gcsfs
df = pl.read_parquet("gs://bucket/data.parquet")
lf = pl.scan_parquet("gs://bucket/path/*.parquet")
```
### Azure Blob Storage
```python
# Requires: pip install adlfs
df = pl.read_parquet("az://container/data.parquet")
lf = pl.scan_parquet("abfss://container@account.dfs.core.windows.net/path/*.parquet")
```
## HTTP/URLs
```python
# Read directly from URL
df = pl.read_csv("https://example.com/data.csv")
df = pl.read_parquet("https://example.com/data.parquet")
# Large files - download first for better performance
import urllib.request
urllib.request.urlretrieve("https://example.com/large.parquet", "local.parquet")
df = pl.read_parquet("local.parquet")
```
## IPC (Arrow/Feather)
```python
# Fast binary format, good for inter-process communication
# Read
df = pl.read_ipc("data.arrow")
df = pl.read_ipc("data.feather")
# Lazy scan
lf = pl.scan_ipc("data.arrow")
# Write
df.write_ipc("output.arrow")
df.write_ipc("output.feather", compression="zstd")
```
## Avro Files
```python
# Avro support is built into polars (no extra install needed)
df = pl.read_avro("data.avro")
df.write_avro("output.avro", compression="snappy")
```
## Delta Lake
```python
# Requires: pip install deltalake
df = pl.read_delta("delta_table_path/")
lf = pl.scan_delta("delta_table_path/")
# With version
# Version is passed directly: pl.read_delta("path/", version=5)
df = pl.read_delta("delta_table/", version=5)
```
## Common Patterns
### Type Inference Control
```python
# More rows for type inference
lf = pl.scan_csv("data.csv", infer_schema_length=100000)
# Disable inference (use explicit schema)
df = pl.read_csv("data.csv", infer_schema=False) # All strings
# Explicit schema
df = pl.read_csv(
"data.csv",
schema={
"id": pl.Int64,
"name": pl.String,
"date": pl.Date,
"value": pl.Float64
}
)
```
### Handling Large Files
```python
# Use lazy + streaming
result = (
pl.scan_csv("huge_file.csv")
.filter(pl.col("category") == "A")
.group_by("region")
.agg(pl.col("sales").sum())
.collect(streaming=True)
)
# Or sink directly to file
(
pl.scan_csv("input.csv")
.filter(pl.col("active"))
.sink_parquet("output.parquet")
)
```
### Reading Specific Rows/Columns
```python
# Only specific columns
df = pl.read_parquet("data.parquet", columns=["id", "name", "value"])
# Only first N rows
df = pl.read_csv("data.csv", n_rows=1000)
# Skip rows
df = pl.read_csv("data.csv", skip_rows=10, skip_rows_after_header=5)
```
## Summary
| Format | Read (Eager) | Scan (Lazy) | Write | Best For |
|--------|--------------|-------------|-------|----------|
| CSV | `read_csv` | `scan_csv` | `write_csv` | Interchange |
| Parquet | `read_parquet` | `scan_parquet` | `write_parquet` | Production (recommended) |
| JSON | `read_json` | - | `write_json` | APIs |
| NDJSON | `read_ndjson` | `scan_ndjson` | `write_ndjson` | Streaming JSON |
| IPC/Arrow | `read_ipc` | `scan_ipc` | `write_ipc` | Inter-process |
| Excel | `read_excel` | - | `write_excel` | Spreadsheets |
| Database | `read_database` | - | - | SQL queries |
**Key recommendation**: Use Parquet with lazy scanning (`scan_parquet`) for large datasets.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |