ericrisco/rsc-harness / skills/duckdb
duckdb
FlaggedUse when analytical SQL must run in-process with no server: Parquet/CSV/JSON/Arrow queried in place, OLAP embedded in an app or notebook, a slow pandas groupby on multi-GB data, or S3/lakehouse data read without downloading. NOT a multi-user analytics server (that is clickhouse-analytics), NOT an app's transactional CRUD store (that is postgresdb).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 10.5 KB |
| evals/README.md | markdown | 2.0 KB |
| evals/cases.yaml | config | 4.0 KB |
| references/python-and-interop.md | markdown | 4.0 KB |
| references/remote-and-lakehouse.md | markdown | 3.1 KB |
| scripts/verify.shexecutable | script | 2.6 KB |
references/remote-and-lakehouse.md
View on GitHub# DuckDB — remote data & lakehouse
Offloaded from the SKILL body. DuckDB v1.5.2 / LTS 1.4.x.
## httpfs: read from S3 / GCS / Azure / HTTP in place
```sql
INSTALL httpfs; -- one-time per database/install
LOAD httpfs; -- per session
SELECT count(*) FROM read_parquet('https://host/path/data.parquet'); -- plain HTTP works immediately
```
`httpfs` streams remote files with HTTP range requests, so Parquet row-group and column pushdown still
apply — you fetch only the bytes you read, not the whole object.
## CREATE SECRET — credentials done right
Never inline keys into SQL. Use a secret; prefer the credential chain so creds come from the environment
or an instance/role rather than being typed.
```sql
-- Pick up env vars / instance role / SSO automatically (recommended):
CREATE SECRET s3 (TYPE s3, PROVIDER credential_chain);
-- Explicit, when you must (e.g. a non-default region / endpoint):
CREATE SECRET s3 (
TYPE s3,
KEY_ID '...', SECRET '...',
REGION 'eu-west-1'
);
-- GCS (S3-compatible) and Azure have their own TYPEs:
CREATE SECRET gcs (TYPE gcs, KEY_ID '...', SECRET '...');
CREATE SECRET az (TYPE azure, CONNECTION_STRING '...');
```
```sql
SELECT region, sum(amount) AS rev
FROM read_parquet('s3://bkt/sales/year=*/month=*/*.parquet', filename = true)
GROUP BY ALL;
```
## Hive-partitioned globs
A glob over `key=value/` directories auto-derives partition columns; filter on them and DuckDB prunes
whole directories before reading any data.
```sql
SELECT *
FROM read_parquet('s3://bkt/events/year=*/month=*/*.parquet', hive_partitioning = true)
WHERE year = 2025 AND month = 4; -- only the matching partition dirs are scanned
```
## Iceberg
```sql
INSTALL iceberg; LOAD iceberg;
SELECT * FROM iceberg_scan('s3://bkt/warehouse/db/table'); -- read a table snapshot
```
## Delta Lake
```sql
INSTALL delta; LOAD delta;
SELECT * FROM delta_scan('s3://bkt/delta/events');
```
## DuckLake — DuckDB's own lakehouse catalog
DuckLake (ducklake.select) stores table metadata in a SQL catalog with Parquet data files — DuckDB's
native lakehouse format. Attach it and query like a normal schema.
```sql
INSTALL ducklake; LOAD ducklake;
ATTACH 'ducklake:metadata.ducklake' AS lake (DATA_PATH 's3://bkt/lake/');
USE lake;
CREATE TABLE sales AS FROM read_parquet('s3://bkt/raw/*.parquet');
```
## Partitioned write (handoff)
```sql
COPY (SELECT * FROM 'raw/*.parquet')
TO 's3://bkt/curated/' (FORMAT parquet, PARTITION_BY (year, month), OVERWRITE_OR_IGNORE);
```
Writes a hive-partitioned dataset so any reader (DuckDB, Spark, ClickHouse, pandas) can prune partitions.
## MotherDuck — scale-out / share escape hatch
MotherDuck is managed DuckDB-as-a-service with hybrid local+cloud execution. Reach for it to share a
database or push heavy work to the cloud without running a server yourself — not the default.
```sql
ATTACH 'md:'; -- connect to your MotherDuck account (auth via token)
USE my_cloud_db;
SELECT * FROM cloud_table LIMIT 10; -- query can run partly local, partly in the cloud
```
When the real need is many concurrent users or always-on production serving, that is
`clickhouse-analytics`, not MotherDuck.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |