Skip to content

ericrisco/rsc-harness / skills/duckdb

duckdb

Flagged

Use when analytical SQL must run in-process with no server: Parquet/CSV/JSON/Arrow queried in place, OLAP embedded in an app or notebook, a slow pandas groupby on multi-GB data, or S3/lakehouse data read without downloading. NOT a multi-user analytics server (that is clickhouse-analytics), NOT an app's transactional CRUD store (that is postgresdb).

FileKindSize
SKILL.mdskill md10.5 KB
evals/README.mdmarkdown2.0 KB
evals/cases.yamlconfig4.0 KB
references/python-and-interop.mdmarkdown4.0 KB
references/remote-and-lakehouse.mdmarkdown3.1 KB
scripts/verify.shexecutablescript2.6 KB

references/remote-and-lakehouse.md

View on GitHub
# DuckDB — remote data & lakehouse

Offloaded from the SKILL body. DuckDB v1.5.2 / LTS 1.4.x.

## httpfs: read from S3 / GCS / Azure / HTTP in place

```sql
INSTALL httpfs;        -- one-time per database/install
LOAD httpfs;           -- per session
SELECT count(*) FROM read_parquet('https://host/path/data.parquet');  -- plain HTTP works immediately
```

`httpfs` streams remote files with HTTP range requests, so Parquet row-group and column pushdown still
apply — you fetch only the bytes you read, not the whole object.

## CREATE SECRET — credentials done right

Never inline keys into SQL. Use a secret; prefer the credential chain so creds come from the environment
or an instance/role rather than being typed.

```sql
-- Pick up env vars / instance role / SSO automatically (recommended):
CREATE SECRET s3 (TYPE s3, PROVIDER credential_chain);

-- Explicit, when you must (e.g. a non-default region / endpoint):
CREATE SECRET s3 (
    TYPE s3,
    KEY_ID '...', SECRET '...',
    REGION 'eu-west-1'
);

-- GCS (S3-compatible) and Azure have their own TYPEs:
CREATE SECRET gcs (TYPE gcs, KEY_ID '...', SECRET '...');
CREATE SECRET az  (TYPE azure, CONNECTION_STRING '...');
```

```sql
SELECT region, sum(amount) AS rev
FROM read_parquet('s3://bkt/sales/year=*/month=*/*.parquet', filename = true)
GROUP BY ALL;
```

## Hive-partitioned globs

A glob over `key=value/` directories auto-derives partition columns; filter on them and DuckDB prunes
whole directories before reading any data.

```sql
SELECT *
FROM read_parquet('s3://bkt/events/year=*/month=*/*.parquet', hive_partitioning = true)
WHERE year = 2025 AND month = 4;     -- only the matching partition dirs are scanned
```

## Iceberg

```sql
INSTALL iceberg; LOAD iceberg;
SELECT * FROM iceberg_scan('s3://bkt/warehouse/db/table');   -- read a table snapshot
```

## Delta Lake

```sql
INSTALL delta; LOAD delta;
SELECT * FROM delta_scan('s3://bkt/delta/events');
```

## DuckLake — DuckDB's own lakehouse catalog

DuckLake (ducklake.select) stores table metadata in a SQL catalog with Parquet data files — DuckDB's
native lakehouse format. Attach it and query like a normal schema.

```sql
INSTALL ducklake; LOAD ducklake;
ATTACH 'ducklake:metadata.ducklake' AS lake (DATA_PATH 's3://bkt/lake/');
USE lake;
CREATE TABLE sales AS FROM read_parquet('s3://bkt/raw/*.parquet');
```

## Partitioned write (handoff)

```sql
COPY (SELECT * FROM 'raw/*.parquet')
  TO 's3://bkt/curated/' (FORMAT parquet, PARTITION_BY (year, month), OVERWRITE_OR_IGNORE);
```

Writes a hive-partitioned dataset so any reader (DuckDB, Spark, ClickHouse, pandas) can prune partitions.

## MotherDuck — scale-out / share escape hatch

MotherDuck is managed DuckDB-as-a-service with hybrid local+cloud execution. Reach for it to share a
database or push heavy work to the cloud without running a server yourself — not the default.

```sql
ATTACH 'md:';                       -- connect to your MotherDuck account (auth via token)
USE my_cloud_db;
SELECT * FROM cloud_table LIMIT 10; -- query can run partly local, partly in the cloud
```

When the real need is many concurrent users or always-on production serving, that is
`clickhouse-analytics`, not MotherDuck.
deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall