matlab/matlab-agentic-toolkit / skills-catalog/reporting-and-database-access/matlab-use-duckdb
matlab-use-duckdb
VerifiedUse DuckDB from MATLAB via Database Toolbox (R2026a+) as a non-math operations engine on large tabular files (CSV/Parquet/JSON) and as a zero-config embedded database. Use when connecting to DuckDB, querying CSV, Parquet, and JSON files directly with SQL, reducing or profiling large data before MATLAB analysis, creating portable development databases, or installing DuckDB extensions. Triggers on: DuckDB, duckdb(), large CSV/Parquet/JSON, file too large for readtable, filter/aggregate at source, deduplicate, reduce before analysis, profile large file, persistent file import, analytical engine, SQL on CSV, SQL on Parquet, SQL on JSON, query CSV with SQL, query Parquet with SQL, run SQL on files, SQL queries on files, query files directly, SQL without database, in-process SQL.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 15.2 KB |
| manifest.yaml | config | 621 B |
| reference/INDEX.md | markdown | 594 B |
| reference/cards/development-database.md | markdown | 2.6 KB |
| reference/cards/extensions.md | markdown | 1.8 KB |
| reference/cards/file-analytics.md | markdown | 3.4 KB |
| reference/cards/reduce-large-data.md | markdown | 3.2 KB |
reference/cards/reduce-large-data.md
View on GitHub# Reduce Large Data with DuckDB
DuckDB's role is reduction: take data that doesn't fit in memory, apply SQL operations to select, reshape, or clean it, and produce an in-memory result. Once the result is in memory, DuckDB's job is done.
## Workflow: Profile → Operate → Close
### Step 1: Profile
Query only the information needed to decide which reduction operation to apply. Do not run a generic "describe everything" query.
**Required:** Row count and column schema. Use `DESCRIBE SELECT * FROM ...` to get column names and types without reading the full file.
```matlab
conn = duckdb();
profile = fetch(conn, "SELECT COUNT(*) as row_count FROM read_parquet('orders.parquet')");
columns = fetch(conn, "DESCRIBE SELECT * FROM read_parquet('orders.parquet')");
```
**Conditional — query only if relevant to the reduction you're considering:**
| Considering | Query |
|-------------|-------|
| GROUP BY aggregation | `SELECT col, COUNT(*) FROM ... GROUP BY col` — check cardinality is much less than row count |
| WHERE filter | `SELECT MIN(col), MAX(col)` or `SELECT DISTINCT status` — check filterable range |
| Deduplication | `SELECT COUNT(*) - COUNT(DISTINCT key_col)` — check duplicate volume |
| NULL cleanup | `SELECT SUM(CASE WHEN col IS NULL THEN 1 ELSE 0 END)` — check if NULLs are prevalent |
### Step 2: Operate
Apply operations from the DO list to reduce data to in-memory size. Combine multiple operations in a single query when possible.
```matlab
reduced = fetch(conn, "SELECT " + ...
"customer_id, " + ...
"CAST(order_date AS DATE) as order_date, " + ...
"SUM(total) as daily_total, " + ...
"COUNT(*) as order_count " + ...
"FROM read_parquet('orders.parquet') " + ...
"WHERE status = 'completed' " + ...
" AND order_date >= '2024-01-01' " + ...
"GROUP BY customer_id, CAST(order_date AS DATE)");
```
### Step 3: Close
Close the connection immediately after the last fetch. The reduced result is now an in-memory table. DuckDB's role is complete.
```matlab
close(conn);
```
Do not keep the connection open for "just in case" follow-up queries. If more reduction is needed later, open a new connection.
## SQL/MATLAB Semantic Mismatches
This table is not exhaustive. Do not assume SQL operations match MATLAB semantics exactly — verify NULL/NaN handling, sort order, and boundary behavior before claiming equivalence.
| SQL Behavior | MATLAB Behavior | Mitigation |
|-------------|-----------------|------------|
| `NULL` propagates (NULL + 1 = NULL) | `NaN` propagates; `ismissing` detects it | Use `COALESCE(col, 0)` or filter NULLs before aggregation |
| `CAST` throws on invalid data | Type conversion produces NaN/NaT | Use `TRY_CAST` — returns NULL instead of erroring |
| `DISTINCT` treats NULLs as equal (collapses to one row) | `unique` treats NaN as distinct (`NaN ~= NaN`) | Use `TreatMissingAsDistinct=false` in MATLAB to match SQL behavior, or filter NULLs before `DISTINCT` |
| `EXTRACT(DOW FROM date)` → 0=Sunday | `weekday(dt)` → 1=Sunday (default) | Document which convention; remap after close if needed |
| String comparison is case-sensitive | `matches` is case-sensitive; use `IgnoreCase=true` for insensitive | Use `LOWER(col)` in SQL for case-insensitive matching |
----
Copyright 2026 The MathWorks, Inc.
----
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |