matlab/matlab-agentic-toolkit / skills-catalog/matlab-data-import-and-analysis/matlab-choose-big-data-solution
matlab-choose-big-data-solution
VerifiedGuide users or agents to the correct MATLAB tool for processing large tabular data in file-based formats (CSV, Parquet, delimited text, spreadsheets, MDF) that may not fit in memory. Use when a user or agent mentions large files, big data, out-of-memory errors, OOM, scaling up, tall arrays, datastores, or needs to process multiple tabular files. Covers the decision between datastore + tall, datastore + transform, and parallel execution. Also use when a user or agent has working in-memory code (readtable, parquetread) that runs out of memory and needs a migration path. Also covers building custom datastore classes for proprietary or non-standard formats — use when the task requires subclassing matlab.io.Datastore, implementing a custom reader, building an extensible datastore, or integrating a new file format with tall arrays or parallel computing. Do NOT use for MAT files (use matfile instead).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 18.8 KB |
| manifest.yaml | config | 576 B |
| references/custom-datastore/implementation.md | markdown | 20.0 KB |
| references/custom-datastore/mixin-decision-tree.md | markdown | 7.0 KB |
| references/custom-datastore/testing-guidelines.md | markdown | 5.2 KB |
| references/parquetread-to-datastore-migration.md | markdown | 2.2 KB |
| references/readtable-to-datastore-migration.md | markdown | 3.7 KB |
| references/tall-array-effective-use.md | markdown | 8.4 KB |
references/parquetread-to-datastore-migration.md
View on GitHub# Migrating parquetread to parquetDatastore
Parquet is self-describing: types are encoded in file metadata and preserved
exactly by both `parquetread` and `parquetDatastore`. No format specifiers or
type conversion workarounds are needed.
## Property Mapping
All `parquetread` Name-Value pairs map 1:1 to writable datastore properties:
| `parquetread` option | `parquetDatastore` property | Notes |
|---------------------|----------------------------|-------|
| `SelectedVariableNames` | `pds.SelectedVariableNames` | Writable after creation |
| `RowFilter` | `pds.RowFilter` | Writable after creation |
| `OutputType` | `pds.OutputType` | Writable after creation |
| `RowTimes` | `pds.RowTimes` | Writable after creation |
| `VariableNamingRule` | `pds.VariableNamingRule` | Writable after creation |
| `RowGroups` | **No direct equivalent** | Datastore iterates row groups sequentially via `ReadSize="rowgroup"` |
All properties are writable after creation — no creation-time-only restrictions.
## Basic Migration Pattern
```matlab
% Before (OOM on large files):
T = parquetread("measurements.parquet", ...
SelectedVariableNames=["Timestamp","Value","Region"], ...
RowFilter=(rowfilter("Value").Value > 100));
stats = groupsummary(T, "Region", "mean", "Value");
% After (works on any size):
pds = parquetDatastore("measurements.parquet");
tt = tall(pds);
tt = tt(tt.Value > 100, ["Timestamp","Value","Region"]);
stats = gather(groupsummary(tt, "Region", "mean", "Value"));
```
Tall transparently leverages both projection pushdown and predicate pushdown
when the user indexes columns or filters rows. No explicit `SelectedVariableNames`
or `RowFilter` on the datastore is needed — indexing directly on the tall array
achieves the same I/O optimization.
## RowGroups Parameter
`parquetread(..., RowGroups=N)` reads specific row groups by index.
`parquetDatastore` has no equivalent — it iterates row groups sequentially when
`ReadSize="rowgroup"` (the default). For workflows that need specific row group
access, keep using `parquetread` with `parquetinfo` to identify target row
groups, or iterate the datastore and filter by position.
----
Copyright 2026 The MathWorks, Inc.
----
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |