matlab/matlab-agentic-toolkit / skills-catalog/matlab-data-import-and-analysis/matlab-choose-big-data-solution
matlab-choose-big-data-solution
VerifiedGuide users or agents to the correct MATLAB tool for processing large tabular data in file-based formats (CSV, Parquet, delimited text, spreadsheets, MDF) that may not fit in memory. Use when a user or agent mentions large files, big data, out-of-memory errors, OOM, scaling up, tall arrays, datastores, or needs to process multiple tabular files. Covers the decision between datastore + tall, datastore + transform, and parallel execution. Also use when a user or agent has working in-memory code (readtable, parquetread) that runs out of memory and needs a migration path. Also covers building custom datastore classes for proprietary or non-standard formats — use when the task requires subclassing matlab.io.Datastore, implementing a custom reader, building an extensible datastore, or integrating a new file format with tall arrays or parallel computing. Do NOT use for MAT files (use matfile instead).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 18.8 KB |
| manifest.yaml | config | 576 B |
| references/custom-datastore/implementation.md | markdown | 20.0 KB |
| references/custom-datastore/mixin-decision-tree.md | markdown | 7.0 KB |
| references/custom-datastore/testing-guidelines.md | markdown | 5.2 KB |
| references/parquetread-to-datastore-migration.md | markdown | 2.2 KB |
| references/readtable-to-datastore-migration.md | markdown | 3.7 KB |
| references/tall-array-effective-use.md | markdown | 8.4 KB |
references/readtable-to-datastore-migration.md
View on GitHub# Migrating readtable to tabularTextDatastore
`tabularTextDatastore` uses a textscan-based parser that is stricter and less
forgiving than `readtable`. When migrating code that works on small data, watch
for these differences.
## Property Mapping
Common `readtable` options and their datastore equivalents:
| `readtable` option | `tabularTextDatastore` equivalent | Notes |
|--------------------|-----------------------------------|-------|
| `TextType="string"` | `TextType="string"` | Must set at **creation time** — read-only after construction |
| `Delimiter` | `ds.Delimiter` | Writable after creation |
| `NumHeaderLines` | `ds.NumHeaderLines` | Writable after creation |
| `TreatAsMissing` | `ds.TreatAsMissing` | Writable after creation |
| `DateLocale` | `ds.DatetimeLocale` | Writable after creation |
| `opts.SelectedVariableNames` | `ds.SelectedVariableNames` | Writable after creation |
| `setvartype(opts, var, type)` | `ds.TextscanFormats{col}` | See format specifiers below |
| `TrimNonNumeric` | **Not supported** | See "Numbers with units" below |
| `ExtraColumnsRule` | **Not applicable** | See "Extra columns" below |
| `detectImportOptions` | Not used — datastore auto-detects | Configure via properties instead |
## Type Overrides via TextscanFormats
The `setvartype` idiom from import options maps to textscan format specifiers:
| readtable type | TextscanFormats specifier |
|---------------|---------------------------|
| `"double"` | `%f` |
| `"char"` / `"string"` | `%q` (reads as cell of char or string depending on TextType) |
| `"datetime"` | `%{dd-MMM-uuuu}D` (format inside braces) |
| `"duration"` | `%T` |
| `"categorical"` | `%C` |
| `"logical"` | **No equivalent** — read as `%q`, convert to logical after reading |
```matlab
% Override a column type (e.g., force column 3 to categorical):
ds = tabularTextDatastore("data.csv", TextType="string");
ds.TextscanFormats{3} = '%C';
```
## Numbers with Units (TrimNonNumeric)
`readtable` with `TrimNonNumeric` can parse "34mm" as 34.
`tabularTextDatastore` cannot. Workaround: use `%q` to read the column as text,
then convert to numeric in a transform function or on the tall array, depending
on which workflow from this skill is being used. This handles mixed columns
(some values with units, some without) correctly because `%q` reads all values
as plain text regardless of content.
## Missing or Malformed Data
`readtable` silently inserts `NaN` or `missing` for unparseable values.
`tabularTextDatastore` with `%f` errors on non-numeric content.
Workaround: read the column as `%q` (text), then convert with `str2double` or
`double(string(...))` — either in a transform function or on the tall array.
This avoids the parse error but hurts performance due to the string-to-number
conversion.
## Extra Columns
`readtable` has `ExtraColumnsRule` to handle rows with more columns than the
header. `tabularTextDatastore` has no equivalent and will error.
Workaround: append `"%*[^\r\n]"` to `TextscanFormats` at creation time — this
reads and discards all characters to the end of the line, ignoring extra columns.
The `*` means "ignore," which deselects the extra content from
`SelectedVariableNames`. This requires `ReadVariableNames=false` and
`NumHeaderLines` set manually so the datastore does not infer column count from
the header:
```matlab
ds = tabularTextDatastore("data.csv", TextType="string", ...
ReadVariableNames=false, NumHeaderLines=1, ...
TextscanFormats={'%q', '%f', '%*[^\r\n]'});
```
This approach works when extra columns are consistently present across rows. For
files where the number of columns varies unpredictably between rows, a custom
datastore or preprocessing step is required.
----
Copyright 2026 The MathWorks, Inc.
----
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| Agent Cost Reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| Agent Cost Reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |