matlab/matlab-agentic-toolkit / skills-catalog/matlab-data-import-and-analysis/matlab-choose-big-data-solution
matlab-choose-big-data-solution
VerifiedGuide users or agents to the correct MATLAB tool for processing large tabular data in file-based formats (CSV, Parquet, delimited text, spreadsheets, MDF) that may not fit in memory. Use when a user or agent mentions large files, big data, out-of-memory errors, OOM, scaling up, tall arrays, datastores, or needs to process multiple tabular files. Covers the decision between datastore + tall, datastore + transform, and parallel execution. Also use when a user or agent has working in-memory code (readtable, parquetread) that runs out of memory and needs a migration path. Also covers building custom datastore classes for proprietary or non-standard formats — use when the task requires subclassing matlab.io.Datastore, implementing a custom reader, building an extensible datastore, or integrating a new file format with tall arrays or parallel computing. Do NOT use for MAT files (use matfile instead).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 18.8 KB |
| manifest.yaml | config | 576 B |
| references/custom-datastore/implementation.md | markdown | 20.0 KB |
| references/custom-datastore/mixin-decision-tree.md | markdown | 7.0 KB |
| references/custom-datastore/testing-guidelines.md | markdown | 5.2 KB |
| references/parquetread-to-datastore-migration.md | markdown | 2.2 KB |
| references/readtable-to-datastore-migration.md | markdown | 3.7 KB |
| references/tall-array-effective-use.md | markdown | 8.4 KB |
references/custom-datastore/testing-guidelines.md
View on GitHub# Custom Datastore Testing Guidelines
Write and run these tests after implementing a custom datastore. Tests verify
the datastore integrates correctly with the MATLAB ecosystem.
Reference: https://www.mathworks.com/help/matlab/import_export/testing-guidelines-for-custom-datastores.html
**When the task asks for tests, writing the test file is a required deliverable
— do not stop after the class.** Always include these core tests, in a separate
test-class file:
- `testConstruction`, `testReadReturnsData`, `testReadallReturnsAllData`
- `testHasdataExhaustion`
- **`testProgressRange`** — the single most commonly omitted test. Include it
verbatim from the template below: assert `progress(ds) == 0` *before* any
read, `progress(ds) >= 0 && progress(ds) <= 1` *during* the read loop, and
`progress(ds) == 1` *after* the datastore is exhausted. All three checks must
be present, and they belong in the test file — not inlined in the datastore
class.
Add the partition/shuffle tests below only for the mixins the class actually uses.
## Test Class Template
```matlab
classdef tMyDatastore < matlab.unittest.TestCase
properties (Constant)
DataPath = "path/to/test/data"
end
methods (Test)
function testConstruction(testCase)
ds = MyDatastore(testCase.DataPath);
testCase.verifyClass(ds, "MyDatastore");
testCase.verifyTrue(hasdata(ds));
end
function testReadReturnsData(testCase)
ds = MyDatastore(testCase.DataPath);
[data, info] = read(ds);
testCase.verifyNotEmpty(data);
testCase.verifyTrue(isstruct(info));
end
function testReadallReturnsAllData(testCase)
ds = MyDatastore(testCase.DataPath);
allData = readall(ds);
testCase.verifyNotEmpty(allData);
testCase.verifyTrue(hasdata(ds), ...
"readall must not advance the original datastore.");
end
function testPreviewDoesNotAdvancePosition(testCase)
ds = MyDatastore(testCase.DataPath);
preview(ds);
testCase.verifyTrue(hasdata(ds), ...
"preview must not advance the original datastore.");
[data, ~] = read(ds);
testCase.verifyNotEmpty(data);
end
function testResetAfterPartialRead(testCase)
ds = MyDatastore(testCase.DataPath);
read(ds);
read(ds);
reset(ds);
testCase.verifyTrue(hasdata(ds));
end
function testHasdataExhaustion(testCase)
ds = MyDatastore(testCase.DataPath);
while hasdata(ds)
read(ds);
end
testCase.verifyFalse(hasdata(ds));
end
function testProgressRange(testCase)
ds = MyDatastore(testCase.DataPath);
testCase.verifyEqual(progress(ds), 0);
while hasdata(ds)
read(ds);
p = progress(ds);
testCase.verifyGreaterThanOrEqual(p, 0);
testCase.verifyLessThanOrEqual(p, 1);
end
testCase.verifyEqual(progress(ds), 1);
end
end
end
```
## Partition Tests (if Partitionable)
```matlab
function testPartitionSumsToWhole(testCase)
ds = MyDatastore(testCase.DataPath);
allData = readall(ds);
numParts = 3;
partData = [];
for idx = 1:numParts
subds = partition(ds, numParts, idx);
partData = [partData; readall(subds)]; %#ok<AGROW>
end
testCase.verifyEqual(numel(partData), numel(allData));
end
function testPartitionDoesNotAffectOriginal(testCase)
ds = MyDatastore(testCase.DataPath);
partition(ds, 3, 1);
testCase.verifyTrue(hasdata(ds));
allData = readall(ds);
testCase.verifyNotEmpty(allData);
end
function testTallConstruction(testCase)
ds = MyDatastore(testCase.DataPath);
t = tall(ds);
testCase.verifyClass(t, "tall");
end
```
## Shuffle Tests (if Shuffleable)
```matlab
function testShuffleReturnsNewDatastore(testCase)
ds = MyDatastore(testCase.DataPath);
dsShuffled = shuffle(ds);
testCase.verifyClass(dsShuffled, "MyDatastore");
testCase.verifyTrue(hasdata(dsShuffled));
end
function testShuffleDoesNotAffectOriginal(testCase)
ds = MyDatastore(testCase.DataPath);
shuffle(ds);
testCase.verifyTrue(hasdata(ds));
end
```
## What Each Test Verifies
| Test | What it catches |
|------|----------------|
| readall doesn't advance position | Missing or broken `copyElement` |
| preview doesn't advance position | Missing or broken `copyElement` |
| partition sums to whole | Incorrect `partition`/`maxpartitions` delegation |
| partition doesn't affect original | `partition` mutates instead of copying |
| tall construction | Missing `Partitionable` mixin or broken `partition` |
| progress range | Incorrect progress calculation |
| hasdata exhaustion | Broken iteration logic in `read`/`hasdata` |
----
Copyright 2026 The MathWorks, Inc.
----
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| Agent Cost Reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| Agent Cost Reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |