Skip to content

matlab/matlab-agentic-toolkit / skills-catalog/matlab-data-import-and-analysis/matlab-choose-big-data-solution

matlab-choose-big-data-solution

Verified

Guide users or agents to the correct MATLAB tool for processing large tabular data in file-based formats (CSV, Parquet, delimited text, spreadsheets, MDF) that may not fit in memory. Use when a user or agent mentions large files, big data, out-of-memory errors, OOM, scaling up, tall arrays, datastores, or needs to process multiple tabular files. Covers the decision between datastore + tall, datastore + transform, and parallel execution. Also use when a user or agent has working in-memory code (readtable, parquetread) that runs out of memory and needs a migration path. Also covers building custom datastore classes for proprietary or non-standard formats — use when the task requires subclassing matlab.io.Datastore, implementing a custom reader, building an extensible datastore, or integrating a new file format with tall arrays or parallel computing. Do NOT use for MAT files (use matfile instead).

references/custom-datastore/testing-guidelines.md

View on GitHub
# Custom Datastore Testing Guidelines

Write and run these tests after implementing a custom datastore. Tests verify
the datastore integrates correctly with the MATLAB ecosystem.

Reference: https://www.mathworks.com/help/matlab/import_export/testing-guidelines-for-custom-datastores.html

**When the task asks for tests, writing the test file is a required deliverable
— do not stop after the class.** Always include these core tests, in a separate
test-class file:

- `testConstruction`, `testReadReturnsData`, `testReadallReturnsAllData`
- `testHasdataExhaustion`
- **`testProgressRange`** — the single most commonly omitted test. Include it
  verbatim from the template below: assert `progress(ds) == 0` *before* any
  read, `progress(ds) >= 0 && progress(ds) <= 1` *during* the read loop, and
  `progress(ds) == 1` *after* the datastore is exhausted. All three checks must
  be present, and they belong in the test file — not inlined in the datastore
  class.

Add the partition/shuffle tests below only for the mixins the class actually uses.

## Test Class Template

```matlab
classdef tMyDatastore < matlab.unittest.TestCase

    properties (Constant)
        DataPath = "path/to/test/data"
    end

    methods (Test)
        function testConstruction(testCase)
            ds = MyDatastore(testCase.DataPath);
            testCase.verifyClass(ds, "MyDatastore");
            testCase.verifyTrue(hasdata(ds));
        end

        function testReadReturnsData(testCase)
            ds = MyDatastore(testCase.DataPath);
            [data, info] = read(ds);
            testCase.verifyNotEmpty(data);
            testCase.verifyTrue(isstruct(info));
        end

        function testReadallReturnsAllData(testCase)
            ds = MyDatastore(testCase.DataPath);
            allData = readall(ds);
            testCase.verifyNotEmpty(allData);
            testCase.verifyTrue(hasdata(ds), ...
                "readall must not advance the original datastore.");
        end

        function testPreviewDoesNotAdvancePosition(testCase)
            ds = MyDatastore(testCase.DataPath);
            preview(ds);
            testCase.verifyTrue(hasdata(ds), ...
                "preview must not advance the original datastore.");
            [data, ~] = read(ds);
            testCase.verifyNotEmpty(data);
        end

        function testResetAfterPartialRead(testCase)
            ds = MyDatastore(testCase.DataPath);
            read(ds);
            read(ds);
            reset(ds);
            testCase.verifyTrue(hasdata(ds));
        end

        function testHasdataExhaustion(testCase)
            ds = MyDatastore(testCase.DataPath);
            while hasdata(ds)
                read(ds);
            end
            testCase.verifyFalse(hasdata(ds));
        end

        function testProgressRange(testCase)
            ds = MyDatastore(testCase.DataPath);
            testCase.verifyEqual(progress(ds), 0);
            while hasdata(ds)
                read(ds);
                p = progress(ds);
                testCase.verifyGreaterThanOrEqual(p, 0);
                testCase.verifyLessThanOrEqual(p, 1);
            end
            testCase.verifyEqual(progress(ds), 1);
        end
    end
end
```

## Partition Tests (if Partitionable)

```matlab
        function testPartitionSumsToWhole(testCase)
            ds = MyDatastore(testCase.DataPath);
            allData = readall(ds);
            numParts = 3;

            partData = [];
            for idx = 1:numParts
                subds = partition(ds, numParts, idx);
                partData = [partData; readall(subds)]; %#ok<AGROW>
            end
            testCase.verifyEqual(numel(partData), numel(allData));
        end

        function testPartitionDoesNotAffectOriginal(testCase)
            ds = MyDatastore(testCase.DataPath);
            partition(ds, 3, 1);
            testCase.verifyTrue(hasdata(ds));
            allData = readall(ds);
            testCase.verifyNotEmpty(allData);
        end

        function testTallConstruction(testCase)
            ds = MyDatastore(testCase.DataPath);
            t = tall(ds);
            testCase.verifyClass(t, "tall");
        end
```

## Shuffle Tests (if Shuffleable)

```matlab
        function testShuffleReturnsNewDatastore(testCase)
            ds = MyDatastore(testCase.DataPath);
            dsShuffled = shuffle(ds);
            testCase.verifyClass(dsShuffled, "MyDatastore");
            testCase.verifyTrue(hasdata(dsShuffled));
        end

        function testShuffleDoesNotAffectOriginal(testCase)
            ds = MyDatastore(testCase.DataPath);
            shuffle(ds);
            testCase.verifyTrue(hasdata(ds));
        end
```

## What Each Test Verifies

| Test | What it catches |
|------|----------------|
| readall doesn't advance position | Missing or broken `copyElement` |
| preview doesn't advance position | Missing or broken `copyElement` |
| partition sums to whole | Incorrect `partition`/`maxpartitions` delegation |
| partition doesn't affect original | `partition` mutates instead of copying |
| tall construction | Missing `Partitionable` mixin or broken `partition` |
| progress range | Incorrect progress calculation |
| hasdata exhaustion | Broken iteration logic in `read`/`hasdata` |

----

Copyright 2026 The MathWorks, Inc.

----
deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
Agent Cost Reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
Agent Cost Reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall