matlab/matlab-agentic-toolkit / skills-catalog/matlab-data-import-and-analysis/matlab-choose-big-data-solution
matlab-choose-big-data-solution
VerifiedGuide users or agents to the correct MATLAB tool for processing large tabular data in file-based formats (CSV, Parquet, delimited text, spreadsheets, MDF) that may not fit in memory. Use when a user or agent mentions large files, big data, out-of-memory errors, OOM, scaling up, tall arrays, datastores, or needs to process multiple tabular files. Covers the decision between datastore + tall, datastore + transform, and parallel execution. Also use when a user or agent has working in-memory code (readtable, parquetread) that runs out of memory and needs a migration path. Also covers building custom datastore classes for proprietary or non-standard formats — use when the task requires subclassing matlab.io.Datastore, implementing a custom reader, building an extensible datastore, or integrating a new file format with tall arrays or parallel computing. Do NOT use for MAT files (use matfile instead).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 18.8 KB |
| manifest.yaml | config | 576 B |
| references/custom-datastore/implementation.md | markdown | 20.0 KB |
| references/custom-datastore/mixin-decision-tree.md | markdown | 7.0 KB |
| references/custom-datastore/testing-guidelines.md | markdown | 5.2 KB |
| references/parquetread-to-datastore-migration.md | markdown | 2.2 KB |
| references/readtable-to-datastore-migration.md | markdown | 3.7 KB |
| references/tall-array-effective-use.md | markdown | 8.4 KB |
references/custom-datastore/mixin-decision-tree.md
View on GitHub# Mixin Decision Tree
## Complete API Reference
| Class / Utility | Type | Purpose | Since |
|----------------|------|---------|-------|
| `matlab.io.Datastore` | Base class | Abstract base for all custom datastores | R2017b |
| `matlab.io.datastore.Partitionable` | Mixin | Add parallelization support (tall, parallel pools) | R2017b |
| `matlab.io.datastore.Subsettable` | Mixin | Add subset and fine-grained parallelization support (subsumes Partitionable + Shuffleable) | R2022b |
| `matlab.io.datastore.HadoopLocationBased` | Mixin | Add Hadoop/Spark support for HDFS data | R2017b |
| `matlab.io.datastore.Shuffleable` | Mixin | Add shuffling support for deep learning training | R2019b |
| `matlab.io.datastore.FileWritable` | Mixin | Add file writing support (`writeall`) | R2020a |
| `matlab.io.datastore.FoldersPropertyProvider` | Mixin | Add `Folders` property for folder-based access and writing | R2020a |
| `matlab.io.datastore.FileSet` | Utility | Whole-file collection management | R2020a |
| `matlab.io.datastore.BlockedFileSet` | Utility | Block-based file collection (fixed-size records within files) | R2020a |
## Required Methods Per Mixin
### matlab.io.Datastore (base)
**With `matlab.io.datastore.FileSet` (preferred):**
| Method | Access | Implementation | Required? |
|--------|--------|---------------|-----------|
| `hasdata` | public | `obj.CurrentFileIndex <= obj.FileSet.NumFiles` | Yes |
| `read` | public | Load from `obj.FileSet.FileInfo.Filename(CurrentFileIndex)`, advance index | Yes |
| `reset` | public | `obj.CurrentFileIndex = 1` | Yes |
| `progress` | Hidden | `(CurrentFileIndex-1) / obj.FileSet.NumFiles` | Yes |
| `copyElement` | protected | `copyElement@matlab.mixin.Copyable` + `copy(obj.FileSet)` | Yes — `readall` and `preview` always copy internally |
**With `matlab.io.datastore.BlockedFileSet` (fixed-size records):**
| Method | Access | Implementation | Required? |
|--------|--------|---------------|-----------|
| `hasdata` | public | `hasNextBlock(obj.BlockedFileSet)` | Yes |
| `read` | public | `nextblock` → custom read function (typically `fopen`/`fseek`/`fread` internally) | Yes |
| `reset` | public | `reset(obj.BlockedFileSet)` + reset block counter | Yes |
| `progress` | Hidden | `obj.CurrentBlockIndex / obj.TotalBlocks` (use own counter, not `NumBlocks`) | Yes |
| `copyElement` | protected | `copyElement@matlab.mixin.Copyable` + `copy(obj.BlockedFileSet)` | Yes — `readall` and `preview` always copy internally |
**Properties:**
| Property | Access | Description | Required? |
|----------|--------|-------------|-----------|
| `AlternateFileSystemRoots` | Dependent | Enables cross-platform portability of datastore across different file systems | Optional |
| `get.AlternateFileSystemRoots` | public | Getter — return `obj.FileSet.AlternateFileSystemRoots` | Only if `AlternateFileSystemRoots` is defined |
| `set.AlternateFileSystemRoots` | public | Setter — set on FileSet, then call `reset(obj)` | Only if `AlternateFileSystemRoots` is defined |
### matlab.io.datastore.Partitionable
| Method | Access | Implementation | Required? |
|--------|--------|---------------|-----------|
| `partition` | public | `copy(obj)` then `partition(obj.FileSet, n, idx)` then `reset` | Yes |
| `maxpartitions` | **protected** | `maxpartitions(obj.FileSet)` | Yes |
### matlab.io.datastore.HadoopLocationBased
| Method | Access | Implementation | Required? |
|--------|--------|---------------|-----------|
| `initializeDatastore` | Hidden | Rebuild FileSet from `info` struct (FileName, Offset, Size) | Yes |
| `getLocation` | Hidden | Return `obj.FileSet` | Yes |
| `isfullfile` | Hidden | `isequal(obj.FileSet.FileSplitSize, 'file')` | Optional |
### matlab.io.datastore.Shuffleable
| Method | Access | Implementation | Required? |
|--------|--------|---------------|-----------|
| `shuffle` | public | Copy self, randomize file order via permutation index | Yes |
With `FileSet`: create a new `FileSet` from the shuffled file list.
With `BlockedFileSet`: implement via a `BlockOrder` property holding a `randperm`
index vector for block indices.
### matlab.io.datastore.Subsettable
Use `Subsettable` on R2022b+ when you need to extract non-adjacent or specific
data slices by index, randomized shuffling, or parallel minibatch processing.
Requires that every data read can be accessed independently (i.e., you can jump
to any read by index without sequential iteration). If you only need basic
tall/parallel support, `Partitionable` is sufficient.
| Method | Access | Implementation | Required? |
|--------|--------|---------------|-----------|
| `maxpartitions` | **protected** | Return total number of independent reads (e.g., `obj.FileSet.NumFiles`) | Yes |
| `subsetByReadIndices` | **protected** | Copy self, keep only the reads at the given indices (e.g., rebuild FileSet from indexed file list) | Yes |
`Subsettable` provides `subset`, `partition`, and `shuffle` automatically based
on your `subsetByReadIndices` implementation — you do not implement those methods
yourself.
Note: `Subsettable` subsumes both `Partitionable` and `Shuffleable`. If using
`Subsettable`, do not also inherit from those two separately.
### matlab.io.datastore.FileWritable
| Property/Method | Access | Implementation | Required? |
|----------------|--------|---------------|-----------|
| `SupportedOutputFormats` | Constant | String array of format names | Yes |
| `DefaultOutputFormat` | Constant | Default format string | Yes |
| `write` | protected | Write data to custom format | Only if custom format (not built-in) |
| `getFiles` | public | Return file list | Only if no `Files` property |
| `getFolders` | protected | Return folder list | Only if no `Folders` property |
| `validateOutputLocation` | public | Custom validation of output path | Only if string validation is insufficient |
| `getCurrentFilename` | public | Return current file name during write | Only for multi-read-per-file datastores |
| `currentFileIndexComparator` | public | Compare file indices | Only for multi-read-per-file datastores |
**Parallel write support (`UseParallel` in `writeall`):**
Requires inheriting from both `matlab.io.datastore.FileWritable` AND
`matlab.io.datastore.Partitionable`, with a `partition` method that supports
the syntax `partition(ds, 'Files', index)`.
### matlab.io.datastore.FoldersPropertyProvider
Adds a `Folders` property for folder-based access and writing patterns. Useful when
data is organized hierarchically (e.g., class-per-folder for image classification)
or when the datastore needs write support via `writeall` with `FolderLayout`.
| Method/Property | Access | Implementation | Required? |
|----------------|--------|---------------|-----------|
| `Folders` | public (Dependent) | Return unique folder paths from file list | Provided by mixin |
| `set.Folders` | public | Filter files to only those in specified folders, then reset | Yes |
**Usage in constructor:**
```matlab
ds.Folders = populateFoldersFromLocation(ds, location);
```
Enables the `FolderLayout` name-value pair of `writeall` when combined with
`FileWritable`.
----
Copyright 2026 The MathWorks, Inc.
----
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |