Skip to content

matlab/matlab-agentic-toolkit / skills-catalog/ai-and-statistics/matlab-classify-tabular-data

matlab-classify-tabular-data

Unverified

Use this skill to classify tabular data end-to-end in MATLAB — load a dataset, prepare and clean it, select promising classifiers, train them, and compare accuracies with cross-validation, holdout, or hyperparameter optimization plus statistical tests. TRIGGER when: user asks to classify tabular data, pick classifiers for a dataset, compare classifier accuracy, run cross-validation or a holdout evaluation, or find the best model with statistical uncertainty. DO NOT TRIGGER when: user has non-tabular inputs (images, sequences, time series), wants a regression model, is training a specific neural network architecture (use matlab-train-network), or wants cost-sensitive learning or an arbitrary class-prior vector (this skill only supports the built-in uniform-prior toggle for imbalanced data).

FileKindSize
SKILL.mdskill md36.1 KB
manifest.yamlconfig524 B
references/README.mdmarkdown3.1 KB
references/dataprep.mdmarkdown7.5 KB
references/hpo.mdmarkdown11.5 KB
references/save-and-export.mdmarkdown7.0 KB
references/select-classifiers-imbalanced.mdmarkdown3.0 KB
references/select-classifiers.mdmarkdown9.4 KB
scripts/README.mdmarkdown2.3 KB
scripts/helpers/aggregate_nested_cv_loss.mother470 B
scripts/helpers/apply_preproc.mother4.1 KB
scripts/helpers/build_model_definitions.mother16.6 KB
scripts/helpers/compute_data_flags.mother2.8 KB
scripts/helpers/compute_pairwise_pvalues_cv.mother623 B
scripts/helpers/compute_pairwise_pvalues_holdout.mother556 B
scripts/helpers/count_categorical_levels.pbinary376 B
scripts/helpers/export_workflow_script.mother15.2 KB
scripts/helpers/find_zero_variance_columns.mother810 B
scripts/helpers/is_bag_of_tokens.mother181 B
scripts/helpers/make_holdout_split.mother387 B
scripts/helpers/plot_accuracy_bars.mother994 B
scripts/helpers/plot_boosting_curve.mother1.2 KB
scripts/helpers/plot_pvalue_heatmap.mother465 B
scripts/helpers/plot_training_time.mother797 B
scripts/helpers/resolve_recipe.mother1.1 KB
scripts/helpers/resume_boosting_cv.mother543 B
scripts/helpers/resume_boosting_holdout.mother448 B
scripts/helpers/retrain_with_hpo.mother2.7 KB
scripts/helpers/retrain_with_resume.mother2.5 KB
scripts/helpers/sanitize_model_name.mother799 B
scripts/helpers/save_selected_models.mother4.0 KB
scripts/helpers/score_holdout_test.mother278 B
scripts/helpers/stratified_subsample.pbinary367 B
scripts/helpers/train_and_score_cv.mother405 B
scripts/helpers/train_and_score_holdout.mother1.1 KB
scripts/model_catalog/branch_categorical.mother4.2 KB
scripts/model_catalog/branch_many_missing.mother3.1 KB
scripts/model_catalog/branch_regular.mother4.1 KB
scripts/model_catalog/branch_sparse.mother1.3 KB
scripts/model_catalog/branch_wide.mother2.0 KB
scripts/model_catalog/classifier_branches.mother2.2 KB
scripts/model_catalog/classifier_thresholds.mother2.2 KB
scripts/model_catalog/imbalanced_boosting.mother3.0 KB
scripts/model_catalog/model_recipe.mother3.5 KB

references/hpo.md

View on GitHub
# Hyperparameter Optimization for Classification Models

This reference is dispatched from the orchestrator after all training, comparisons, and boosting resume are complete. It optimizes hyperparameters for user-selected models using Bayesian optimization via `'OptimizeHyperparameters','auto'`.

## Required inputs (from parent skill)

- `modelNames` — cell array of model names
- `acc` — baseline accuracy for each model
- `accCI` — baseline accuracy confidence intervals
- Top-tier model indices (from pairwise comparison)
- `hasHoldout` — evaluation path flag
- `cv` — `cvpartition` object created by the parent skill (CV path only — used as outer folds for nested CV)
- `X`, `Y` — full data (CV path)
- `XTrain`, `YTrain`, `XTest`, `YTest` — split data (holdout path)
- `Xsub`, `Ysub` — subsampled data (if subsampling was applied in holdout path)
- `N` — number of observations used for accuracy estimation
- `useUniformPrior` — whether to pass `'Prior','uniform'`

## Step 1: Ask the user

Present the model list and ask which to optimize:

> Would you like to optimize hyperparameters for any models? Select from the list below, or type "none" to skip.
>
> **Top tier (suggested):** [list top-tier model names]
> **Other models:** [list remaining model names]
>
> Which models to optimize? (comma-separated names, "all", "top", or "none")

If the user says "none", this reference is complete — return to the parent skill's summary step.

## Step 2: Run optimization

For each selected model, call the appropriate `fitc*` function with `'OptimizeHyperparameters','auto'`. Do NOT pass `'OptimizeHyperparameters'` to template objects — templates do not recognize this parameter. You must call the `fitc*` function directly.

**Uniform prior — mandatory when `useUniformPrior = true`:** Assemble a `priorNV` cell once, and prepend it to every `fitc*` call in this reference. If HPO forgets `'Prior','uniform'` while baseline training used it, imbalanced-data before/after comparisons silently switch prior between the two runs and are meaningless:

```matlab
if useUniformPrior
    priorNV = {'Prior', 'uniform'};
else
    priorNV = {};
end
% Every fitc* call below:  fitcXXX(X, Y, priorNV{:}, 'OptimizeHyperparameters', 'auto', ...)
```

### Template-to-fitc mapping for optimization

| Model type | Optimization call |
|---|---|
| Tree / TreeIC | `fitctree(X, Y, 'OptimizeHyperparameters','auto', ...)` |
| GaussianSVM (binary) | `fitcsvm(X, Y, 'OptimizeHyperparameters','auto', 'KernelFunction','rbf', ...)` |
| GaussianSVM (multiclass) | `fitcecoc(X, Y, 'Learners','svm', 'OptimizeHyperparameters','auto', ...)` — do NOT specify `'Coding'`; it is optimized automatically |
| LinearSVM / LogisticRegression (binary) | `[mdl, fitInfo, hpoResults] = fitclinear(X, Y, 'OptimizeHyperparameters','auto', ...)` — use 3 outputs; the 3rd output contains optimization results |
| LinearSVM / LogisticRegression (multiclass) | `fitcecoc(X, Y, 'Learners',templateLinear(...), 'OptimizeHyperparameters','auto', ...)` — do NOT specify `'Coding'`; it is optimized automatically |
| NeuralNetwork | `fitcnet(X, Y, 'OptimizeHyperparameters','auto', ...)` |
| KNN | `fitcknn(X, Y, 'OptimizeHyperparameters','auto', ...)` |
| NaiveBayes | `fitcnb(X, Y, 'OptimizeHyperparameters','auto', ...)` |
| QuadraticDiscriminant / LinearDiscriminant | `fitcdiscr(X, Y, 'OptimizeHyperparameters','auto', 'FillCoeffs','off','SaveMemory','on', ...)` |
| Ensemble (Bag/Boost) | `fitcensemble(X, Y, 'Method','...', 'OptimizeHyperparameters','auto', ...)` |
| GAM | `fitcgam(X, Y, 'OptimizeHyperparameters','auto', ...)` |
| KernelSVM / KernelLogistic (binary) | `fitckernel(X, Y, 'OptimizeHyperparameters','auto', ...)` |
| KernelSVM / KernelLogistic (multiclass) | `fitcecoc(X, Y, 'Learners',templateKernel(...), 'OptimizeHyperparameters','auto', ...)` — do NOT specify `'Coding'`; it is optimized automatically |

**ECOC deduplication:** If the user selected both OVO and OVA variants of the same base learner (e.g., GaussianSVM-OVO and GaussianSVM-OVA), run only ONE `fitcecoc` optimization without specifying `'Coding'` — the optimizer will search over coding designs automatically. Report the result for both variants.

**Base learner determination:** Determine shared base learners by inspecting `modelDefs(k).innerLearner_fnName` — do NOT infer from model names. Two models are "the same base learner" only if they have identical `innerLearner_fnName` values. For example, LinearSVM-OVA (innerLearner: templateLinear) and LinearStandardizedSVM-OVA (innerLearner: templateSVM) are distinct and require separate optimization calls.

**Ensemble deduplication:** If the user selected multiple ensemble models (e.g., RoughRUSBoost, FineRUSBoost, RoughAdaBoost, DarkRandomForest), run only ONE `fitcensemble` optimization without specifying `'Method'` — the `'auto'` option searches over ensemble methods automatically. Report the result for all ensemble variants the user selected.

**ECOC with templateLinear or templateKernel (CRITICAL):** You MUST call `fitcecoc` with 2 outputs when optimizing models that use `templateLinear` or `templateKernel`. The returned model object is compact and does NOT have a `HyperparameterOptimizationResults` property — the only way to access the optimization results is via the 2nd output. Calling with 1 output will lose all hyperparameter information.

```matlab
[optimModel, hpoResults] = fitcecoc(X, Y, 'Learners', templateKernel(...), ...
    'OptimizeHyperparameters','auto', ...
    'HyperparameterOptimizationOptions', struct(...));
% hpoResults.HyperparameterOptimizationResults contains the optimization table
% hpoResults.HyperparameterOptimizationResults.XAtMinObjective has the best params
```

### Holdout path (`hasHoldout = true`)

Cross-validate on the training set only. Do NOT pass the parent's `cv` partition — it was created for the full dataset `X` and its size does not match `XTrain`. Create a new partition for the training data and pass it via `HyperparameterOptimizationOptions`:

Before starting each model, display: `fprintf('Optimizing %s...\n', modelName);`

```matlab
cvHPO = cvpartition(YTrain, 'KFold', 5, 'Stratify', true);

fprintf('Optimizing %s...\n', modelName);
tic;
optimModel = fitctree(XTrain, YTrain, 'OptimizeHyperparameters','auto', ...
    'HyperparameterOptimizationOptions', struct('CVPartition',cvHPO,'ShowPlots',false,'Verbose',0));
elapsed = toc;
fprintf('%s optimization complete (%.1f s)\n', modelName, elapsed);
```

(Use `Xsub`/`Ysub` instead of `XTrain`/`YTrain` if subsampling was applied.)

The optimization returns a trained (non-cross-validated) model. Extract the optimized hyperparameters and retrain with cross-validation:

```matlab
bestParams = optimModel.HyperparameterOptimizationResults.XAtMinObjective;
% Retrain with CV using the optimized params — example for fitctree:
cvOptModel = fitctree(XTrain, YTrain, 'CVPartition', cvHPO, ...
    'MaxNumSplits', bestParams.MaxNumSplits, ...
    'MinLeafSize', bestParams.MinLeafSize);
errOpt = kfoldLoss(cvOptModel);
accOpt = 1 - errOpt;
```

Then evaluate on the held-out test set for the final reported accuracy via `scripts/helpers/score_holdout_test.m`:

```matlab
[accHoldout, accCIHoldout] = score_holdout_test(optimModel, XTest, YTest);
```

### CV path (`hasHoldout = false`) — nested cross-validation

Use nested CV to avoid optimistic bias: for each outer fold of the existing `cv` partition, optimize on the training portion and predict on the test portion. This gives unbiased accuracy estimates for the optimized model.

Before starting each model, display: `fprintf('Optimizing %s...\n', modelName);`

```matlab
% Preallocate an out-of-fold prediction vector in Y's own type (categorical,
% string, cellstr, or numeric). Using Y for the seed guarantees the
% container matches what fitc*/predict will emit fold by fold.
YHatOpt = Y;
YHatOpt(:) = Y(1);  % overwrite every entry so leftover values from Y do not leak into folds not yet visited
nFolds = cv.NumTestSets;

fprintf('Optimizing %s...\n', modelName);
tic;
for k = 1:nFolds
    trainIdx = training(cv, k);
    testIdx  = test(cv, k);
    
    % Optimize on outer training fold (inner CV is automatic)
    optimFold = fitctree(X(trainIdx,:), Y(trainIdx), ...
        'OptimizeHyperparameters','auto', ...
        'HyperparameterOptimizationOptions', struct('ShowPlots',false,'Verbose',0));
    
    % Predict on held-out outer fold
    YHatOpt(testIdx) = predict(optimFold, X(testIdx,:));
    fprintf('  Fold %d/%d done\n', k, nFolds);
end
elapsed = toc;
fprintf('%s optimization complete (%.1f s)\n', modelName, elapsed);

% Aggregate out-of-fold predictions into accuracy + 95% CI. Pass Y (the
% original response) so aggregate_nested_cv_loss can compare like-for-like.
[accOpt, accCIOpt] = aggregate_nested_cv_loss(YHatOpt, Y, useUniformPrior, X);
```

Replace `fitctree` with the appropriate `fitc*` function for each model (see mapping table above). Prepend `priorNV{:}` (defined at the top of this step) to every `fitc*` call inside the loop — this is what carries `'Prior','uniform'` through to HPO when `useUniformPrior = true`.

## Step 3: Report optimization results

### Holdout path

Report both the hyperparameter changes and accuracy changes for each optimized model:

> **Optimization results:**
>
> **[ModelName]:**
>
> | Hyperparameter | Before (default/set) | After (optimized) |
> |---|---|---|
> | MaxNumSplits | 100 | 27 |
> | MinLeafSize | 1 | 12 |
> | ... | ... | ... |
>
> | Metric | Before | After |
> |---|---|---|
> | Accuracy | 82.3% | 86.1% |
> | 95% CI | [79.8%, 84.8%] | [83.8%, 88.4%] |
> | Optimization time | — | 120.3s |

For each model, identify which hyperparameters were optimized by comparing `optimModel.HyperparameterOptimizationResults.XAtMinObjective` against the defaults or values originally set in the model template. Only show hyperparameters that changed or were searched over.

### CV path (nested)

Do NOT report a single set of optimized hyperparameters — each outer fold produced its own optimized model, so there is no single "best" parameter set to display. Instead, report only the accuracy change:

> **Optimization results:**
>
> **[ModelName]:**
> Optimized via nested 5-fold CV (each fold optimized independently — no single parameter set to report).
>
> | Metric | Before | After |
> |---|---|---|
> | Accuracy | 82.3% | 86.1% |
> | 95% CI | [79.8%, 84.8%] | [83.8%, 88.4%] |
> | Optimization time | — | 120.3s |

## Step 4: Re-run pairwise statistical comparisons (holdout path only)

After optimization, re-run pairwise statistical comparisons only if `hasHoldout = true`. Use `testcholdout` with the predictions on the holdout test set. Obtain predictions from each optimized model via `predict(optimModel, XTest)` and compare against stored predictions from non-optimized models:

```matlab
[h, p] = testcholdout(YHatOpt{i}, YHat{j}, YTest, 'Alpha', alpha_corrected);
```

Apply Bonferroni correction: `alpha_corrected = 0.05 / nchoosek(nModels, 2)` where `nModels` is the total number of models (optimized + non-optimized).

### Report updated top tier

After running all pairwise comparisons, report the updated p-value matrix and identify the new top tier. Note any changes from the pre-optimization top tier (e.g., "Optimized TreeIC has entered the top tier" or "GAM remains the sole top-tier model").

### CV path (`hasHoldout = false`) — no post-HPO comparison

Do NOT re-run pairwise statistical comparisons in the CV path. If the user asks for a statistical comparison between optimized and non-optimized models, explain that this would require a nested cross-validation design and is not available at present.

---

Copyright 2026 The MathWorks, Inc.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 197kUpdated Claude CodeCodexVerifiedInstall