Skip to content

matlab/matlab-agentic-toolkit / skills-catalog/image-processing-and-computer-vision/matlab-process-images

matlab-process-images

Verified

Load this first for any MATLAB task involving images, photos, scans, frames, volumes, or visual data — reading, writing, filtering, enhancing, denoising, sharpening, deblurring, thresholding, segmenting, masking, detecting, counting objects, removing backgrounds, measuring, registering, stitching, aligning, cropping, resizing, or transforming. Covers image I/O (TIFF, PNG, JPEG, RAW, HDR), histograms, edge detection, color, morphology, deep learning for images, quality assessment, and 3-D volumes. Also for image problems (dark, noisy, blurry, low contrast, uneven lighting, hazy, color cast) or data exploration (folder of images). Includes medical imaging I/O (DICOM, NIfTI, NRRD, medicalVolume) and large/blocked images (blockedImage, whole-slide). Skip if the user just wants you to look at an image, with no MATLAB work. Not for Simulink video-block models, C/C++/CUDA code generation or GPU deployment of a network, or point clouds. Hyperspectral and visual inspection have sibling skills.

FileKindSize
SKILL.mdskill md15.8 KB
manifest.yamlconfig811 B
references/3d-volume-processing.mdmarkdown20.2 KB
references/analysis-measurement.mdmarkdown18.5 KB
references/app-building.mdmarkdown16.9 KB
references/blocked-3d-volumes.mdmarkdown9.9 KB
references/blocked-data-discovery.mdmarkdown22.1 KB
references/blocked-geospatial.mdmarkdown3.2 KB
references/blocked-specialized-workflows.mdmarkdown4.3 KB
references/blocked-training-and-labeling.mdmarkdown7.0 KB
references/blocked-visualization.mdmarkdown5.1 KB
references/blocked-whole-slide-images.mdmarkdown3.1 KB
references/color-processing.mdmarkdown9.0 KB
references/deblurring.mdmarkdown8.1 KB
references/deep-learning-imaging.mdmarkdown14.8 KB
references/dicom-anonymization.mdmarkdown3.3 KB
references/dicom-rt-workflows.mdmarkdown7.4 KB
references/filtering.mdmarkdown21.8 KB
references/geometric-transforms.mdmarkdown14.3 KB
references/image-io.mdmarkdown15.5 KB
references/image-understanding.mdmarkdown20.0 KB
references/ipt-reading-writing.mdmarkdown4.8 KB
references/large-image-processing.mdmarkdown12.3 KB
references/medical-data-io.mdmarkdown8.3 KB
references/mit-medical-volume.mdmarkdown8.2 KB
references/mit-spatial-referencing.mdmarkdown5.9 KB
references/morphological-ops.mdmarkdown18.0 KB
references/preprocessing.mdmarkdown14.4 KB
references/quality-assessment.mdmarkdown13.5 KB
references/registration.mdmarkdown24.7 KB
references/segmentation-automated.mdmarkdown9.7 KB
references/segmentation-interactive.mdmarkdown15.3 KB
references/texture-transforms.mdmarkdown10.4 KB
references/video-io.mdmarkdown3.4 KB

references/deep-learning-imaging.md

View on GitHub
# Deep Learning for Image Processing

Decision guidance for combining deep learning with image processing: data preparation, semantic segmentation, pretrained networks, and inference pipelines. Requires Deep Learning Toolbox; some features require Computer Vision Toolbox.

> **Often loaded with:** `image-io.md` (datastores and batch loading), `analysis-measurement.md` (post-processing segmentation output).

## Using Pretrained PyTorch Vision Models

For tasks where a pretrained PyTorch model is the best starting point (e.g., depth estimation, style transfer, foundation models not available natively in MATLAB), use the **`matlab-integrate-pytorch-vision`** skill to interface with Python model repositories from MATLAB.

## Toolbox Requirements

| Task | Required toolboxes |
|------|-------------------|
| Training/inference | Deep Learning Toolbox |
| Semantic segmentation architectures | Computer Vision Toolbox |
| Object detection architectures | Computer Vision Toolbox |
| Image data augmentation | Deep Learning Toolbox |
| Pretrained feature extraction | Deep Learning Toolbox |

**Check availability before generating code:**
```matlab
hasDeepLearning = ~isempty(ver("nnet"));
hasComputerVision = ~isempty(ver("vision"));
```

## 1. Data Preprocessing

### Image Datastores

#### Classification (folder-per-class)

```matlab
imds = imageDatastore("data/images/", ...
    IncludeSubfolders=true, ...
    LabelSource="foldernames");  % Labels from folder names (e.g., cat/, dog/)
```

#### Semantic Segmentation (image + pixel label pairs)

```matlab
imds = imageDatastore("data/images/");

classNames = ["background", "tumor", "healthy"];
labelIDs = [0 1 2];
pxds = pixelLabelDatastore("data/labels/", classNames, labelIDs);

ds = combine(imds, pxds);
```

#### Image Regression (image + continuous target)

```matlab
imds = imageDatastore("data/images/");

% Targets from a table or file (e.g., depth maps, angles, coordinates)
targets = readtable("data/targets.csv");  % columns: filename, value1, value2, ...
targetDs = arrayDatastore(targets{:, 2:end});

ds = combine(imds, targetDs);
```

For image-to-image regression (e.g., denoising, super-resolution, depth estimation):

```matlab
inputImds = imageDatastore("data/noisy/");
targetImds = imageDatastore("data/clean/");
ds = combine(inputImds, targetImds);
```

### Resizing for Network Input

Networks require fixed input size. Resize images to match:

```matlab
targetSize = [256 256];  % [rows cols] — NOT [width height]

% For datastores — use transform
dsResized = transform(ds, @(data) resizeForNetwork(data, targetSize));

function out = resizeForNetwork(data, targetSize)
    out{1} = imresize(data{1}, targetSize);
    out{2} = imresize(data{2}, targetSize, "nearest");  % Labels: nearest!
end
```

**Gotcha:** Always use `"nearest"` interpolation when resizing label maps or masks. Default bicubic creates invalid fractional labels.

#### Aspect-Ratio-Preserving Resize

When aspect ratio distortion hurts accuracy (e.g., object detection), pad to the target size instead of stretching:

```matlab
function out = letterboxResize(img, targetSize)
    [h, w, ~] = size(img);
    scale = min(targetSize(1)/h, targetSize(2)/w);
    newSize = round([h w] * scale);
    resized = imresize(img, newSize);

    padValue = 114;  % common gray fill for detection networks
    out = ones(targetSize(1), targetSize(2), size(img,3), class(img)) * padValue;
    offset = floor((targetSize - newSize) / 2) + 1;
    out(offset(1):offset(1)+newSize(1)-1, offset(2):offset(2)+newSize(2)-1, :) = resized;
end
```

For semantic segmentation, wrap as a paired `transform` that also pads labels with an ignore value (e.g., 255) so padded pixels do not contribute to loss.

### Input Normalization — Critical

Networks expect specific input ranges. Common patterns:

| Network type | Expected input | Prepare with |
|-------------|---------------|--------------|
| Pretrained (ImageNet) | Per-channel zero-mean, unit-variance | Subtract ImageNet channel means, divide by stds |
| Pretrained (COCO) | Per-channel zero-mean, unit-variance | Subtract COCO channel means, divide by stds |
| Custom (trained on [0,1]) | [0, 1] single | `im2single(img)` or `rescale(img, 0, 1)` for arbitrary-range data |
| Custom (trained on [-1,1]) | [-1, 1] | `rescale(img, -1, 1)` |
| Arbitrary range (medical HU, float sensors) | Dataset-specific z-score | Compute dataset mean/std, normalize |

**Gotcha:** If training data was uint8 [0,255] but inference images are double [0,1], the network can produce incorrect predictions. Always match inference preprocessing to training preprocessing.

#### Pretrained Networks (ImageNet / COCO)

Most pretrained backbones use fixed per-channel statistics:

```matlab
% ImageNet (ResNet, VGG, EfficientNet, etc.)
channelMeans = single([0.485, 0.456, 0.406]);
channelStds  = single([0.229, 0.224, 0.225]);

% COCO (detection/segmentation models pretrained on COCO)
channelMeans = single([0.471, 0.448, 0.408]);
channelStds  = single([0.234, 0.239, 0.242]);

% Apply: input must be single [0,1] first
img = im2single(img);  % uint8 [0,255] → single [0,1]
img = (img - reshape(channelMeans, 1, 1, 3)) ./ reshape(channelStds, 1, 1, 3);
```

#### Custom Networks (Compute-Then-Apply)

Compute mean/std from your training set, save, then apply via `transform`:

```matlab
% Step 1: Compute channel-wise statistics
imds = imageDatastore("train/", IncludeSubfolders=true);
pixelSum = single([0 0 0]);
pixelSqSum = single([0 0 0]);
nPixels = 0;

while hasdata(imds)
    img = single(read(imds));
    nPx = size(img, 1) * size(img, 2);
    pixelSum = pixelSum + sum(img, [1 2]);
    pixelSqSum = pixelSqSum + sum(img.^2, [1 2]);
    nPixels = nPixels + nPx;
end
reset(imds);

channelMeans = pixelSum / nPixels;
channelStds = sqrt(pixelSqSum / nPixels - channelMeans.^2);
save("normalization_params.mat", "channelMeans", "channelStds");

% Step 2: Apply in training pipeline
params = load("normalization_params.mat");
dsTrain = transform(dsTrain, @(data) normalizeForTraining(data, params));

function out = normalizeForTraining(data, params)
    img = single(data{1});
    img = (img - reshape(params.channelMeans, 1, 1, [])) ./ ...
          reshape(params.channelStds, 1, 1, []);
    out = {img, data{2}};  % {normalized image, label}
end
```

### Data Augmentation

```matlab
augmenter = imageDataAugmenter( ...
    RandXReflection=true, ...
    RandRotation=[-10 10], ...
    RandScale=[0.9 1.1]);

augDs = augmentedImageDatastore(targetSize, imds, ...
    DataAugmentation=augmenter);
```

For segmentation (paired image + label augmentation), use `transform` with custom function:

```matlab
ds = combine(imds, pxds);
dsAug = transform(ds, @augmentPair);

function out = augmentPair(data)
    img = data{1};
    lbl = data{2};
    % Apply same geometric transform to both
    if rand > 0.5
        img = fliplr(img);
        lbl = fliplr(lbl);
    end
    tform = randomAffine2d(Rotation=[-15 15], Scale=[0.8 1.2]);
    rout = affineOutputView(size(img), tform);
    img = imwarp(img, tform, OutputView=rout);
    lbl = imwarp(lbl, tform, OutputView=rout, Interpolation="nearest");
    out = {img, lbl};
end
```

### Channel Count Mismatches

| Scenario | Solution |
|----------|----------|
| Grayscale image → network expects RGB (3-ch) | `repmat(grayImg, 1, 1, 3)` |
| RGB image → network expects grayscale (1-ch) | `im2gray(rgbImg)` |
| Volumetric data with a 2-D network | Process slices independently only when a 2-D workflow is appropriate; otherwise, consider a 3-D architecture such as `unet3d` |
| Multi-channel (>3) → standard network | Custom input layer or channel reduction |


## 2. Modeling

### Building from Scratch (Layer-by-Layer)

Define custom architectures using Deep Learning Toolbox layers:

```matlab
layers = [
    imageInputLayer([256 256 3], Normalization="none")
    convolution2dLayer(3, 32, Padding="same")
    batchNormalizationLayer
    reluLayer
    maxPooling2dLayer(2, Stride=2)
    convolution2dLayer(3, 64, Padding="same")
    batchNormalizationLayer
    reluLayer
    globalAveragePooling2dLayer
    fullyConnectedLayer(numClasses)
    softmaxLayer
];
net = dlnetwork(layers);
```

Use `addLayers` and `connectLayers` for branching or multi-input architectures. Each layer must have a unique `Name`, and all inputs of multi-input layers (e.g., `concatenationLayer`) must be connected:

```matlab
net = dlnetwork;

% Main path
net = addLayers(net, [
    imageInputLayer([256 256 3], Name="input")
    convolution2dLayer(3, 32, Padding="same", Name="conv1")
    reluLayer(Name="relu1")
    ]);

% Branch path
net = addLayers(net, [
    convolution2dLayer(1, 32, Padding="same", Name="conv_branch")
    reluLayer(Name="relu_branch")
    ]);

% Merge
net = addLayers(net, concatenationLayer(3, 2, Name="concat"));

% Connect: branch takes same input, concat merges both paths
net = connectLayers(net, "input", "conv_branch");
net = connectLayers(net, "relu1", "concat/in1");
net = connectLayers(net, "relu_branch", "concat/in2");
```

Use **Deep Network Designer** app (`deepNetworkDesigner`) to visually inspect, edit, and validate network architectures.

### Classification with Pretrained Encoders

Use `pretrainedEncoderNetwork` to create a feature extractor from a pretrained backbone, then add classification layers:

```matlab
[encoder, outputNames] = pretrainedEncoderNetwork("resnet18", 4);
```

### Semantic Segmentation Architectures

| Architecture | Function | Best for |
|-------------|----------|----------|
| U-Net | `unet` | Limited training data and segmentation of small or intricate structures |
| 3-D U-Net | `unet3d` | Volumetric image data, such as CT and MRI |
| DeepLab v3+ | `deeplabv3plus` | Precise boundary delineation and objects at multiple scales |
| BiSeNet v2 | `bisenetv2` | Real-time segmentation and applications with limited computational resources |

```matlab
inputSize = [256 256 1];  % [H W C]
numClasses = 3;

% U-Net
net = unet(inputSize, numClasses, EncoderDepth=4);

% 3-D U-Net (volumetric)
net = unet3d([64 64 64 1], numClasses, EncoderDepth=3);

% DeepLab v3+ with pretrained backbone
net = deeplabv3plus([256 256 3], numClasses, "resnet18");

% BiSeNet v2 — pretrained transfer learning (keeps pretrained weights)
net = bisenetv2(inputSize, numClasses);

% BiSeNet v2 — custom uninitialized network (no pretrained weights)
net = bisenetv2(inputSize, numClasses, NetworkType="custom");
```

To adapt segmentation architectures for **image-to-image regression** (e.g., depth estimation, denoising), replace the final classification layer with a regression output (1 or N channels, no softmax):

```matlab
net = unet([256 256 3], 2, EncoderDepth=4);  % Start with 2-class U-Net
% Replace final conv + softmax for single-channel regression output
net = replaceLayer(net, "encoderDecoderFinalConvLayer", ...
    convolution2dLayer(1, 1, Name="encoderDecoderFinalConvLayer"));
net = removeLayers(net, "FinalNetworkSoftmax-Layer");
```

### Custom Encoder-Decoder

For more control over skip connections and decoder structure, use `encoderDecoderNetwork` with a pretrained encoder:

```matlab
[encoder, outputNames] = pretrainedEncoderNetwork("resnet18", 4);
net = encoderDecoderNetwork([256 256 3], encoder, outputNames, ...
    NumClasses=numClasses);
```

## 3. Training

```matlab
options = trainingOptions("adam", ...
    InitialLearnRate=1e-3, ...
    MaxEpochs=50, ...
    MiniBatchSize=8, ...
    Shuffle="every-epoch", ...
    ValidationData=valDs, ...
    ValidationFrequency=50, ...
    Plots="training-progress");

net = trainnet(dsTrain, net, "crossentropy", options);
```

**Modern API:** Use `trainnet`, not the legacy `trainNetwork`. `trainnet` returns a `dlnetwork` directly.

## 4. Evaluation

### Inference

```matlab
% Single image
result = semanticseg(testImage, net);  % Returns categorical label map

% Batch via datastore
pxdsResults = semanticseg(testImds, net, WriteLocation="results/");
```

#### Inference — Reuse Training Normalization

Always load and apply the same normalization at inference:

```matlab
params = load("normalization_params.mat");
testImg = im2single(imread("test.png"));
testImg = (testImg - reshape(params.channelMeans, 1, 1, [])) ./ ...
          reshape(params.channelStds, 1, 1, []);
result = semanticseg(testImg, net);
```

#### Tiled or Patch-Based Inference (Large Images/Volumes)

For images larger than GPU memory, use `blockedImage` with `apply` to process tiles:

```matlab
bim = blockedImage(largeImage);

patchSize = [256 256];
overlap = [32 32];
result = apply(bim, @(block) semanticseg(block.Data, net), ...
    BlockSize=patchSize, ...
    BorderSize=overlap, ...
    PadPartialBlocks=true);
```

For tiled training workflows that use `blockedImage` or `blockedImageDatastore` to prepare image tiles, see `references/large-image-processing.md`.

### Classification Metrics

```matlab
scores = minibatchpredict(net, dsTest);
YPred = scores2label(scores, classNames);

% ROC curve and AUC
rocObj = rocmetrics(YTest, scores, classNames);
plot(rocObj);

% Confusion matrix
confusionchart(YTest, YPred);
```

### Segmentation Quality Metrics

```matlab
% Confusion matrix metrics
metrics = evaluateSemanticSegmentation(pxdsResults, pxdsGT);
disp(metrics.ClassMetrics);    % Per-class IoU, accuracy
disp(metrics.DataSetMetrics);  % Mean IoU, mean accuracy, weighted IoU
```

### Post-Processing Segmentation Output

Raw network output often needs cleanup:

```matlab
% Convert categorical to numeric labels
labelMap = uint8(result);

% Remove small regions (per-class)
for c = 1:numClasses
    classMask = labelMap == c;
    classMask = bwareaopen(classMask, minPixels);
    labelMap(labelMap == c & ~classMask) = 0;
end

% Fill holes in segmentation
for c = 1:numClasses
    classMask = labelMap == c;
    classMask = imfill(classMask, "holes");
    labelMap(classMask) = c;
end
```

## Common Mistakes

| Mistake | Why it's wrong | Correct approach |
|---------|---------------|-----------------|
| `trainNetwork` (legacy) | Deprecated, returns `SeriesNetwork`/`DAGNetwork` | Use `trainnet`, returns `dlnetwork` |
| Resizing labels with default interpolation | Bicubic creates invalid fractional labels | Always `imresize(lbl, sz, "nearest")` |
| Mismatched input range at inference | Network trained on [0,255] but given [0,1] | Match preprocessing exactly to training |
| Grayscale into 3-channel network | Dimension mismatch error | `repmat(gray, 1, 1, 3)` |
| Training segmentation without class weighting | Rare classes never learned | Use class weights or `"focal"` loss |
| Processing a large image at full resolution | Can exceed available memory | Use tiled or patch-based processing when the full-resolution image exceeds available memory (`blockedImage` or `randomPatchExtractionDatastore`) |
| `augmentedImageDatastore` for segmentation | Only augments images, not labels | Use `transform` with paired augmentation function |
| Forgetting `"nearest"` for warping labels | `imwarp` default is linear → fractional labels | `imwarp(lbl, tform, Interpolation="nearest")` |

----

Copyright 2026 The MathWorks, Inc.

----
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall