Skip to content

matlab/matlab-agentic-toolkit / skills-catalog/image-processing-and-computer-vision/matlab-process-images

matlab-process-images

Verified

Load this first for any MATLAB task involving images, photos, scans, frames, volumes, or visual data — reading, writing, filtering, enhancing, denoising, sharpening, deblurring, thresholding, segmenting, masking, detecting, counting objects, removing backgrounds, measuring, registering, stitching, aligning, cropping, resizing, or transforming. Covers image I/O (TIFF, PNG, JPEG, RAW, HDR), histograms, edge detection, color, morphology, deep learning for images, quality assessment, and 3-D volumes. Also for image problems (dark, noisy, blurry, low contrast, uneven lighting, hazy, color cast) or data exploration (folder of images). Includes medical imaging I/O (DICOM, NIfTI, NRRD, medicalVolume) and large/blocked images (blockedImage, whole-slide). Skip if the user just wants you to look at an image, with no MATLAB work. Not for Simulink video-block models, C/C++/CUDA code generation or GPU deployment of a network, or point clouds. Hyperspectral and visual inspection have sibling skills.

FileKindSize
SKILL.mdskill md15.8 KB
manifest.yamlconfig811 B
references/3d-volume-processing.mdmarkdown20.2 KB
references/analysis-measurement.mdmarkdown18.5 KB
references/app-building.mdmarkdown16.9 KB
references/blocked-3d-volumes.mdmarkdown9.9 KB
references/blocked-data-discovery.mdmarkdown22.1 KB
references/blocked-geospatial.mdmarkdown3.2 KB
references/blocked-specialized-workflows.mdmarkdown4.3 KB
references/blocked-training-and-labeling.mdmarkdown7.0 KB
references/blocked-visualization.mdmarkdown5.1 KB
references/blocked-whole-slide-images.mdmarkdown3.1 KB
references/color-processing.mdmarkdown9.0 KB
references/deblurring.mdmarkdown8.1 KB
references/deep-learning-imaging.mdmarkdown14.8 KB
references/dicom-anonymization.mdmarkdown3.3 KB
references/dicom-rt-workflows.mdmarkdown7.4 KB
references/filtering.mdmarkdown21.8 KB
references/geometric-transforms.mdmarkdown14.3 KB
references/image-io.mdmarkdown15.5 KB
references/image-understanding.mdmarkdown20.0 KB
references/ipt-reading-writing.mdmarkdown4.8 KB
references/large-image-processing.mdmarkdown12.3 KB
references/medical-data-io.mdmarkdown8.3 KB
references/mit-medical-volume.mdmarkdown8.2 KB
references/mit-spatial-referencing.mdmarkdown5.9 KB
references/morphological-ops.mdmarkdown18.0 KB
references/preprocessing.mdmarkdown14.4 KB
references/quality-assessment.mdmarkdown13.5 KB
references/registration.mdmarkdown24.7 KB
references/segmentation-automated.mdmarkdown9.7 KB
references/segmentation-interactive.mdmarkdown15.3 KB
references/texture-transforms.mdmarkdown10.4 KB
references/video-io.mdmarkdown3.4 KB

references/segmentation-interactive.md

View on GitHub
# Interactive Segmentation

Decision guidance for segmentation methods that use user input — point/box prompts, scribbles, ROIs, seed points, or initial masks. **Prompted SAM** is a strong general-purpose option for point or bounding-box prompts; classical methods may be more appropriate when the task is defined by intensity similarity, edge contrast, or an initial mask.

> **Often loaded with:** `morphological-ops.md` (cleaning up masks), `analysis-measurement.md` (measuring segmented regions), `segmentation-automated.md` (when no user input is available).

## Method Selection

| Task | User input | Algorithm |
|------|-----------|-----------|
| Segment an object from sparse visual prompts | Point or bounding box | `segmentAnythingModel` (prompted SAM 2 / SAM) **(CVT+DLT)** |
| Foreground/background separation | ROI rectangle | `grabcut` (requires superpixels) |
| Foreground/background from strokes | Scribbles | `lazysnapping` (requires superpixels) |
| 2–3 regions from strokes | Scribbles | `imseggeodesic` |
| Flood-fill from seed | Seed point + tolerance | `grayconnected` |
| Region growing from seeds | Seed points | `imsegfmm` |
| Evolving boundary to edges | Initial mask | `activecontour` |
| Medical organ/tumor (interactive box) | Bounding box | `medicalSegmentAnythingModel` **(MIT)** |
| Cell/nucleus instances | Automatic or prompted | `cellpose` → `segmentCells2D` / `segmentCells3D` **(MIT)** |

**General-purpose recommendation:** Use prompted SAM when available — it handles a wide range of scenarios with minimal user input and no training data. For tasks defined by intensity similarity (flood-fill, region growing) or evolving an initial mask to edges, the classical methods below may be more direct.

**Coordinate order varies by function:** SAM functions (`ForegroundPoints`, `BoundingBox`) use **(x, y)** order. `grayconnected` uses **(row, col)** order. `graydiffweight` and coordinate-based `imsegfmm` use **(col, row)** order (column first, then row). Active contour and masks use standard **(row, col)** array indexing.

## Prompted SAM 2 and SAM (R2024b+)

Requires IPT Model for Segment Anything Model support package and Computer Vision Toolbox + Deep Learning Toolbox. By default, `segmentAnythingModel` uses `"sam2-large"` but also supports other SAM 2 variants (e.g., `"sam2-tiny"`, `"sam2-small"`, `"sam2-base-plus"`) and the base SAM model (`"sam-base"`).

```matlab
try
    sam = segmentAnythingModel;  % Verifies support package and required add-on are available
    hasSAM = true;
catch
    hasSAM = false;  % Help the user install the support package
end
```

> **Do not use `which` alone** — it can return true even when required add-ons are missing. Attempt construction to confirm usability.

### Setup

```matlab
sam = segmentAnythingModel;              % Uses "sam2-large" by default
embeddings = extractEmbeddings(sam, I);  % Expensive — do ONCE per image
```

### Point Prompts

```matlab
% Single foreground point — [x, y] coordinates on the object
masks = segmentObjectsFromEmbeddings(sam, embeddings, size(I), ...
    ForegroundPoints=[200 150]);

% Refine with background points
masks = segmentObjectsFromEmbeddings(sam, embeddings, size(I), ...
    ForegroundPoints=[200 150], BackgroundPoints=[50 50; 350 350]);
```

### Bounding Box Prompt

```matlab
% [x, y, width, height]
masks = segmentObjectsFromEmbeddings(sam, embeddings, size(I), ...
    BoundingBox=[50 80 200 180]);
```

### Multiple Objects from Same Image

```matlab
% Extract embeddings once, segment multiple times
embeddings = extractEmbeddings(sam, I);

mask1 = segmentObjectsFromEmbeddings(sam, embeddings, size(I), ...
    ForegroundPoints=[100 200]);
mask2 = segmentObjectsFromEmbeddings(sam, embeddings, size(I), ...
    BoundingBox=[300 100 150 200]);
```

**Key rules:**
- `segmentObjectsFromEmbeddings` returns an **H×W logical mask** (single best mask by default). Use `ReturnMultiMask=true` to get H×W×3 with three ranked candidates.
- Extract embeddings once per image, segment multiple objects from same embeddings
- `size(I)` (third argument) is required — maps embeddings back to image coordinates
- `ForegroundPoints` and `BoundingBox` use **(x, y)** order, not (row, col)
- Combine foreground + background points for difficult cases

**When SAM is unavailable:** Use `grabcut` for foreground/background, `activecontour` for boundary refinement, or `imsegfmm` for region growing.

## GrabCut — Foreground/Background from ROI

**Gotcha:** `grabcut` requires a superpixel label matrix in addition to the image. Always compute superpixels first:

```matlab
[L, N] = superpixels(I, 500);    % REQUIRED before grabcut
ROI = false(size(I,1), size(I,2));
ROI(50:300, 100:350) = true;
BW = grabcut(I, L, ROI);
```

Works on 3-D volumes too (with `superpixels3`).

## Lazy Snapping — Foreground/Background from Scribbles

Requires scribble masks indicating foreground and background regions. `fgScribble` and `bgScribble` are **logical masks** (same size as image) where `true` marks user-drawn strokes:

```matlab
[L, N] = superpixels(I, 500);    % REQUIRED before lazysnapping
BW = lazysnapping(I, L, fgScribble, bgScribble);
```

## Geodesic Segmentation — 2–3 Regions from Scribbles

For partitioning an image into a small number of regions based on user-drawn scribbles. **Requires RGB input** — does not work on grayscale. Returns a label matrix (not a binary mask):

```matlab
L = imseggeodesic(RGB, scribble1, scribble2);           % 2 regions
L = imseggeodesic(RGB, scribble1, scribble2, scribble3); % 3 regions
```

## Gray Connected — Flood-Fill from Seed

Simple region growing based on intensity similarity to a seed pixel:

```matlab
BW = grayconnected(I, row, col, 20);  % tolerance is positional (not a Name-Value pair)

% R2026b: restrict to 4-connected pixels (horizontal/vertical only)
BW = grayconnected(I, row, col, 20, Connectivity=4);
```

## Active Contours — Evolving Boundary

Evolves an initial mask toward object boundaries. Users struggle most with creating the initial mask. Options:

```matlab
% From rough threshold + dilate (most common)
mask = imbinarize(I);
mask = imdilate(mask, strel("disk", 10));
BW = activecontour(I, mask, 200, "Chan-Vese");

% From bounding rectangle
mask = false(size(I));
mask(r1:r2, c1:c2) = true;
BW = activecontour(I, mask, 200, "edge");
```

| Method | Use when |
|--------|----------|
| `"Chan-Vese"` | Region-based, objects without strong edges |
| `"edge"` | Objects with clear boundary edges |

Works on 3-D volumes with same syntax.

## Fast Marching — `imsegfmm`

Region growing from seeds using a weight array:

```matlab
% Weight from intensity difference to seed
% graydiffweight and imsegfmm take (C, R) order — column first, then row
W = graydiffweight(I, col, row, GrayDifferenceCutoff=25);
BW = imsegfmm(W, col, row, 0.01);  % threshold on travel time

% Weight from gradient magnitude
W = gradientweight(I);
BW = imsegfmm(W, seedMask, 0.01);
```

## Medical Segmentation (Medical Imaging Toolbox)

For medical images with MIT available:

| Task | Function |
|------|----------|
| 2-D organ/tumor (interactive box) | `medicalSegmentAnythingModel` |
| 3-D automatic (organs) | `medicalImageLabeler` with MONAI Label |
| Cell/nucleus segmentation | `cellpose` → `segmentCells2D` / `segmentCells3D` |
| Radiomic features from ROI | `radiomics` |

For full medical segmentation workflows (MedSAM, Cellpose, radiomics), see Medical Imaging Toolbox documentation.

## Building Interactive Segmentation Apps

The **Image Segmenter** app (`imageSegmenter`) provides built-in interactive segmentation tools, including SAM. Use it for ad-hoc interactive segmentation sessions.

For custom apps, use App Designer with SAM. The pattern combines a `viewer2d` for display, annotation-based prompts, and real-time mask preview. For a complete walkthrough, see the doc example "Create an App to Segment Images with SAM" (R2026b).

### Architecture Pattern

Build a custom UI component inheriting from `matlab.ui.componentcontainer.ComponentContainer`:

```matlab
classdef SAMSegmenter < matlab.ui.componentcontainer.ComponentContainer
    properties
        Image
        Mask
        SavedMask
        SegmentationAlpha = 0.5
    end
    properties (Access = private)
        Model           % segmentAnythingModel object
        Embeddings      % Extracted once per image
        Viewer          % images.ui.graphics.Viewer
        ImageObject     % Image displayed in viewer
        Bbox            % Current bounding box
        SegmentRequired % Flag to trigger update
        CurrentLabelID = 1
        IsForegroundMode = true
    end
    methods (Access = protected)
        function setup(comp)
            comp.Viewer = viewer2d(comp);
            comp.ImageObject = imageshow([], Parent=comp.Viewer);
            comp.Viewer.AnnotationAddedFcn = @(~,evt) annotationAdded(comp, evt);
        end
    end
end
```

### Key Workflow: Load → Embed → Annotate → Preview → Commit

```matlab
% 1. Load image and extract embeddings ONCE
function loadImage(comp, imageData)
    comp.Image = imageData;
    comp.Model = segmentAnythingModel("sam2-large");
    comp.Embeddings = extractEmbeddings(comp.Model, imageData);
    comp.ImageObject.Data = imageData;
    comp.SavedMask = zeros(size(imageData, [1 2]), "uint8");
end
```

### Annotation-Driven Segmentation (update method)

The `update` method fires whenever annotations change, providing real-time preview:

```matlab
function update(comp)
    if ~comp.SegmentRequired, return; end
    comp.SegmentRequired = false;

    imageSize = size(comp.ImageObject.Data);
    annotations = comp.Viewer.Annotations;

    % Collect foreground/background points from annotations
    foreground = [];
    background = [];
    for idx = 1:numel(annotations)
        if isa(annotations(idx), "images.ui.graphics.roi.Point") && isvalid(annotations(idx))
            if annotations(idx).UserData  % true = foreground
                foreground(end+1,:) = annotations(idx).Position(1:2);
            else
                background(end+1,:) = annotations(idx).Position(1:2);
            end
        end
    end

    if isempty(foreground) && isempty(comp.Bbox), return; end

    % Segment using current prompts
    mask = segmentObjectsFromEmbeddings(comp.Model, comp.Embeddings, imageSize, ...
        ForegroundPoints=foreground, BoundingBox=comp.Bbox, BackgroundPoints=background);

    % Preview: overlay new mask on saved mask
    priorMask = comp.SavedMask;
    priorMask(mask) = uint8(comp.CurrentLabelID);
    comp.ImageObject.OverlayData = priorMask;
end
```

### Handling Annotations via Viewer Events

Use the `AnnotationAdded` event from `Viewer` to process user interactions:

```matlab
function annotationAdded(comp, event)
    event.Annotation.Interactions = "click";

    if isa(event.Annotation, "images.ui.graphics.roi.Rectangle")
        % Bounding box drawn — save position, switch to point mode
        comp.Bbox = event.Annotation.Position;
        comp.Viewer.Mode.Annotate.Style = "point";
        comp.Viewer.Mode.Annotate.ContinueAnnotating = true;
        comp.Viewer.Mode.CurrentMode = "annotate";
    else
        % Point — validate it's inside bounding box
        if ~isempty(comp.Bbox)
            pos = event.Annotation.Position;
            if pos(1) < comp.Bbox(1) || pos(2) < comp.Bbox(2) || ...
                    pos(1) > (comp.Bbox(1)+comp.Bbox(3)) || pos(2) > (comp.Bbox(2)+comp.Bbox(4))
                delete(event.Annotation);
                return;
            end
        end
        event.Annotation.UserData = comp.IsForegroundMode;
    end
    comp.SegmentRequired = true;
end
```

### Commit / Revert Pattern

```matlab
% Commit current preview to saved mask
function saveMask(comp)
    comp.SavedMask = comp.ImageObject.OverlayData;
    comp.Mask = comp.SavedMask;
    comp.Viewer.Annotations = [];
    comp.Bbox = [];
end

% Revert to last saved state
function clearAll(comp)
    comp.Viewer.Annotations = [];
    comp.ImageObject.OverlayData = comp.SavedMask;
    comp.Bbox = [];
end
```

### App-Level Integration

Wrap the component in an App Designer app with toolstrip controls:

```matlab
function startupFcn(app)
    app.SamSegmenter = SAMSegmenter(app.WorkingAreaGrid);
end

function loadImageFromFile(app)
    [file, location] = uigetfile("*.*", "Select an image");
    if ~isequal(file, 0)
        d = uiprogressdlg(app.UIFigure, Title="Please Wait", ...
            Message="Generating SAM embeddings...", Indeterminate="on");
        imageData = imread(fullfile(location, file));
        loadImage(app.SamSegmenter, imageData);
        close(d);
    end
end

function exportMask(app)
    mask = app.SamSegmenter.Mask;
    assignin("base", "SegmentedImage", mask);
end
```

### Design Guidelines

- **Extract embeddings once** per image — show a progress dialog since it's expensive
- **Use `Viewer` with `OverlayData`** for real-time mask overlay (not `labeloverlay` + redraw)
- **Validate point positions** — discard points outside the active bounding box
- **Commit/revert pattern** — let users build up multi-label masks incrementally
- **`ComponentContainer` inheritance** — isolates segmentation logic from app layout for reusability
- Use `Viewer.Mode.Annotate` to control annotation style (point vs rectangle) and continuation

This pattern extends beyond SAM — replace the `segmentObjectsFromEmbeddings` call with any interactive segmentation method (`grabcut`, `activecontour`, `imsegfmm`, etc.) to build custom annotation apps for any interactive segmentation workflow.

## Displaying Segmentation Results

```matlab
% Direct overlay display (preferred — one step, interactive)
imageshow(I, OverlayData=BW);

% For generating an RGB array (e.g., saving to file or further processing)
rgb = labeloverlay(I, BW, Transparency=0.5);
```

## Conventions

- Always compute `superpixels` before `grabcut` or `lazysnapping`
- For prompted SAM, extract embeddings once per image — never per query
- Use `bwareafilt` or `imfill` to clean up after interactive segmentation (see `morphological-ops.md`)
- Prompted SAM is a strong default when CVT+DLT are available; use classical methods when the task is better defined by intensity, edges, or an initial mask

## Common Mistakes

| Mistake | Why it's wrong | Correct approach |
|---------|---------------|-----------------|
| `grabcut(I, [], ROI)` without superpixels | Requires label matrix input | `[L,N] = superpixels(I,500)` then `grabcut(I,L,ROI)` |
| `extractEmbeddings` per bounding box | Embeddings are expensive | Extract once, call `segmentObjectsFromEmbeddings` multiple times |
| `activecontour` without initial mask | Requires a mask to evolve from | Threshold + dilate, or bounding rectangle |
| `regiongrowing(I, seed)` | Does not exist in IPT | Use `activecontour`, `imsegfmm`, or `grayconnected` |
| SAM coordinates in (row, col) | SAM uses (x, y) order | `ForegroundPoints=[x y]`, not `[row col]` |
| `graydiffweight(I, row, col)` or `imsegfmm(W, row, col, t)` | These take **(C, R)** order — column first, then row | `graydiffweight(I, col, row)`, `imsegfmm(W, col, row, t)` |
| `grayconnected(I, r, c, Tolerance=20)` | Tolerance is positional, not a Name-Value pair | `grayconnected(I, r, c, 20)` |
| `imseggeodesic(grayImg, ...)` | Requires RGB input; returns label matrix, not binary | `L = imseggeodesic(RGB, scribble1, scribble2)` |
| `imshow(I, OverlayData=BW)` | `imshow` does not support `OverlayData` | `imageshow(I, OverlayData=BW)` (R2024b+) |

----

Copyright 2026 The MathWorks, Inc.

----
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 321kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall