Skip to content

matlab/matlab-agentic-toolkit / skills-catalog/ai-and-statistics/matlab-engineer-tabular-features

matlab-engineer-tabular-features

Unverified

Use when engineering or selecting the best features for single-response classification or regression in MATLAB, whatever the data's modality — for non-tabular data it routes extraction to a domain skill, then selects, assesses, and delivers on the resulting table. Not for multi-response problems, model training, or raw data acquisition.

FileKindSize
SKILL.mdskill md18.9 KB
manifest.yamlconfig570 B
references/assess.mdmarkdown16.7 KB
references/deliver.mdmarkdown13.0 KB
references/domain-routing.mdmarkdown18.1 KB
references/feature-pool.mdmarkdown10.9 KB
references/intake.mdmarkdown16.9 KB
references/internal-helpers.mdmarkdown12.2 KB
references/provider-protocol.mdmarkdown12.3 KB
references/report.mdmarkdown9.7 KB
references/reproduce-pool.mdmarkdown7.1 KB
references/select.mdmarkdown7.1 KB
scripts/BordaVoter.mother2.3 KB
scripts/ConsensusVoter.mother2.5 KB
scripts/MajorityVoter.mother2.0 KB
scripts/RRFVoter.mother1.9 KB
scripts/assessGenerationStability.mother11.9 KB
scripts/assessKFold.mother20.6 KB
scripts/assessSelectionStability.mother9.8 KB
scripts/baselineComparison.mother18.9 KB
scripts/buildRecipe.mother8.2 KB
scripts/chooseNumBins.mother5.0 KB
scripts/chooseStandardization.mother4.5 KB
scripts/computeNSelect.mother2.9 KB
scripts/createConsensusVoter.mother2.2 KB
scripts/describeFeatures.mother5.1 KB
scripts/expandBlockMap.pbinary440 B
scripts/featureSetQuality.mother12.3 KB
scripts/generateFeatureReport.mother67.2 KB
scripts/generateFeatureReportPdf.mother64.9 KB
scripts/generateFeatures.mother14.6 KB
scripts/hasReportGenerator.mother630 B
scripts/isDomainRecipe.mother1.4 KB
scripts/isStringEmpty.mother577 B
scripts/lambdaSequence.pbinary637 B
scripts/lassoFeatureRanking.pbinary971 B
scripts/macroF1Score.mother931 B
scripts/mrmrFeatureRanking.pbinary239 B
scripts/mutualInformationPanel.pbinary1.1 KB
scripts/ncaFeatureRanking.pbinary464 B
scripts/nogueiraStability.mother3.8 KB
scripts/oobFeatureRanking.pbinary414 B
scripts/plotSelectionConsensus.mother2.7 KB
scripts/plotSelectionDecision.mother6.3 KB
scripts/plotSelectionStability.mother10.2 KB
scripts/plotValidationBaseline.mother6.1 KB
scripts/printSummary.mother18.3 KB
scripts/profileForSplit.mother6.2 KB
scripts/rankAgreement.mother1.3 KB
scripts/rankFeatures.mother6.2 KB
scripts/reserveHoldoutForUser.mother5.5 KB
scripts/runConsensusSelection.mother9.4 KB
scripts/sampleSkewness.mother600 B
scripts/saveResults.mother2.1 KB
scripts/scoreFeaturePanel.pbinary1.6 KB
scripts/screenPredictors.mother13.3 KB
scripts/selectFeatures.mother2.3 KB
scripts/splitStrategy.mother5.4 KB
scripts/stratifiedDownsampleRows.mother2.2 KB
scripts/stratifiedSubsampleIndices.mother3.4 KB
scripts/transformFeatures.mother6.1 KB
scripts/validateRecipe.mother12.3 KB
scripts/writeInferenceScript.mother10.5 KB

scripts/screenPredictors.m

View on GitHub
function [ScreenedTbl, ScreenInfo] = screenPredictors(Predictors, Response, Options)
%screenPredictors Assemble the table and drop degenerate predictors (intake).
%
%   [ScreenedTbl, ScreenInfo] = screenPredictors(RawTbl, Response) returns RawTbl
%   with two kinds of useless predictor removed, so the generator and the
%   consensus selector never waste effort on columns that carry no signal:
%     - CONSTANT predictors: at most one distinct non-missing value (zero
%       variance -- nothing to learn from, and a source of degenerate transforms).
%     - MOSTLY-MISSING predictors: more than MissingThreshold (default 0.95) of
%       the rows missing -- effectively empty, nothing to engineer or impute. The
%       cutoff is deliberately high: many learners (and the generator's own
%       imputation) handle moderately-missing columns natively, so intake only
%       drops the near-empty ones and leaves the missingness call to the producer.
%
%   Response is polymorphic. When it is a scalar string or char (a NAME), it
%   identifies the response column already in Predictors (current behavior above;
%   a Wilkinson formula is accepted). When it is anything else -- a column vector
%   of values (numeric / categorical / logical / string labels / ...) or a
%   single-variable table -- it is treated as the response DATA supplied
%   separately, and is CONCATENATED to Predictors before screening. This handles
%   the common intake shape of an X table plus a y vector: a nameless vector is
%   appended under Options.ResponseName (default "Response"), while a
%   single-variable table keeps its own variable name. Downstream sees one
%   assembled predictors + response table either way.
%
%   This is a light-weight, structural screen run at INTAKE, on the raw input
%   predictors, before problem profiling / splitting / generation. It is NOT
%   response-dependent (it never inspects the response values), so running it on
%   the whole table before the train/test split introduces no leakage. The
%   response column is always kept, whatever its contents.
%
%   Screening is deliberately minimal: everything else (imputation, encoding,
%   scaling, richer feature construction) is delegated to the generator
%   (gencfeatures) or a domain skill. This step only removes columns that would
%   be dead weight for any producer.
%
%   Inputs:
%     Predictors - table of raw predictors. When Response is a NAME it must also
%                  contain that response column; when Response is DATA it holds
%                  predictors only.
%     Response   - either a response variable NAME (scalar string/char; a
%                  Wilkinson formula is accepted, its left-hand side naming the
%                  column to protect) already in Predictors, OR the response DATA
%                  supplied separately (a column vector or single-variable table,
%                  height matching Predictors) to concatenate before screening. A
%                  single-variable table is SELF-NAMING: its variable name becomes
%                  the response column name.
%     Options.ResponseName - (1,1) string, the column name under which a NAMELESS
%                  response vector is appended. Default "Response". Ignored when
%                  Response is a name, or a single-variable table (which names
%                  itself).
%     Options.MissingThreshold - (1,1) double in [0,1], the missing-fraction above
%                  which a predictor is dropped. Default 0.95 (drop when strictly
%                  more than 95% of rows are missing). Kept high on purpose:
%                  models with native missing handling and the generator's
%                  imputation can use moderately-missing columns, so only the
%                  near-empty ones are pruned here.
%     Options.ConstantTolerance - (1,1) nonnegative double, the tolerance for the
%                  constant check on NUMERIC columns. A numeric column is constant
%                  when its spread (max - min of present values) is at most
%                  ConstantTolerance * max(scale, 1), where scale is the largest
%                  absolute present value. Default 1e-9 (catches floating-point
%                  noise around a genuinely single value without pruning columns
%                  that carry real, if small, variation). Non-numeric columns
%                  ignore this and use an exact distinct-value count.
%
%   Outputs:
%     ScreenedTbl - the assembled predictors + response with dropped predictors
%                   removed (survivor column order, and the response, preserved)
%     ScreenInfo  - struct with fields:
%                     .Kept            - string array of kept predictor names
%                     .Dropped         - string array of dropped predictor names
%                     .DroppedReasons  - string array parallel to .Dropped
%                                        ("constant" | "missing>NN%")
%                     .NumDropped      - number of predictors dropped
%                     .MissingThreshold- the missing-fraction threshold applied
%                     .ConstantTolerance- the numeric constant tolerance applied
%                     .CombinedResponse- true if ResponseData was appended here
%                     .ResponseVar     - the resolved response column name
%                     .Reasoning       - human-readable summary
%
%   Errors with id screenPredictors:tooFewObservations if the assembled table has
%   fewer than 2 rows, and with id screenPredictors:noPredictorsRemain if every
%   predictor is dropped (neither can proceed to generation/selection).

% Copyright 2026 The MathWorks, Inc.

    arguments
        Predictors table
        Response
        Options.ResponseName (1,1) string = "Response"
        Options.MissingThreshold (1,1) double {mustBeGreaterThanOrEqual(Options.MissingThreshold, 0), ...
            mustBeLessThanOrEqual(Options.MissingThreshold, 1)} = 0.95
        Options.ConstantTolerance (1,1) double {mustBeNonnegative} = 1e-9
    end

    % Response is a NAME (scalar string/char) -> already a column of Predictors;
    % otherwise it is response DATA supplied separately -> concatenate it.
    CombinedResponse = ~isResponseName(Response);
    if CombinedResponse
        [RawTbl, ResponseVar] = combineResponse(Predictors, Options.ResponseName, Response);
    else
        RawTbl = Predictors;
        ResponseVar = resolveResponseName(string(Response), RawTbl);
    end

    % Too few observations to do anything meaningful (generate, split, rank).
    % Guard the assembled table BEFORE screening...
    checkEnoughRows(RawTbl);

    AllVars = string(RawTbl.Properties.VariableNames);
    PredictorVars = AllVars(AllVars ~= ResponseVar);

    Dropped = string.empty(1, 0);
    DroppedReasons = string.empty(1, 0);
    MissingPct = round(100 * Options.MissingThreshold);

    for Name = PredictorVars
        Col = RawTbl.(Name);
        MissingFraction = mean(ismissing(Col));
        if MissingFraction > Options.MissingThreshold
            Dropped(end+1) = Name; %#ok<AGROW>
            DroppedReasons(end+1) = sprintf("missing>%d%%", MissingPct); %#ok<AGROW>
        elseif isConstantColumn(Col, Options.ConstantTolerance)
            Dropped(end+1) = Name; %#ok<AGROW>
            DroppedReasons(end+1) = "constant"; %#ok<AGROW>
        end
    end

    Kept = setdiff(PredictorVars, Dropped, "stable");
    if isempty(Kept)
        error("screenPredictors:noPredictorsRemain", ...
            ['All %d predictor(s) were dropped as constant or mostly-missing; ' ...
            'no usable predictor remains to engineer features from.'], ...
            numel(PredictorVars));
    end

    % Keep the survivors in their original order, response included.
    ScreenedTbl = RawTbl(:, [Kept, ResponseVar]);

    % ...and again AFTER screening. Screening never removes rows, so this is a
    % defensive invariant on the delivered table rather than a second filter.
    checkEnoughRows(ScreenedTbl);

    ScreenInfo = struct( ...
        Kept = Kept, ...
        Dropped = Dropped, ...
        DroppedReasons = DroppedReasons, ...
        NumDropped = numel(Dropped), ...
        MissingThreshold = Options.MissingThreshold, ...
        ConstantTolerance = Options.ConstantTolerance, ...
        CombinedResponse = CombinedResponse, ...
        ResponseVar = ResponseVar, ...
        Reasoning = buildReasoning(Kept, Dropped, DroppedReasons, MissingPct));

    fprintf('Intake screen: kept %d predictor(s), dropped %d (constant / >%d%% missing).\n', ...
        numel(Kept), numel(Dropped), MissingPct);
end


function Tf = isConstantColumn(Col, Tolerance)
%isConstantColumn True when a column has at most one distinct non-missing value.
%   Works for numeric, categorical, string, and other scalar-valued columns; an
%   all-missing column reads as constant, but the missing screen catches those
%   first, so this only fires on genuinely single-valued columns.
%
%   NUMERIC columns use a scale-relative TOLERANCE rather than exact equality: a
%   column is constant when the spread (max - min) of its present values is at
%   most Tolerance * max(scale, 1), where scale is the largest absolute present
%   value. This treats floating-point noise around one value as constant without
%   demanding bit-exact repeats, and the scaling keeps the test meaningful for
%   both tiny- and large-magnitude columns. Non-numeric columns (categorical,
%   string, logical, ...) have no meaningful tolerance and use an exact
%   distinct-value count.
    Present = Col(~ismissing(Col));
    if isnumeric(Present)
        if numel(Present) <= 1
            Tf = true;
            return;
        end
        Spread = max(Present) - min(Present);
        Scale = max(abs(Present), [], "omitnan");
        Tf = Spread <= Tolerance * max(Scale, 1);
    else
        Tf = numel(unique(Present)) <= 1;
    end
end


function ResponseVar = resolveResponseName(Response, RawTbl)
%resolveResponseName Extract the response variable name from a name or formula.
    Response = string(Response);
    if contains(Response, "~")
        Parts = split(Response, "~");
        ResponseVar = strtrim(Parts(1));
    else
        ResponseVar = strtrim(Response);
    end
    if ~ismember(ResponseVar, string(RawTbl.Properties.VariableNames))
        error("screenPredictors:responseNotFound", ...
            'Response variable "%s" is not a column of the input table.', ResponseVar);
    end
end


function checkEnoughRows(Tbl)
%checkEnoughRows Halt when a table has fewer than 2 observations.
%   Called on the assembled table before screening and on the screened table
%   after, so a too-small dataset is refused at both boundaries.
    if height(Tbl) < 2
        error("screenPredictors:tooFewObservations", ...
            ['Input has %d observation(s); at least 2 are required to engineer ' ...
            'and select features.'], height(Tbl));
    end
end


function Tf = isResponseName(Response)
%isResponseName True when Response is a NAME (scalar string / char row) rather
%   than response DATA. A string ARRAY (vector of labels) reads as data, matching
%   the rule "a vector is a separately-supplied response".
    Tf = (isstring(Response) && isscalar(Response)) || (ischar(Response) && isrow(Response));
end


function [RawTbl, ResponseVar] = combineResponse(Predictors, ResponseName, ResponseData)
%combineResponse Append a separately-supplied response to the predictor table.
%   ResponseData is a raw column vector (nameless) or a single-variable table
%   (self-naming), with height matching Predictors. A single-variable table
%   carries its own variable name, which is honored as the response column name;
%   a nameless vector is named by ResponseName. The response is appended as the
%   last column and its name must not collide with an existing predictor.

    % Accept a single-variable table (self-naming) or a raw column vector.
    if istable(ResponseData)
        if width(ResponseData) ~= 1
            error("screenPredictors:responseDataNotSingleVar", ...
                "A response table must have exactly one variable.");
        end
        ResponseVar = string(ResponseData.Properties.VariableNames(1));
        ResponseCol = ResponseData{:, 1};
    else
        ResponseVar = strtrim(string(ResponseName));
        ResponseCol = ResponseData;
    end

    if ismember(ResponseVar, string(Predictors.Properties.VariableNames))
        error("screenPredictors:responseNameCollision", ...
            'Response name "%s" already exists in the predictor table.', ResponseVar);
    end
    if size(ResponseCol, 1) ~= height(Predictors)
        error("screenPredictors:responseHeightMismatch", ...
            'The response has %d rows but the predictor table has %d.', ...
            size(ResponseCol, 1), height(Predictors));
    end

    RawTbl = Predictors;
    RawTbl.(ResponseVar) = ResponseCol;
end


function Reasoning = buildReasoning(Kept, Dropped, DroppedReasons, MissingPct)
%buildReasoning Compose a one-paragraph, human-readable screen summary.
    if isempty(Dropped)
        Reasoning = sprintf("No predictors screened out: all %d are non-constant " + ...
            "and at most %d%% missing.", numel(Kept), MissingPct);
        return;
    end
    Parts = strings(1, numel(Dropped));
    for i = 1:numel(Dropped)
        Parts(i) = sprintf("%s (%s)", Dropped(i), DroppedReasons(i));
    end
    Reasoning = sprintf("Dropped %d of %d predictors: %s. Kept %d.", ...
        numel(Dropped), numel(Kept) + numel(Dropped), ...
        strjoin(Parts, ", "), numel(Kept));
end
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 466kUpdated Claude CodeCodexVerifiedInstall
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 466kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 319kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 319kUpdated Claude CodeCodexVerifiedInstall