Skip to content

K-Dense-AI/scientific-agent-skills / skills/shap

Explain and audit machine-learning predictions with SHAP. Use for selecting SHAP explainers and maskers, computing and validating feature attributions, handling multi-output explanations, and producing local or global SHAP visualizations.

FileKindSize
SKILL.mdskill md16.1 KB
references/data-maskers.mdmarkdown11.6 KB
references/explainers.mdmarkdown17.5 KB
references/migration.mdmarkdown9.8 KB
references/modalities.mdmarkdown12.3 KB
references/plots.mdmarkdown12.2 KB
references/theory.mdmarkdown12.7 KB
references/troubleshooting.mdmarkdown14.3 KB
references/workflows.mdmarkdown19.3 KB
scripts/tabular_report.pyscript11.1 KB

references/modalities.md

View on GitHub
# Text, Images, and Deep Models

Framework/model integration fragments below are illustrative: they were checked against SHAP 0.52.0 source but not run with pretrained weights or a TensorFlow/PyTorch/Transformers installation during this review. Tiny local text and constant-image masking games are tested separately.

Structured inputs require structured maskers. Token deletion, image inpainting, and neural-network reference activations define different explanation games; do not reduce them to ordinary independent tabular columns without justification.

## General Pattern

1. Define a batched model callable with named outputs.
2. Choose a masker that produces valid model inputs.
3. Restrict outputs and evaluation rows before expensive computation.
4. Inspect result shapes.
5. Validate output reconstruction when the explainer supports it.
6. Run masking and baseline sensitivity checks.

The `max_evals`/`outputs` pattern below applies to supported permutation/partition paths; it is not a universal call signature for tree, linear, or deep explainers.

```python
explainer = shap.Explainer(
    model_fn,
    masker,
    output_names=output_names,
    seed=7,
)
explanation = explainer(
    inputs,
    max_evals=evaluation_budget,
    outputs=selected_outputs,
)
```

## Text Classification

Use the model's tokenizer:

```python
masker = shap.maskers.Text(tokenizer)
explainer = shap.Explainer(
    model_fn,
    masker,
    algorithm="partition",
    output_names=class_names,
    seed=7,
)

all_outputs = explainer(texts)
class_exp = all_outputs[..., class_index]
shap.plots.text(class_exp)
```

The model callable should accept a batch of strings and return a 2-D numeric array `(batch, outputs)`.

```python
def model_fn(text_batch):
    encoded = tokenizer(
        list(text_batch),
        padding=True,
        truncation=True,
        return_tensors="pt",
    ).to(device)

    with torch.inference_mode():
        logits = model(**encoded).logits

    return logits.detach().cpu().numpy()
```

Decide whether to explain:

- raw class logits (multiclass softmax logits are not per-class log-odds);
- explicit binary or one-versus-rest log-odds, when that is the intended output;
- probabilities, which stakeholders may understand but couple classes through normalization;
- another scalar score.

State the choice in every plot.

### Transformers pipelines

SHAP provides `shap.models.TransformersPipeline`. Configure the existing text-classification pipeline with `top_k=None` so every class score is returned; a top-one pipeline leaves other columns at zero in this wrapper. Validate the complete label mapping and finite output values on a tiny batch before explaining:

```python
wrapped = shap.models.TransformersPipeline(
    classifier_pipeline,
    rescale_to_logits=True,
)
masker = shap.maskers.Text(classifier_pipeline.tokenizer)
explainer = shap.Explainer(wrapped, masker, algorithm="partition")
explanation = explainer(texts)
```

`rescale_to_logits=True` applies `log(p / (1-p))` independently to each returned class probability. These are one-versus-rest log-odds, not recovered multiclass model logits; scores at 0 or 1 produce infinities. Prefer an explicit model callable when original logits are required. Do not compare these values directly with probability-space explanations.

SHAP 0.52.0's upstream test extras pin `transformers<4.54.0`; this is a compatibility warning, not proof of current Transformers 5 support. Test a separately locked integration rather than promising compatibility from the wrapper's existence. Generation wrappers below have the same framework-validation requirement.

### Text masking choices

Check:

- tokenizer special tokens;
- subword splitting;
- mask token;
- whether consecutive masked tokens are collapsed;
- maximum sequence length and truncation;
- normalization and whitespace behavior;
- language-specific token boundaries.

SHAP can merge tokens hierarchically for display. A displayed phrase attribution may be a grouped coalition, not the sum from an independent-token game.

### Generative and sequence-to-sequence models

SHAP includes model wrappers such as:

- `TeacherForcing`;
- `TextGeneration`;
- `TopKLM`.

Generation explanations are target- and decoding-dependent. Record:

- generated or fixed target sequence;
- teacher-forcing setup;
- decoding parameters;
- top-k output selection;
- tokenizer/model revision.

Do not describe one generation explanation as a general explanation of the language model.

## Image Classification

`shap.maskers.Image` needs OpenCV (`cv2`) for string-based blur/inpainting, not for a numeric constant/array baseline. OpenCV is not installed by the `plots` extra. Add a compatible OpenCV build only when those methods are required. Inpainting converts pixels to `uint8`; use a suitable 0–255 image representation and put the model's normalization inside the callable. Applying it directly to normalized floats can destroy image information.

Wrap preprocessing inside the model callable so masked images receive the same transformations:

```python
def image_model_fn(image_batch):
    prepared = preprocess(image_batch.copy())
    return model(prepared)
```

Choose an image masker:

```python
masker = shap.maskers.Image(
    "inpaint_telea",
    shape=image_shape,
)

explainer = shap.Explainer(
    image_model_fn,
    masker,
    output_names=class_names,
    seed=7,
)

explanation = explainer(
    images,
    max_evals=500,
    batch_size=50,
    outputs=shap.Explanation.argsort.flip[:3],
)

shap.plots.image(explanation)
```

The current constructor also accepts blur specifications or constant mask values. Compare plausible alternatives:

```python
inpaint = shap.maskers.Image("inpaint_telea", image_shape)
blur = shap.maskers.Image("blur(32,32)", image_shape)
zero = shap.maskers.Image(0, image_shape)
```

These answer different questions:

- inpainting replaces a region using nearby pixels;
- blur removes high-frequency detail while retaining coarse structure;
- a constant value creates a fixed visual baseline.

None guarantees an in-distribution counterfactual.

### Output selection

Image classifiers may have thousands of outputs. Use:

```python
outputs=shap.Explanation.argsort.flip[:k]
```

The selected outputs can vary by row. Preserve `output_indexes`/`output_names` and never assume position zero refers to the same class for every image when ranked outputs are used.

### Interpreting image overlays

Attribution overlays:

- explain a model score under a region-masking game;
- do not identify objects with segmentation accuracy;
- do not prove the model "looked at" a region causally;
- can change with resize, crop, normalization, and masker.

Test:

- multiple maskers;
- nearby images/augmentations;
- different evaluation budgets;
- class-output selection;
- consistency with predictive errors.

## PyTorch Deep Models

Before constructing the explainer, switch the PyTorch `nn.Module` to evaluation mode with its standard `eval` method. This disables training behavior such as dropout; it is not Python's built-in code-evaluation function.

```python
background = training_tensor[background_indices].to(device)
to_explain = test_tensor[:batch_size].to(device)

explainer = shap.DeepExplainer(model, background)
values = explainer.shap_values(to_explain, check_additivity=True)
```

For layer-input attribution:

```python
explainer = shap.DeepExplainer(
    (model, model.feature_extractor.target_layer),
    background,
)
```

Requirements:

- model inputs match background tensor structure;
- PyTorch output has shape `(batch, outputs)`, including `(batch, 1)` for one output, rather than a squeezed `(batch,)`;
- all relevant operations have supported attribution rules;
- dropout/batch normalization are in inference mode.

Do not disable `check_additivity` merely to suppress unsupported-operation failures.

## TensorFlow/Keras Deep Models

```python
background = X_train[background_indices]
to_explain = X_test[:batch_size]

explainer = shap.DeepExplainer(model, background)
values = explainer.shap_values(to_explain, check_additivity=True)
```

SHAP 0.46 added NumPy 2, Keras 3, and TensorFlow 2.16 compatibility. Later versions may still have architecture-specific limitations. Verify installed TensorFlow/Keras versions against SHAP release notes and test a minimal batch.

For graph-specific use, TensorFlow may accept an `(input_tensors, output_tensor)` pair. The selected output tensor should be one-dimensional per sample.

## `GradientExplainer`

Expected gradients can be an alternative when DeepExplainer lacks an operator rule:

```python
explainer = shap.GradientExplainer(
    model,
    background,
    batch_size=50,
    local_smoothing=0,
)
values = explainer.shap_values(
    to_explain,
    nsamples=200,
    rseed=7,
)
```

Report:

- background;
- `nsamples`;
- seed;
- smoothing;
- selected output;
- whether values were stable at a larger sample count.

## Output Shapes

Since SHAP 0.45:

- one input, one output: `(samples, *input_shape)` or an additional final singleton output axis; the PyTorch Deep 0.52.0 backend stacks even one output;
- one input, multiple outputs: `(samples, *input_shape, outputs)`;
- multiple inputs: a list with one array per model input.

For `deep_explainer.shap_values(X, ranked_outputs=k)`, the return is `(values, indexes)`; `ranked_outputs` is not a constructor argument. The indexes tell which outputs were selected for each row.

Always inspect rather than branch on a remembered version:

```python
if isinstance(values, list):
    print([value.shape for value in values])
else:
    print(values.shape)
```

## Genomics and Other Sequences

Define the attribution unit:

- nucleotide or amino acid;
- one-hot channel;
- k-mer;
- motif;
- position window;
- assay channel.

Independent one-hot channels can create invalid bases. Prefer a masker or grouping that preserves valid sequence states. For motif-level claims, aggregate or test motifs explicitly and validate across background sequences.

For genomic models, background choice can represent:

- genomic distribution;
- dinucleotide-shuffled controls;
- matched GC/content cohorts;
- experimentally defined controls.

Each implies a different baseline and scientific question.

## Multiple Inputs

For models with multiple inputs, use a compatible list/composite masker or framework explainer:

```python
values = deep_explainer.shap_values([input_a, input_b])
```

Do not sum values across inputs unless they share the same additive output and the aggregation is documented. Keep names and shapes per input.

## Performance

- Explain a small, predeclared row subset first.
- Restrict outputs.
- Batch model calls.
- Keep background modest and test convergence.
- Increase `max_evals`/`nsamples` only after validating shape and semantics.
- Measure peak accelerator memory.
- Avoid silently falling back to CPU.

Approximation variance and model stochasticity are different. Put the model in deterministic inference mode before estimating explainer variability.

## Privacy

Text and image plots can reproduce sensitive input content. Before export:

- redact direct identifiers;
- avoid embedding full clinical notes or faces when unnecessary;
- separate row identifiers from plot artifacts;
- restrict report access;
- review HTML/JavaScript force or text outputs for embedded raw values.

## Sources

- Text examples: https://shap.readthedocs.io/en/latest/text_examples.html
- Image examples: https://shap.readthedocs.io/en/latest/image_examples.html
- Genomic examples: https://shap.readthedocs.io/en/latest/genomic_examples.html
- Text masker: https://shap.readthedocs.io/en/latest/generated/shap.maskers.Text.html
- Image masker: https://shap.readthedocs.io/en/latest/generated/shap.maskers.Image.html
- DeepExplainer: https://shap.readthedocs.io/en/latest/generated/shap.DeepExplainer.html
- GradientExplainer: https://shap.readthedocs.io/en/latest/generated/shap.GradientExplainer.html
- TransformersPipeline: https://shap.readthedocs.io/en/latest/generated/shap.models.TransformersPipeline.html

- Released source / upstream contract: https://github.com/shap/shap/blob/v0.52.0/pyproject.toml
- Released source / upstream contract: https://github.com/shap/shap/blob/v0.52.0/shap/models/_transformers_pipeline.py
- Released source / upstream contract: https://github.com/shap/shap/blob/v0.52.0/shap/explainers/_deep/deep_pytorch.py
- Released source / upstream contract: https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.TextClassificationPipeline
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall