Walsworth-Group/physkit / skills/model-selection
model-selection
VerifiedCompare and discriminate competing physical models (campaign archetype I).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 6.2 KB |
SKILL.md
View on GitHub---
name: model-selection
description: >-
Compare and discriminate competing physical models (campaign archetype I).
---
# model-selection — choose the explanation, honestly
Some campaigns decide which mechanism is true rather than optimize a number. Given competing models or
hypotheses, this skill ranks them on the evidence, states the strength of the ranking, designs the
experiment that would settle it when one is needed, and states when no candidate is adequate. It is
archetype I. It pairs with `frame-problem`, which enumerates the physics the competing models embody, and
`verify`, which certifies the verdict.
The deliverable is a decision between explanations, with its strength and scope, not a single best-fit
curve. A model that wins by a narrow margin, or wins only because its rivals were misspecified, is not a
confident answer. Report the margin and its cause.
## Two modes
### Static — compare models on fixed data
When the data are in hand and the task is to rank the models:
- Score by evidence, penalizing complexity. Use the marginal likelihood / Bayes factor, cross-validated
predictive performance, or an information criterion (AIC/BIC/WAIC). Each embodies Occam: a simpler model
that fits comparably wins, because extra flexibility buys fit without buying truth. A change that raises
raw fit error cannot be overfitting; distinguish the two.
- State the priors and assumptions. Bayes factors depend on the priors over each model's parameters. Report
them, and check sensitivity to reasonable alternatives. A conclusion that flips under a defensible prior
change is not robust.
- Report the comparison with uncertainty, not a bare "model B wins": the posterior odds, the
cross-validation spread, and how decisive the margin is.
- Interrogate the residuals of the front-runner (`verify`). Structured residuals mean even the best
candidate has unaccounted-for physics: expand the set rather than crown it.
### Active — design experiments to discriminate (expensive/sequential)
When discriminating requires new, costly experiments, compose `experiment-design` (archetype H) with a
discrimination utility:
- Measure where the models disagree most. The informative experiment is where the candidates predict
different outcomes, not where they already agree: T-optimality, maximum expected change in the model
posterior, or mutual information between outcome and model identity.
- Maintain a calibrated posterior over models and over each model's parameters. Update it as each
experiment lands. Stop when one model is decisively favored or returns diminish.
- Inherit H's disciplines: surrogate and posterior calibration is cardinal, budget tracking, benchmarking
against naive sampling. For hardware, inherit archetype E's safety and drift rules.
## The honest-set discipline (the failure mode that matters most)
The true model may lie outside the candidate set. This is model-form error, and it is how model selection
most often misleads.
- Include a flexible "none-of-the-above" alternative in the set: a more expressive or non-parametric model.
The set must be able to register that no candidate is adequate.
- A best-but-still-poor fit is a signal to expand the set, not a winner. Judge absolute adequacy, not only
relative ranking: does the front-runner explain the data, within its uncertainty?
- Do not let one model's misspecification masquerade as discrimination. A rival can lose because it was
implemented badly rather than because it is wrong. Rule out the implementation with a `debug` or `verify`
pass on each candidate before declaring a mechanism refuted.
- Test where the models are distinguishable, not in a degenerate regime. A negative result measured where
the candidates predict the same outcome proves nothing: comparing a stiff and a soft magnet only in
saturation, where both predict zero canting, cannot separate them. Before writing off a hypothesis,
confirm the experiment sat in the regime where the models diverge. Active mode's "measure where they
disagree most" selects that regime.
## Credit the mechanism, not just the metric
A model-selection result is explanatory: the win is a recovered understanding of why the system behaves as
it does, and it updates the project overview (`PROJECT.md`). Credit the mechanism, the parameter values it implies, and the
predictions it newly makes, not the fact that it scored highest. A descriptive "B fits better" is weaker
than an explanatory "B fits better and implies effect X, which was then observed."
## Composition
- `frame-problem` supplies the candidate models (the competing physics inventories) and the a-priori
expectations each implies.
- `experiment-design` (archetype H) is the active-mode engine for choosing discriminating experiments.
- `verify` certifies the verdict: confirm the margin is real, not a prior artifact, recompute the evidence,
run the anti-overfitting and residual-structure checks, and state what is decided, what is still open,
and over what experimental range.
- `record-decision` captures why a model was selected and the conditions and priors it rests on. `report`
documents the comparison, the evidence, and the recovered mechanism with its uncertainty.
- `new-campaign` scaffolds archetype I. A campaign may pull H and I together (actively discriminate), or
either alone.
## Before you trust the verdict — checklist
- [ ] Models scored by an evidence or predictive criterion that penalizes complexity (Occam), not by raw fit.
- [ ] Priors and assumptions stated; the conclusion is robust to reasonable prior and criterion changes.
- [ ] The front-runner is absolutely adequate (residuals unstructured), not only relatively best.
- [ ] Any hypothesis written off was tested in a regime where the models are distinguishable, not a degenerate regime where both predict the same outcome.
- [ ] A flexible "none-of-the-above" was in the set; model-form error considered.
- [ ] Each candidate's implementation was checked, so a loss reflects the model, not a bug.
- [ ] (Active mode) experiments targeted maximum model disagreement; the model posterior is calibrated; a decisive stop was used.
- [ ] The result is reported as a decision with its strength, scope, and the recovered mechanism, and independently verified.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |