Skip to content

brycewang-stanford/Auto-Empirical-Research-Skills / skills/23-Learning-Bayesian-Statistics-baygent-skills/bayesian-workflow

bayesian-workflow

Verified

Opinionated Bayesian modeling workflow with PyMC and ArviZ. Contains critical guardrails (nutpie sampler, prior/posterior predictive checks, LOO-PIT calibration, prior sensitivity checks, 94% HDI, non-centered parameterizations, reproducible seeds) that agents won't apply unprompted — always consult before writing Bayesian model code. Trigger on: building probabilistic/Bayesian models, prior elicitation, MCMC inference, convergence diagnostics (divergences, R-hat, ESS), model comparison (LOO-CV, ELPD, stacking weights), hierarchical/multilevel models, count regressions, logistic regression with uncertainty, prior sensitivity analysis, reporting Bayesian results, or mentions of PyMC, ArviZ, InferenceData, credible intervals, posterior distributions, shrinkage, uncertainty quantification. Also trigger for model comparison, diagnosing sampling problems, choosing priors, or presenting stats to non-technical audiences.

FileKindSize
README.mdmarkdown3.9 KB
SKILL.mdskill md13.9 KB
main.pyscript95 B
pyproject.tomlmanifest181 B
references/diagnostics.mdmarkdown5.6 KB
references/hierarchical.mdmarkdown7.8 KB
references/model-comparison.mdmarkdown3.9 KB
references/model-criticism.mdmarkdown11.4 KB
references/priors.mdmarkdown7.8 KB
references/reporting.mdmarkdown8.8 KB
references/sensitivity.mdmarkdown4.5 KB
scripts/calibration_check.pyscript9.8 KB
scripts/diagnose_model.pyscript6.1 KB

references/sensitivity.md

View on GitHub
# Prior/Likelihood Sensitivity Analysis

Power-scaling sensitivity analysis checks whether your posterior conclusions are robust to reasonable changes in prior (or likelihood) strength — without refitting the model. It uses Pareto-smoothed importance sampling (PSIS) to simulate what would happen if you made your priors stronger or weaker.

## Contents
- Requirements
- Running sensitivity checks
- Interpreting results
- Which variables to check
- Visual diagnostics
- Key principle

## Requirements

The InferenceData object must contain both `log_likelihood` and `log_prior` groups.

**Important**: nutpie silently ignores `idata_kwargs={"log_likelihood": True}` and `idata_kwargs={"log_prior": True}`. You must compute these after sampling:

```python
# After pm.sample():
pm.compute_log_likelihood(idata, model=model)
pm.compute_log_prior(idata, model=model)
```

Verify they're present before running sensitivity checks:

```python
assert "log_likelihood" in idata, "Missing log_likelihood — run pm.compute_log_likelihood(idata, model=model)"
assert "log_prior" in idata, "Missing log_prior — run pm.compute_log_prior(idata, model=model)"
```

## Running sensitivity checks

```python
from arviz_stats import psense_summary

summary = psense_summary(idata)
summary
```

This returns a per-variable table with Cumulative Jensen-Shannon (CJS) divergence values for both prior and likelihood perturbations. CJS > 0.05 flags sensitivity (roughly equivalent to a 0.3 standard deviation shift in the posterior mean).

## Interpreting results

Four diagnostic patterns:

| Pattern | Prior CJS | Likelihood CJS | What it means | What to do |
|---------|-----------|----------------|---------------|------------|
| **Low sensitivity** | < 0.05 | < 0.05 | Posterior is robust to prior/likelihood changes | Nothing — this is the ideal outcome |
| **Prior-data conflict** | > 0.05 | > 0.05 | Prior and data pull in different directions | Investigate whether the prior reflects genuine domain knowledge or is just wrong. Consider empirically-scaled priors (e.g., `β ~ Normal(0, 2.5 * sd_y / sd_x)`) |
| **Strong prior / weak likelihood** | > 0.05 | < 0.05 | Prior dominates the posterior | Check if this is intentional (e.g., strong domain constraint). If not, weaken the prior or collect more data |
| **Likelihood-driven** | < 0.05 | > 0.05 | Data dominates the posterior | Usually fine — note in report for transparency |

## Which variables to check

Not every parameter needs sensitivity analysis. Focus on what matters:

- **Check**: interpretable coefficients, effect sizes, predictions, derived quantities (e.g., Bayesian R², contrasts)
- **Skip**: group-specific parameters in hierarchical models (power-scale only the top-level hyperpriors), spline/GP basis coefficients, variance components you don't interpret directly

For hierarchical models, sensitivity of the hyperprior (e.g., the group-level standard deviation) is more informative than sensitivity of individual group effects.

```python
# Check only specific variables
summary = psense_summary(idata, var_names=["beta", "sigma"])
```

## Visual diagnostics

### `plot_psense_dist` — How the posterior shifts

Shows posterior marginals at three power-scaling levels (α = 0.8, 1.0, 1.25). Use this to see the *direction and magnitude* of the shift, not just whether it exceeds the threshold.

```python
from arviz_plots import plot_psense_dist

plot_psense_dist(idata, var_names=["beta"])
```

Requires `arviz-plots` (`pip install arviz-plots`).

### `plot_psense_quantities` — Sensitivity of derived quantities

Shows how predictions or summary statistics shift under perturbation. Use this when you care more about predictive robustness than individual parameter sensitivity.

```python
from arviz_plots import plot_psense_quantities

plot_psense_quantities(idata)
```

## Key principle

**Sensitivity warnings are not automatic problems.** An intentionally informative prior — grounded in domain knowledge or previous studies — will legitimately flag as sensitive. That's expected: if you have a strong prior and modest data, the prior *should* matter.

The correct response to a sensitivity flag is:
1. **Document** the flag and its magnitude
2. **Justify** why the prior is appropriate (or acknowledge it isn't)
3. **Report** the sensitivity transparently — readers should know which conclusions depend on prior choices

Do not reflexively loosen priors to silence diagnostics. A well-justified informative prior that flags sensitivity is better science than a vague prior that passes all checks but encodes no knowledge.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall