Skip to content

DAAF-Contribution-Community/daaf / .claude/skills/statsmodels

statsmodels

Verified

Statistical modeling: OLS/WLS/GLS, GLM (logit, probit, Poisson), time series (ARIMA, VAR), mixed effects, diagnostics. Formula API. Use for regressions without fixed effects, GLMs, or time series. For FE/DiD and IV with FE use pyfixest; for IV without FE (2SLS, LIML, GMM), panel RE/between and SUR/3SLS use linearmodels. R equivalent: r-stats (use when execution language is R; VAR in R escalates).

FileKindSize
SKILL.mdskill md14.5 KB
references/diagnostics.mdmarkdown12.1 KB
references/glm-discrete.mdmarkdown23.9 KB
references/gotchas.mdmarkdown19.5 KB
references/hypothesis-testing.mdmarkdown11.8 KB
references/linear-models.mdmarkdown19.0 KB
references/quickstart.mdmarkdown17.0 KB
references/time-series.mdmarkdown28.9 KB

references/linear-models.md

View on GitHub
# statsmodels Linear Models Reference

statsmodels v0.14.6 — syntax and library guidance only.

---

## Contents

1. [OLS (Ordinary Least Squares)](#ols-ordinary-least-squares)
2. [WLS (Weighted Least Squares)](#wls-weighted-least-squares)
3. [GLS (Generalized Least Squares)](#gls-generalized-least-squares)
4. [Robust Regression (RLM)](#robust-regression-rlm)
5. [Quantile Regression (QuantReg)](#quantile-regression-quantreg)
6. [Polynomial and Spline Regression](#polynomial-and-spline-regression)
7. [Interaction Terms](#interaction-terms)
8. [Mixed Effects / Multilevel Models (MixedLM)](#mixed-effects--multilevel-models-mixedlm)
9. [References and Further Reading](#references-and-further-reading)

---

## OLS (Ordinary Least Squares)

### Formula API

```python
import statsmodels.formula.api as smf

results = smf.ols("y ~ x1 + x2 + x3", data=df).fit()
print(results.summary())
```

### Array API

```python
import statsmodels.api as sm

X = sm.add_constant(df[["x1", "x2", "x3"]])
results = sm.OLS(df["y"], X).fit()
print(results.summary())
```

`sm.add_constant()` prepends a column of ones to the design matrix. Without it, the model has no intercept. The constant column is named `"const"` by default.

### Accessing Coefficients

```python
# Point estimates
results.params          # Series indexed by variable name

# P-values (two-tailed)
results.pvalues

# Confidence intervals (default alpha=0.05)
results.conf_int()              # 95% CI — returns DataFrame with columns [0, 1] (lower, upper)
results.conf_int(alpha=0.01)    # 99% CI
```

Example — extract a single coefficient with its CI:

```python
coef = results.params["x1"]
ci_lo, ci_hi = results.conf_int().loc["x1"]
print(f"x1: {coef:.4f} [{ci_lo:.4f}, {ci_hi:.4f}]")
```

### Model Fit Statistics

```python
results.rsquared        # R²
results.rsquared_adj    # Adjusted R²
results.aic             # Akaike Information Criterion
results.bic             # Bayesian Information Criterion
results.fvalue          # F-statistic (overall model significance)
results.f_pvalue        # p-value for F-test
results.nobs            # Number of observations
results.df_resid        # Residual degrees of freedom
results.df_model        # Model degrees of freedom (excl. constant)
results.mse_resid       # Mean squared error of residuals (= σ²)
```

### Fitted Values and Residuals

```python
results.fittedvalues    # ŷ — Series, same index as training data
results.resid           # ε = y - ŷ (raw residuals)
results.resid_pearson   # Pearson (standardized) residuals for OLS same as resid / RMSE
```

Studentized residuals (externally studentized, for outlier detection):

```python
influence = results.get_influence()
student_resid = influence.resid_studentized_external  # values > |2| flag potential outliers
```

### Prediction

In-sample fitted values:

```python
results.fittedvalues
```

Out-of-sample point prediction:

```python
new_data = pd.DataFrame({"x1": [1.5, 2.0], "x2": [3.0, 4.5], "x3": [0.1, 0.2]})
y_pred = results.predict(new_data)   # returns Series
```

Out-of-sample prediction with confidence and prediction intervals:

```python
pred = results.get_prediction(new_data)
frame = pred.summary_frame(alpha=0.05)
# Columns: mean, mean_se, mean_ci_lower, mean_ci_upper, obs_ci_lower, obs_ci_upper
# mean_ci_*: confidence interval for the mean response
# obs_ci_*: prediction interval for an individual observation
print(frame)
```

Array API — pass an array with the same column structure as the training X:

```python
X_new = sm.add_constant(pd.DataFrame({"x1": [1.5], "x2": [3.0], "x3": [0.1]}))
y_pred = results.predict(X_new)
```

### Saving and Reloading Results

```python
import pickle

# Save
with open("ols_results.pkl", "wb") as f:
    pickle.dump(results, f)

# Load
with open("ols_results.pkl", "rb") as f:
    results = pickle.load(f)
```

---

## WLS (Weighted Least Squares)

Use when errors have non-constant variance (heteroskedasticity) but the variance structure is known or estimable. Observations with smaller error variance receive higher weight.

### Formula API

```python
# weight = 1 / variance: high variance → low weight → less influence
results = smf.wls("y ~ x1 + x2", data=df, weights=1/df["variance"]).fit()
```

### Array API

```python
X = sm.add_constant(df[["x1", "x2"]])
results = sm.WLS(df["y"], X, weights=1/df["variance"]).fit()
```

### Common Weight Specifications

```python
# Precision (analytic) weights proportional to population. WLS does not treat weights as counts:
# nobs and df_resid stay at the row count. For rows that stand for w_i identical records, use GLM freq_weights (glm-discrete.md)
results = smf.wls("y ~ x1 + x2", data=df, weights=df["population"]).fit()

# Precision weights from measurement error variance
results = smf.wls("y ~ x1", data=df, weights=1/df["meas_var"]).fit()

# Inverse of squared fitted standard deviations (feasible WLS)
ols_resid = smf.ols("y ~ x1", data=df).fit().resid
df["abs_resid"] = ols_resid.abs()
aux = smf.ols("abs_resid ~ x1", data=df).fit()
fitted_sd = aux.fittedvalues
results = smf.wls("y ~ x1", data=df, weights=1/fitted_sd**2).fit()
```

The `weights` parameter in statsmodels WLS is the **inverse of the error variance**, not the square root. Larger weight = observation is more reliable.

### Accessing Results

All standard OLS result attributes apply: `.params`, `.pvalues`, `.conf_int()`, `.fittedvalues`, `.resid`, `.rsquared`, etc.

The R² in WLS is weighted; interpret with care relative to OLS R².

---

## GLS (Generalized Least Squares)

Use when errors are correlated or have non-constant variance and the full covariance structure of errors is known (or estimable).

### Array API

```python
import statsmodels.api as sm
import numpy as np

X = sm.add_constant(df[["x1", "x2"]])
y = df["y"]

# sigma: covariance matrix of errors (n x n)
results = sm.GLS(y, X, sigma=covariance_matrix).fit()
```

`sigma` can be:
- A full `(n, n)` covariance matrix
- A 1-d array of length `n` for diagonal (heteroskedastic only, equivalent to WLS)
- `None` (falls back to OLS)

### GLSAR for AR(p) Error Structure

When errors follow an autoregressive process, `GLSAR` is more practical than specifying the full covariance matrix:

```python
# AR(1) errors: rho=1 sets the AR order; .fit() alone keeps rho at 0 (no AR correction)
results = sm.GLSAR(y, X, rho=1).iterative_fit(maxiter=10)

# AR(2) errors
results = sm.GLSAR(y, X, rho=2).iterative_fit(maxiter=10)
```

Iterative estimation (Cochrane-Orcutt / Prais-Winsten style):

```python
from statsmodels.regression.linear_model import GLSAR

model = GLSAR(y, X, rho=1)
results = model.iterative_fit(maxiter=10)
print(f"Estimated rho: {model.rho}")
print(results.summary())
```

### GLS vs. WLS vs. GLSAR

| Situation | Recommended |
|-----------|-------------|
| Known variance weights only | `sm.WLS` |
| Known full error covariance matrix | `sm.GLS` |
| AR(p) error autocorrelation | `sm.GLSAR` |
| Unknown heteroskedasticity | `sm.OLS` + robust standard errors |

---

## Robust Regression (RLM)

M-estimators that iteratively downweight influential observations (outliers). Use when OLS residual diagnostics show influential outliers but removing them is not justified.

### Basic Usage

```python
import statsmodels.api as sm

X = sm.add_constant(df[["x1", "x2"]])
y = df["y"]

# Default: Huber's T norm
results = sm.RLM(y, X).fit()
print(results.summary())
```

Formula API (via `smf`):

```python
import statsmodels.formula.api as smf

results = smf.rlm("y ~ x1 + x2", data=df).fit()
```

### M-Estimator Norms

All norms are in `sm.robust.norms`:

```python
# Huber's T (default, tuning constant 1.345)
results = sm.RLM(y, X, M=sm.robust.norms.HuberT()).fit()

# Tukey's biweight (fully rejects extreme outliers)
results = sm.RLM(y, X, M=sm.robust.norms.TukeyBiweight()).fit()

# Andrew's wave (sinusoidal redescending)
results = sm.RLM(y, X, M=sm.robust.norms.AndrewWave()).fit()

# Hampel (three-part influence function)
results = sm.RLM(y, X, M=sm.robust.norms.Hampel()).fit()

# LeastSquares (equivalent to OLS — comparison baseline)
results = sm.RLM(y, X, M=sm.robust.norms.LeastSquares()).fit()
```

| Norm | Behavior | Use When |
|------|----------|----------|
| `HuberT()` | Default; tuning constant 1.345 | General robustness |
| `TukeyBiweight()` | Fully rejects extreme outliers | Known heavy outliers |
| `AndrewWave()` | Sinusoidal redescending | Moderate outlier resistance |
| `Hampel()` | Three-part influence function | Flexible outlier handling |
| `LeastSquares()` | Equivalent to OLS | Comparison baseline |

### Accessing Weights and Diagnostics

```python
results.weights         # Per-observation weights (0 = completely downweighted)
results.params          # Coefficient estimates
results.pvalues         # P-values
results.conf_int()      # Confidence intervals
```

Identifying downweighted observations:

```python
import pandas as pd

weight_df = pd.DataFrame({
    "weight": results.weights,
    "resid": results.resid
}, index=df.index)

# Observations with weight below 0.5 are substantially downweighted
outliers = weight_df[weight_df["weight"] < 0.5]
print(f"{len(outliers)} observations downweighted below 0.5")
print(outliers.sort_values("weight"))
```

### Comparing OLS and RLM

```python
ols_res = sm.OLS(y, X).fit()
rlm_res = sm.RLM(y, X).fit()

comparison = pd.DataFrame({
    "OLS": ols_res.params,
    "RLM": rlm_res.params,
    "diff": rlm_res.params - ols_res.params
})
print(comparison)
# Large differences signal that outliers are influencing OLS estimates
```

---

## Quantile Regression (QuantReg)

Models conditional quantiles of y rather than the conditional mean. Useful for understanding distributional effects, heterogeneous treatment effects, and when outliers make OLS mean-unstable.

### Basic Usage

```python
# Formula API — median regression
results = smf.quantreg("y ~ x1 + x2", data=df).fit(q=0.5)

# 25th percentile
results = smf.quantreg("y ~ x1 + x2", data=df).fit(q=0.25)

# 90th percentile
results = smf.quantreg("y ~ x1 + x2", data=df).fit(q=0.90)

# Array API
results = sm.QuantReg(y, X).fit(q=0.75)
```

The `q` parameter is the quantile to estimate (0 < q < 1). `q=0.5` is least absolute deviations (LAD) regression, equivalent to median regression.

### Standard Errors

The default is `vcov='robust'`. Both options use one kernel estimate of the sparsity (1 / density of the residuals at zero): `'iid'` scales (X'X)⁻¹ by q(1−q)·sparsity², and `'robust'` weights each observation by q² or (1−q)² times sparsity², according to the residual's sign. Neither uses observation-level density weights, and at `q=0.5` `'robust'` equals `'iid'` exactly. `QuantReg.fit()` has no bootstrap option; resample by hand if heteroskedasticity-robust inference matters. The options, set explicitly:

```python
# Default standard errors (vcov="robust")
results = smf.quantreg("y ~ x1 + x2", data=df).fit(
    q=0.5,
    vcov="robust",    # default; equals vcov="iid" at q=0.5
    kernel="epa",     # Epanechnikov kernel for sparsity estimation
    bandwidth="hsheather"  # bandwidth selection method
)
```

Available kernels (`kernel=`, for the density estimate): `"epa"` (Epanechnikov, default), `"biw"` (biweight), `"cos"`, `"gau"` (Gaussian), `"par"` (Parzen).

### Fitting Multiple Quantiles

```python
quantiles = [0.10, 0.25, 0.50, 0.75, 0.90]
qr_results = {}

for q in quantiles:
    qr_results[q] = smf.quantreg("y ~ x1 + x2", data=df).fit(q=q)

# Compare x1 coefficient across quantiles
x1_coefs = pd.Series(
    {q: qr_results[q].params["x1"] for q in quantiles},
    name="x1_coef"
)
print(x1_coefs)
```

### Accessing Results

All standard result attributes apply:

```python
results.params
results.pvalues
results.conf_int()
results.fittedvalues
results.resid
results.predict(new_data)
```

Pseudo R² (Koenker-Machado):

```python
results.prsquared   # 1 - sum(check_loss) / sum(check_loss_null)
```

---

## Polynomial and Spline Regression

### Polynomial Terms via `I()`

The `I()` wrapper in patsy formulas passes the expression through as raw Python, bypassing formula operator interpretation:

```python
# Quadratic (degree 2)
results = smf.ols("y ~ x1 + I(x1**2)", data=df).fit()

# Cubic (degree 3)
results = smf.ols("y ~ x1 + I(x1**2) + I(x1**3)", data=df).fit()

# Interaction with polynomial
results = smf.ols("y ~ x1 + I(x1**2) + x2 + x1:x2", data=df).fit()
```

In the array API, add polynomial columns to the DataFrame manually:

```python
X = pd.DataFrame({
    "x1": df["x1"],
    "x1_sq": df["x1"] ** 2,
    "x1_cu": df["x1"] ** 3,
    "x2": df["x2"]
})
X = sm.add_constant(X)
results = sm.OLS(df["y"], X).fit()
```

### B-Spline Basis via patsy `bs()`

```python
from patsy import bs

# Cubic B-spline with 4 degrees of freedom (4 basis columns; the formula adds the intercept)
results = smf.ols("y ~ bs(x1, df=4, degree=3)", data=df).fit()

# Cubic B-spline with knots at specific values
results = smf.ols("y ~ bs(x1, knots=[25, 50, 75], degree=3)", data=df).fit()

# Spline without intercept (no separate constant in the spline)
results = smf.ols("y ~ bs(x1, df=5, degree=3, include_intercept=False)", data=df).fit()
```

`df` (degrees of freedom): number of basis functions, equal to `degree + number_of_interior_knots`. Knot locations default to quantiles of x1.

`degree`:
- 1 = piecewise linear
- 2 = piecewise quadratic
- 3 = cubic (most common, C2 continuity)

Generating the spline basis for prediction or inspection:

```python
from patsy import dmatrix, build_design_matrices

# Fit on training data
basis_train = dmatrix("bs(x1, df=4, degree=3)", data=df, return_type="dataframe")
results = sm.OLS(df["y"], basis_train).fit()

# Apply the training transformation (same knots) to new data
basis_new = build_design_matrices([basis_train.design_info], new_df, return_type="dataframe")[0]
y_pred = results.predict(basis_new)
```

Calling `dmatrix` again on new data recomputes the knots from the new data; `build_design_matrices` with the training `design_info` (or the formula API's `results.predict(new_df)`) reuses the training knots.

---

## Interaction Terms

### Formula API Operators

| Operator | Meaning | Equivalent to |
|----------|---------|---------------|
| `x1 * x2` | Full interaction (main effects + interaction) | `x1 + x2 + x1:x2` |
| `x1 : x2` | Interaction term only (no main effects) | product of x1 and x2 |
| `(x1 + x2 + x3)**2` | All pairwise interactions | `x1 + x2 + x3 + x1:x2 + x1:x3 + x2:x3` |

```python
# Full interaction
results = smf.ols("y ~ x1 * x2", data=df).fit()

# Interaction term only (suppresses main effects)
results = smf.ols("y ~ x1:x2", data=df).fit()

# Categorical interaction — C() marks a column as categorical
results = smf.ols("y ~ x1 * C(group)", data=df).fit()

# Three-way interaction (all lower-order terms included)
results = smf.ols("y ~ x1 * x2 * x3", data=df).fit()

# All pairwise interactions among three variables
results = smf.ols("y ~ (x1 + x2 + x3)**2", data=df).fit()
```

### Categorical Reference Level

By default, patsy uses the first category alphabetically as the reference level. Override:

```python
# Set "control" as the reference level
results = smf.ols("y ~ x1 * C(group, Treatment('control'))", data=df).fit()
```

### Array API — Manual Interaction Columns

```python
X = df[["x1", "x2"]].copy()
X["x1_x2"] = X["x1"] * X["x2"]   # continuous × continuous

# Continuous × categorical (dummy)
X["treat"] = (df["group"] == "treatment").astype(int)
X["x1_treat"] = X["x1"] * X["treat"]

X = sm.add_constant(X)
results = sm.OLS(df["y"], X).fit()
```

### Interpreting Interaction Coefficients

```python
results = smf.ols("y ~ x1 * x2", data=df).fit()

# Coefficient on x1:x2 is the marginal effect of x1 on y
# per unit increase in x2 (and vice versa)
print(results.params)
print(results.conf_int())

# For categorical interactions: the coefficient on x1:C(group)[T.treated]
# is the difference in the x1 slope between treated and reference groups
```

---

## Mixed Effects / Multilevel Models (MixedLM)

For hierarchical/nested data (e.g., students in schools, repeated measures):

```python
import statsmodels.formula.api as smf

# Random intercept model (varying intercept by group)
results = smf.mixedlm(
    "score ~ ses + school_type",
    data=df,
    groups=df["school_id"]
).fit()
print(results.summary())

# Random intercept + random slope
results = smf.mixedlm(
    "score ~ ses + school_type",
    data=df,
    groups=df["school_id"],
    re_formula="~ses"           # ses has a random slope by school
).fit()
```

Array API:
```python
import statsmodels.api as sm

model = sm.MixedLM(
    endog=y,
    exog=X,                     # fixed effects design matrix
    groups=groups,              # grouping variable
    exog_re=Z                   # random effects design matrix
)
results = model.fit()
```

Key parameters:
- `groups`: grouping variable defining the clusters (e.g., school ID, subject ID)
- `re_formula`: formula for random effects (default: random intercept only, `"~1"`)
- `vc_formula`: variance components formula for crossed random effects

Results attributes:
- `results.fe_params` — fixed effects coefficients
- `results.random_effects` — dict of random effects by group
- `results.cov_re` — random effects covariance matrix
- `results.bse_fe` — standard errors for fixed effects
- `results.bse_re` — standard errors for random effects variance components

Prediction:
```python
# Fixed effects prediction (population average)
pred = results.predict(new_data)

# Group-specific prediction: predict() takes no exog_re and returns the fixed-effects part only.
# In-sample, results.fittedvalues includes the predicted random effects; for new data add
# results.random_effects[group] times the random-effects design by hand.
```

Common use cases in education research:
- Students nested in classrooms nested in schools (two-level or three-level)
- Repeated test scores for the same student over time (longitudinal)
- Schools nested in districts with district-level predictors

Convergence: MixedLM uses gradient-based optimization by default (BFGS, L-BFGS-B, CG sequence). For convergence issues, try:
```python
results = model.fit(method=["powell"])     # Powell optimizer
results = model.fit(maxiter=200)           # Increase max iterations
results = model.fit(reml=False)            # ML instead of REML
```

---

## References and Further Reading

- statsmodels OLS documentation: https://www.statsmodels.org/stable/regression.html
- statsmodels WLS documentation: https://www.statsmodels.org/stable/regression.html#wls
- statsmodels GLS documentation: https://www.statsmodels.org/stable/regression.html#gls
- statsmodels RLM documentation: https://www.statsmodels.org/stable/rlm.html
- statsmodels QuantReg documentation: https://www.statsmodels.org/stable/generated/statsmodels.regression.quantile_regression.QuantReg.html
- patsy formula language: https://patsy.readthedocs.io/en/latest/formulas.html
- patsy splines: https://patsy.readthedocs.io/en/latest/spline-regression.html
- Greene, W.H. (2018). *Econometric Analysis*, 8th ed. — Chapters 4-9 (OLS, GLS, WLS, robust)
- Wooldridge, J.M. (2019). *Introductory Econometrics*, 7th ed. — Chapters 4, 8 (OLS inference, heteroskedasticity)
- Koenker, R. (2005). *Quantile Regression*. Cambridge University Press.
- Maronna, R.A., Martin, R.D., Yohai, V.J., & Salibian-Barrera, M. (2019). *Robust Statistics: Theory and Methods*, 2nd ed.
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall