brycewang-stanford/Auto-Empirical-Research-Skills / skills/00.1-Full-empirical-analysis-skill_Python
Full-empirical-analysis-skill
VerifiedClassical end-to-end empirical analysis workflow in the traditional Python econometric stack — pandas + numpy + scipy + statsmodels + linearmodels + pyfixest + rdrobust + econml + causalml + matplotlib/seaborn. **Defaults to economics empirical-paper style** (AER / QJE / AEJ) — every run produces a publication-ready output set with a multi-column regression table (M1→M6 progressive controls/FE) as the centerpiece, plus Table 1 (descriptives), mechanism / heterogeneity / robustness tables, and event-study + coefficient + trend figures. Covers the full 8-step pipeline an applied economist or quantitative social scientist runs on every paper — (1) data cleaning, (2) variable construction & transformation, (3) descriptive statistics & Table 1, (4) statistical diagnostic tests, (5) baseline empirical modeling, (6) robustness battery, (7) further analysis (mechanism, heterogeneity, mediation, moderation), (8) publication-ready tables & figures. **Also covers two parallel domain modes that share the same 8-step scaffolding** — **Mode A — Epidemiology / public health** (target-trial emulation via `zepid` / hand-rolled `pandas`, IPTW + g-formula + TMLE doubly-robust triplet via `zepid` / `econml` / `lifelines`, Mendelian randomization via `pymr` / `mrtool` (or `rpy2` → `MendelianRandomization`/`TwoSampleMR`), KM / AFT / Cox survival via `lifelines`, E-value sensitivity, principal stratification — STROBE / TRIPOD reporting), and **Mode B — ML causal inference** (DML via `econml.dml` / `doubleml`, S/T/X/R/DR meta-learners via `econml.metalearners` / `causalml`, causal forest via `econml.grf` / `causalml`, Dragonnet / TARNet / CEVAE neural causal via `causalml`, BCF via `pymc-bart` / `bcf-py`, matrix completion, CATE distribution + policy tree via `econml.policy` / `policytree-py`, off-policy evaluation, conformal causal via `mapie`, fairness audit via `fairlearn`, DAG learning via `causal-learn` / `cdt` / LLM-assisted). Prescribes which library to reach for at each step, shows
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 11.8 KB |
| SKILL.md | skill md | 112.2 KB |
| references/01-data-cleaning.md | markdown | 11.9 KB |
| references/02-data-transformation.md | markdown | 11.7 KB |
| references/03-descriptive-stats.md | markdown | 10.4 KB |
| references/04-statistical-tests.md | markdown | 11.8 KB |
| references/05-modeling.md | markdown | 17.5 KB |
| references/06-robustness.md | markdown | 12.4 KB |
| references/07-further-analysis.md | markdown | 12.6 KB |
| references/08-tables-plots.md | markdown | 16.0 KB |
references/04-statistical-tests.md
View on GitHub# Step 4 — Diagnostic Statistical Tests (Deep Reference)
Goal: before interpreting any coefficient, check the assumptions underlying its standard error and identification. The tests below cover the 5 classical concerns — normality, heteroskedasticity, autocorrelation, multicollinearity, and (for time series) stationarity — plus post-estimation endogeneity / overidentification / Hausman tests specific to IV and panel models.
Each test is presented with: **null → statistic → decision → what to do if rejected**.
## Contents
1. Normality of residuals
2. Heteroskedasticity
3. Autocorrelation (serial correlation)
4. Multicollinearity
5. Stationarity & unit roots (time series)
6. Cointegration (time series)
7. Endogeneity (Hausman / Wu–Hausman / Durbin)
8. Weak instruments (first-stage F, Cragg–Donald, Kleibergen–Paap)
9. Overidentification (Sargan–Hansen J)
10. Panel-specific tests (Hausman FE vs. RE, Breusch–Pagan LM, Chow)
11. Model specification (RESET, link test)
12. Outlier / leverage / influence diagnostics
---
## 1. Normality of residuals
**When it matters**: small-N inference with non-bootstrapped SEs, MLE models. For OLS with N > 200 the CLT usually makes normality a non-issue.
```python
from scipy import stats as sps
from statsmodels.stats.stattools import jarque_bera
resid = ols.resid
# Shapiro-Wilk — most powerful for small samples (N ≤ 5000)
sw_stat, sw_p = sps.shapiro(resid.sample(min(5000, len(resid)), random_state=0))
# Jarque-Bera — asymptotic, based on skew & kurtosis
jb_stat, jb_p, skew, kurt = jarque_bera(resid)
# D'Agostino–Pearson
da_stat, da_p = sps.normaltest(resid)
# Kolmogorov–Smirnov vs. Normal
ks_stat, ks_p = sps.kstest((resid-resid.mean())/resid.std(), "norm")
# Anderson–Darling
ad = sps.anderson(resid, dist="norm")
print(f"Shapiro p={sw_p:.3f}")
print(f"Jarque-Bera p={jb_p:.3f} skew={skew:.2f} kurt={kurt:.2f}")
print(f"D'Agostino p={da_p:.3f}")
print(f"KS p={ks_p:.3f}")
print(f"Anderson stat={ad.statistic:.2f} critical@5%={ad.critical_values[2]:.2f}")
```
**If rejected + small N**: use bootstrap CIs (`fit(cov_type="HC3").bootstrap(...)`) or robust M-estimation (`sm.RLM`).
---
## 2. Heteroskedasticity
**Null**: Var(ε | X) = σ² (homoskedastic). Rejection means SEs without `cov_type="HC3"` (or cluster) are wrong.
```python
from statsmodels.stats.diagnostic import (
het_breuschpagan, het_white, het_goldfeldquandt
)
# Breusch-Pagan (LM) — assumes linear form of heteroskedasticity
lm, lm_p, f, f_p = het_breuschpagan(ols.resid, ols.model.exog)
print(f"BP LM={lm:.2f} p={lm_p:.3f} F={f:.2f} p={f_p:.3f}")
# White (general) — includes squares and cross-products
W, W_p, f, f_p = het_white(ols.resid, ols.model.exog)
print(f"White stat={W:.2f} p={W_p:.3f}")
# Goldfeld-Quandt (subsample F-test) — for known structural break in variance
gq, gq_p, ord_ = het_goldfeldquandt(ols.model.endog, ols.model.exog)
print(f"GQ F={gq:.2f} p={gq_p:.3f}")
# Breusch-Pagan via statsmodels helper
import statsmodels.api as sm
print(sm.stats.diagnostic.het_breuschpagan(ols.resid, ols.model.exog))
```
**Rejected? The fix is in the SE family, not the point estimate**:
```python
# Robust SEs (HC3 is default preferred — White-style, Davidson & MacKinnon)
ols_hc3 = ols.get_robustcov_results(cov_type="HC3")
# Cluster SEs when error correlated within group
ols_cl = ols.get_robustcov_results(cov_type="cluster", groups=df["firm_id"])
# HAC (Newey-West) — heteroskedasticity + autocorrelation consistent (time series)
ols_hac = ols.get_robustcov_results(cov_type="HAC", maxlags=4)
```
---
## 3. Autocorrelation
**When it matters**: time-series, or panel with persistent shocks within unit. Clustered SEs by unit handle it in panel; time series needs HAC.
```python
from statsmodels.stats.stattools import durbin_watson
from statsmodels.stats.diagnostic import acorr_breusch_godfrey, acorr_ljungbox
# Durbin-Watson — detects AR(1); 2 = no autocorrelation, <1.5 or >2.5 red flag
dw = durbin_watson(ols.resid)
print(f"Durbin-Watson = {dw:.2f}")
# Breusch-Godfrey — detects AR(p) up to nlags
bg_stat, bg_p, f, f_p = acorr_breusch_godfrey(ols, nlags=4)
print(f"BG stat={bg_stat:.2f} p={bg_p:.3f}")
# Ljung-Box — portmanteau test of overall autocorrelation up to lag h
lb = acorr_ljungbox(ols.resid, lags=[4, 8, 12], return_df=True)
print(lb)
```
**Fix**: HAC / Newey-West SEs, or add lagged dependent variable, or model AR(1) residuals via GLS.
---
## 4. Multicollinearity
```python
from statsmodels.stats.outliers_influence import variance_inflation_factor
X = ols.model.exog # includes constant
vif = pd.DataFrame({
"var": ols.model.exog_names,
"VIF": [variance_inflation_factor(X, i) for i in range(X.shape[1])]
})
print(vif)
# Condition number (of scaled X)
Xs = (X - X.mean(0)) / X.std(0)
print(f"Condition number (scaled): {np.linalg.cond(Xs):.1f}")
```
**Rules of thumb**:
- `VIF < 5`: fine
- `5 ≤ VIF < 10`: watch
- `VIF ≥ 10`: serious; drop / combine / ridge
- Condition number > 30: warning; > 100: severe
**Fixes**: drop one of the collinear variables; use a composite index (PCA); ridge/lasso regularization; or accept that individual coefficients will be unstable but joint hypotheses are fine.
---
## 5. Stationarity (time series)
Two complementary tests — run both. Decision = intersection of both.
```python
from statsmodels.tsa.stattools import adfuller, kpss
# ADF — Null: unit root (non-stationary). Reject (p<0.05) = stationary.
adf_stat, adf_p, *_ = adfuller(y, autolag="AIC")
# KPSS — Null: stationary. FAIL to reject (p>0.05) = stationary.
kpss_stat, kpss_p, *_ = kpss(y, regression="c", nlags="auto")
print(f"ADF p={adf_p:.3f} KPSS p={kpss_p:.3f}")
```
**Decision table**:
| ADF | KPSS | Conclusion |
|-----|------|------------|
| reject | fail to reject | **stationary** (use levels) |
| fail to reject | reject | **non-stationary** (first-difference or cointegrate) |
| reject | reject | inconclusive (try structural-break test) |
| fail to reject | fail to reject | insufficient info (small sample) |
**Phillips–Perron** (robust to serial correlation; use `arch` package):
```python
from arch.unitroot import PhillipsPerron, DFGLS, ZivotAndrews
print(PhillipsPerron(y).summary())
print(DFGLS(y).summary()) # more power than ADF
print(ZivotAndrews(y).summary()) # allows one structural break
```
---
## 6. Cointegration (time series)
When levels are non-stationary but a linear combination is stationary → levels can be regressed on levels without spurious results.
```python
from statsmodels.tsa.stattools import coint
score, p, crit = coint(y1, y2) # Engle-Granger two-step
print(f"Cointegration p={p:.3f}")
# Johansen for vector / multi-variable
from statsmodels.tsa.vector_ar.vecm import coint_johansen
jres = coint_johansen(df[["y1","y2","y3"]].dropna(), det_order=0, k_ar_diff=1)
print(jres.lr1) # trace statistics
print(jres.cvt) # critical values
```
---
## 7. Endogeneity (Hausman / Wu–Hausman / Durbin)
**Null**: OLS is consistent (i.e., the suspect regressor is exogenous). Rejection → use IV.
```python
from linearmodels.iv import IV2SLS
iv = IV2SLS.from_formula("y ~ 1 + x_exog + [x_endog ~ z1 + z2]", data=df).fit()
print(iv.wu_hausman()) # H0: OLS consistent; rejecting ⇒ IV needed
print(iv.durbin()) # equivalent under conditional homoskedasticity
```
Output: if `p < 0.05`, OLS is inconsistent, IV is preferred (at the cost of efficiency).
---
## 8. Weak instruments
```python
# linearmodels exposes first-stage regressions directly
print(iv.first_stage)
# First-stage F for each endogenous var — target ≥ 104 (Lee et al. 2022), ≥10 rule-of-thumb
```
Cragg–Donald, Kleibergen–Paap (for multiple endogenous variables), and Stock–Yogo critical values are in `iv.diagnostics`:
```python
print(iv.diagnostics)
# Anderson-Rubin confidence set (weak-instrument robust)
print(iv.anderson_rubin)
```
---
## 9. Overidentification (Sargan–Hansen J)
**Null**: all instruments are valid (uncorrelated with error). Requires more instruments than endogenous regressors.
```python
print(iv.sargan) # homoskedastic form
print(iv.basmann)
# For heteroskedasticity-robust version, use IVGMM:
from linearmodels.iv import IVGMM
iv_gmm = IVGMM.from_formula(...).fit()
print(iv_gmm.j_stat) # Hansen J
```
**Interpretation**: if `p > 0.1`, cannot reject instrument validity. If `p < 0.05`, at least one instrument is invalid — the exclusion restriction probably fails.
---
## 10. Panel-specific tests
```python
from linearmodels.panel import PanelOLS, RandomEffects, compare
# FE
fe = PanelOLS.from_formula("y ~ x1 + x2 + EntityEffects + TimeEffects", df.set_index(["id","year"])).fit()
# RE
re = RandomEffects.from_formula("y ~ x1 + x2", df.set_index(["id","year"])).fit()
# Hausman (FE vs. RE) — H0: RE consistent (use RE); reject ⇒ use FE
b_diff = fe.params - re.params
v_diff = fe.cov - re.cov
hausman = b_diff @ np.linalg.pinv(v_diff) @ b_diff
p = 1 - sps.chi2.cdf(hausman, df=len(b_diff))
print(f"Hausman = {hausman:.2f}, p = {p:.3f}")
# Breusch-Pagan LM test — H0: RE not needed (pooled OLS fine)
# Test whether variance of unit effects is zero
# See linearmodels.panel.IVSystemGMM or hand-roll from residuals
# Chow test — structural break (pre/post) with known break
import statsmodels.api as sm
def chow_test(y, X, split_idx):
r = sm.OLS(y, X).fit()
r1 = sm.OLS(y[:split_idx], X[:split_idx]).fit()
r2 = sm.OLS(y[split_idx:], X[split_idx:]).fit()
SSR_p, SSR_1, SSR_2 = r.ssr, r1.ssr, r2.ssr
k, n = X.shape[1], len(y)
F = ((SSR_p - SSR_1 - SSR_2)/k) / ((SSR_1+SSR_2)/(n-2*k))
p = 1 - sps.f.cdf(F, k, n-2*k)
return F, p
```
---
## 11. Model specification
```python
from statsmodels.stats.diagnostic import linear_reset
reset = linear_reset(ols, power=[2,3], test_type="fitted")
print(reset) # H0: no functional-form misspecification; reject ⇒ add polynomial / log
```
Stata-style `linktest` (manual):
```python
# Regress y on yhat and yhat^2; if yhat^2 significant, specification is wrong
yhat = ols.fittedvalues
aux = sm.OLS(df["y"], sm.add_constant(np.column_stack([yhat, yhat**2]))).fit()
print(aux.summary())
```
---
## 12. Outlier / leverage / influence
```python
from statsmodels.stats.outliers_influence import OLSInfluence
inf = OLSInfluence(ols)
cooks_d = inf.cooks_distance[0]
leverage = inf.hat_matrix_diag
std_resid = inf.resid_studentized_internal
dffits = inf.dffits[0]
# Flags
high_cooks = cooks_d > 4/len(df) # rule of thumb
high_leverage = leverage > 2*X.shape[1]/len(df)
high_resid = np.abs(std_resid) > 3
high_dffits = np.abs(dffits) > 2*np.sqrt(X.shape[1]/len(df))
print(f"Cook's D flagged: {high_cooks.sum()}")
print(f"High leverage: {high_leverage.sum()}")
print(f"|stud resid| > 3: {high_resid.sum()}")
print(f"|DFFITS| > threshold: {high_dffits.sum()}")
# Plot
fig, ax = plt.subplots(figsize=(7,5))
ax.scatter(leverage, std_resid, s=30*cooks_d/cooks_d.max()+5, alpha=0.5)
ax.axhline( 3, ls="--", color="red"); ax.axhline(-3, ls="--", color="red")
ax.set_xlabel("Leverage"); ax.set_ylabel("Studentized residual")
plt.savefig("fig_influence.pdf")
```
Refit without the top-K influential observations and report whether the main coefficient survives.
---
## A standard Step-4 deliverable
Before moving to Step 5, produce a **diagnostics report**:
```text
=== Diagnostics: baseline spec log_wage ~ training + age + edu + tenure ===
[Normality] JB p=0.12 SW p=0.08 → OK (N=5432, CLT dominates anyway)
[Hetero] BP p=0.003 White p=0.002 → USE HC3 / cluster SEs
[Autocorr] DW=1.6 BG p=0.01 → CLUSTER by worker_id
[Multicoll] max VIF=3.4 condN=14 → OK
[Stationarity] ADF p=0.04 KPSS p=0.07 → stationary in levels
[Influence] 2 observations flagged (Cook>4/N); rerun without → main coef stable
```
Store this report alongside the regression so later readers can reproduce the choice of SE family.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |