brycewang-stanford/Auto-Empirical-Research-Skills / plugins/empirical-analysis-python/skills/pipeline
Full-empirical-analysis-skill
VerifiedClassical end-to-end empirical analysis workflow in the traditional Python econometric stack — pandas + numpy + scipy + statsmodels + linearmodels + pyfixest + rdrobust + econml + causalml + matplotlib/seaborn. **Defaults to economics empirical-paper style** (AER / QJE / AEJ) — every run produces a publication-ready output set with a multi-column regression table (M1→M6 progressive controls/FE) as the centerpiece, plus Table 1 (descriptives), mechanism / heterogeneity / robustness tables, and event-study + coefficient + trend figures. Covers the full 8-step pipeline an applied economist or quantitative social scientist runs on every paper — (1) data cleaning, (2) variable construction & transformation, (3) descriptive statistics & Table 1, (4) statistical diagnostic tests, (5) baseline empirical modeling, (6) robustness battery, (7) further analysis (mechanism, heterogeneity, mediation, moderation), (8) publication-ready tables & figures. **Also covers two parallel domain modes that share the same 8-step scaffolding** — **Mode A — Epidemiology / public health** (target-trial emulation via `zepid` / hand-rolled `pandas`, IPTW + g-formula + TMLE doubly-robust triplet via `zepid` / `econml` / `lifelines`, Mendelian randomization via `pymr` / `mrtool` (or `rpy2` → `MendelianRandomization`/`TwoSampleMR`), KM / AFT / Cox survival via `lifelines`, E-value sensitivity, principal stratification — STROBE / TRIPOD reporting), and **Mode B — ML causal inference** (DML via `econml.dml` / `doubleml`, S/T/X/R/DR meta-learners via `econml.metalearners` / `causalml`, causal forest via `econml.grf` / `causalml`, Dragonnet / TARNet / CEVAE neural causal via `causalml`, BCF via `pymc-bart` / `bcf-py`, matrix completion, CATE distribution + policy tree via `econml.policy` / `policytree-py`, off-policy evaluation, conformal causal via `mapie`, fairness audit via `fairlearn`, DAG learning via `causal-learn` / `cdt` / LLM-assisted). Prescribes which library to reach for at each step, shows
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 112.2 KB |
| references/01-data-cleaning.md | markdown | 11.9 KB |
| references/02-data-transformation.md | markdown | 11.7 KB |
| references/03-descriptive-stats.md | markdown | 10.4 KB |
| references/04-statistical-tests.md | markdown | 11.8 KB |
| references/05-modeling.md | markdown | 17.5 KB |
| references/06-robustness.md | markdown | 12.4 KB |
| references/07-further-analysis.md | markdown | 12.6 KB |
| references/08-tables-plots.md | markdown | 16.0 KB |
references/06-robustness.md
View on GitHub# Step 6 — Robustness Battery (Deep Reference)
Goal: a headline estimate isn't credible until you've shown it survives reasonable variations. A thorough robustness appendix is the difference between a reviewer-approved paper and a desk-rejected one. This file is the standard battery.
## Contents
1. Alternative specifications (progressive controls, M1 → M6)
2. Alternative standard-error families (cluster level, two-way, wild bootstrap)
3. Subsample splits (pre-defined groups)
4. Alternative outcome / treatment definitions
5. Alternative sample restrictions (winsorization, trimming)
6. Placebo — fake timing
7. Placebo — fake treatment / permutation inference
8. Specification curve (all combinations of controls)
9. Oster (2019) δ* — selection-on-unobservables bound
10. Leave-one-out (LOO) sensitivity
11. Influence / leverage re-checks
12. Sensitivity analysis (Rosenbaum bounds, e-value)
---
## 1. Alternative specifications — progressive controls
Run a ladder of specifications; display coefficients side-by-side. Stability across M1 → M6 supports identification.
```python
import pyfixest as pf
specs = [
# (label, formula)
("M1", "log_wage ~ training"),
("M2", "log_wage ~ training + age + edu"),
("M3", "log_wage ~ training + age + edu + tenure"),
("M4", "log_wage ~ training + age + edu + tenure | worker_id"),
("M5", "log_wage ~ training + age + edu + tenure | worker_id + year"),
("M6", "log_wage ~ training + age + edu + tenure | worker_id + year + industry^year"),
]
results = {name: pf.feols(f, data=df, vcov={"CRV1":"worker_id"}) for name, f in specs}
pf.etable(list(results.values())) # side-by-side
```
**Report**: the main coefficient, its SE, N, and R² (or within-R²) across all specs. A dramatic change from M3 → M5 suggests omitted variable bias; stability is reassuring.
---
## 2. Alternative standard-error families
Cluster level affects inference more than most authors acknowledge. Report the main result at 3–4 cluster levels:
```python
for cl in ["worker_id", "firm_id", "industry", "state"]:
r = pf.feols("log_wage ~ training | worker_id + year",
data=df, vcov={"CRV1": cl})
b = r.coef()["training"]; se = r.se()["training"]
print(f"cluster={cl:10s} β={b:.4f} SE={se:.4f} t={b/se:.2f}")
```
Two-way clustering (Cameron–Gelbach–Miller):
```python
r2 = pf.feols("log_wage ~ training | worker_id + year",
data=df, vcov={"CRV1":"worker_id+firm_id"})
```
Wild cluster bootstrap (essential when # clusters < 50; cluster robust SEs under-cover):
```python
r = pf.feols("log_wage ~ training | worker_id + year", data=df, vcov={"CRV1":"state"})
r.wildboottest(param="training", B=9999, seed=42)
```
Newey–West / HAC (time-series):
```python
import statsmodels.formula.api as smf
ols_hac = smf.ols("y ~ x", data=ts).fit(cov_type="HAC", cov_kwds={"maxlags": 4})
```
---
## 3. Subsample splits
```python
splits = {
"female=0": df["female"]==0, "female=1": df["female"]==1,
"young" : df["age"] < 40, "old" : df["age"] >= 40,
"manuf" : df["industry"]=="manuf",
"service" : df["industry"]=="service",
}
for name, mask in splits.items():
r = pf.feols("log_wage ~ training | worker_id + year",
data=df[mask], vcov={"CRV1":"worker_id"})
print(f"{name:10s} β={r.coef()['training']:.4f} SE={r.se()['training']:.4f} N={int(r._N)}")
```
For heterogeneity *testing* (not just estimation), do a full-sample interaction — see `07-further-analysis.md`.
---
## 4. Alternative outcome / treatment definitions
```python
# Alternative outcome transformations
for y in ["log_wage", "ihs_wage", "wage_w1", "wage_real_log"]:
r = pf.feols(f"{y} ~ training | worker_id + year", data=df)
print(y, r.coef()["training"], r.se()["training"])
# Alternative treatment definitions
for t in ["training_ever", "training_hours", "training_completed", "training_intense"]:
r = pf.feols(f"log_wage ~ {t} | worker_id + year", data=df)
print(t, r.coef()[t], r.se()[t])
```
If the main effect only shows up under one specific definition → suspicious.
---
## 5. Alternative sample restrictions
```python
# Winsorization sensitivity
for lo, hi in [(0.00,0.00), (0.01,0.01), (0.05,0.05)]:
y = winsorize(df["log_wage"], limits=[lo, hi]).data
d2 = df.assign(y=y)
r = pf.feols("y ~ training | worker_id + year", data=d2)
print(f"winsorize {lo*100:.0f}%/{(1-hi)*100:.0f}%: β={r.coef()['training']:.4f}")
# Trim sensitivity
for trim in [0.01, 0.05]:
lo = df["log_wage"].quantile(trim); hi = df["log_wage"].quantile(1-trim)
mask = (df["log_wage"] >= lo) & (df["log_wage"] <= hi)
r = pf.feols("log_wage ~ training | worker_id + year", data=df[mask])
print(f"trim {trim*100:.0f}%: β={r.coef()['training']:.4f}")
```
---
## 6. Placebo — fake timing
Shift the "treatment" backward in time to a period before the real policy. The placebo coefficient should be ~0.
```python
# Fake treatment 3 years earlier
df["fake_first_treat"] = df["first_treat_year"] - 3
df["fake_post"] = (df["year"] >= df["fake_first_treat"]).astype(int)
# Drop the true post-period so the placebo isn't contaminated by the real effect
df_placebo = df[df["year"] < df["first_treat_year"]].copy()
r_placebo = pf.feols("log_wage ~ fake_post | worker_id + year",
data=df_placebo, vcov={"CRV1":"worker_id"})
r_placebo.summary() # expect insignificant coefficient
```
For event studies, drop the real post period and re-estimate pre-period coefficients: all should be indistinguishable from zero.
---
## 7. Placebo — fake treatment / permutation inference
Shuffle treatment across units; re-estimate many times; compare observed coefficient to permutation distribution.
```python
obs_coef = pf.feols("log_wage ~ training | worker_id + year",
data=df, vcov={"CRV1":"worker_id"}).coef()["training"]
# Unit-level permutation (preserves within-unit trajectory; randomizes who gets treated)
unit_treat = df.groupby("worker_id")["training"].max() # whether unit ever treated
perm_coefs = []
for s in range(1000):
new_order = unit_treat.sample(frac=1, random_state=s).values
mapping = dict(zip(unit_treat.index, new_order))
df["training_perm"] = df["worker_id"].map(mapping) * df["training"].groupby(df["worker_id"]) \
.transform(lambda g: (g>0).astype(int))
r = pf.feols("log_wage ~ training_perm | worker_id + year", data=df)
perm_coefs.append(r.coef()["training_perm"])
p_perm = np.mean(np.abs(perm_coefs) >= abs(obs_coef))
print(f"Permutation p = {p_perm:.3f}")
# Plot the distribution
plt.hist(perm_coefs, bins=50, alpha=0.7)
plt.axvline(obs_coef, color="red", lw=2, label=f"observed = {obs_coef:.3f}")
plt.legend(); plt.xlabel("Permuted coefficient")
plt.savefig("fig_permutation.pdf")
```
---
## 8. Specification curve
Fit the model across every combination of controls / fixed effects / outcome / treatment definitions, and plot the distribution of estimates.
```python
import itertools
outcomes = ["log_wage", "wage_w1"]
treatments = ["training", "training_ever"]
controls_opt = [[], ["age"], ["age","edu"], ["age","edu","tenure"]]
fe_opt = ["", "worker_id", "worker_id + year", "worker_id + year + industry^year"]
rows = []
for y, t, ctrls, fe in itertools.product(outcomes, treatments, controls_opt, fe_opt):
rhs = [t] + ctrls
fml = f"{y} ~ {'+'.join(rhs)}"
if fe: fml += f" | {fe}"
try:
r = pf.feols(fml, data=df, vcov={"CRV1":"worker_id"})
rows.append({"y":y, "t":t, "ctrls": ",".join(ctrls) or "none",
"fe": fe or "none",
"coef": r.coef()[t], "se": r.se()[t]})
except Exception as e:
pass
sc = pd.DataFrame(rows).sort_values("coef").reset_index(drop=True)
sc["lb"] = sc["coef"] - 1.96*sc["se"]; sc["ub"] = sc["coef"] + 1.96*sc["se"]
# Plot — the classic specification curve
fig, ax = plt.subplots(figsize=(10,4))
ax.scatter(range(len(sc)), sc["coef"], s=10)
ax.vlines(range(len(sc)), sc["lb"], sc["ub"], color="gray", alpha=0.3)
ax.axhline(0, color="k", ls="--")
ax.set_xlabel("Specification (ordered by coefficient)")
ax.set_ylabel("Coefficient on treatment")
plt.savefig("fig_spec_curve.pdf")
print(f"Median coef: {sc.coef.median():.3f} # positive sig: {((sc.coef-1.96*sc.se)>0).sum()}/{len(sc)}")
```
---
## 9. Oster (2019) δ* — selection on unobservables
Given R² in a "short" and "long" regression, derive the bias-adjusted coefficient and how strong selection on unobservables would need to be (relative to observables) to null the effect.
```python
def oster_delta(beta_short, beta_long, R2_short, R2_long, R_max=None):
"""
Oster (2019) delta*, assuming linear selection.
R_max = hypothetical R^2 if ALL confounders observed (commonly 1.3*R2_long, capped at 1).
Returns delta* (if >1, selection on unobs must exceed selection on obs to eliminate the effect).
"""
if R_max is None: R_max = min(1.3*R2_long, 1.0)
num = (beta_long) * (R2_long - R2_short)
denom = (beta_short - beta_long) * (R_max - R2_long)
return num / denom if denom != 0 else np.nan
r_s = pf.feols("log_wage ~ training", data=df).fit()
r_l = pf.feols("log_wage ~ training + age + edu + tenure + female | worker_id + year", data=df)
print(f"δ* = {oster_delta(r_s.coef()['training'], r_l.coef()['training'], r_s._R2, r_l._R2):.2f}")
# |δ*| > 1 = basic robustness; |δ*| > 2 = strong robustness to unobservables.
```
---
## 10. Leave-one-out sensitivity
```python
units = df["worker_id"].unique()
loo_coefs = []
for u in np.random.choice(units, size=min(500, len(units)), replace=False):
r = pf.feols("log_wage ~ training | worker_id + year",
data=df[df.worker_id != u])
loo_coefs.append(r.coef()["training"])
plt.hist(loo_coefs, bins=50)
plt.axvline(obs_coef, color="red", lw=2)
plt.title("Leave-one-unit-out coefficient distribution")
plt.savefig("fig_loo.pdf")
```
For panel papers, also try leaving out entire years, entire cohorts, or entire geographic regions.
---
## 11. Influence / leverage re-checks
```python
# See 04-statistical-tests.md §12 for the full OLSInfluence recipe.
# For the robustness section: drop the top 1% most influential obs and rerun.
from statsmodels.stats.outliers_influence import OLSInfluence
ols_full = smf.ols("log_wage ~ training + age + edu + tenure", data=df).fit()
inf = OLSInfluence(ols_full)
cd = inf.cooks_distance[0]
mask = cd < np.quantile(cd, 0.99)
r_drop = pf.feols("log_wage ~ training | worker_id + year",
data=df.loc[mask], vcov={"CRV1":"worker_id"})
print("Main:", obs_coef, "Drop top 1% Cook's D:", r_drop.coef()["training"])
```
---
## 12. Sensitivity — Rosenbaum bounds / e-value
### Rosenbaum bounds (for matched-pair studies)
Asks: how strong would an unobserved confounder have to be (in odds ratio) to nullify the significance of the effect?
```python
# Use pysensemakr or hand-roll via matched-pair signed-rank test over Gamma in [1, 1.5, 2, 3]
# See Rosenbaum (2002) for formula.
```
### E-value (VanderWeele & Ding 2017)
```python
def evalue(rr):
"""E-value for a risk ratio (point estimate only; extend to CI lb for conservative)."""
if rr < 1: rr = 1/rr
return rr + np.sqrt(rr*(rr-1))
# Convert OLS coefficient to approximate RR (when outcome is not too rare)
# For continuous outcomes, convert via Cohen's d or another mapping; see literature.
```
---
## What a strong robustness appendix contains
A paper that will pass review has, at minimum:
1. **Progressive specs table** (M1–M6) with the main coefficient across all columns
2. **Cluster-level sensitivity** table (main coef at 3–4 cluster levels, + wild bootstrap if few clusters)
3. **Placebo (fake timing)** — event study on pre-period; placebo coef should be ~0
4. **Placebo (permutation)** — histogram with observed coef vs. null distribution
5. **Specification curve** — main coef across dozens of valid specs, plotted
6. **Oster δ\*** — reported for short→long spec
7. **Subsample splits** — main coef across 4–6 pre-defined subsamples
8. **Alternative outcome / treatment definitions** — main coef at 2–3 alternatives each
9. For DID: Goodman-Bacon weights + HonestDID sensitivity
10. For IV: weak-IV robust AR CI, overid test, Conley SEs if geographic
11. For RD: bandwidth sensitivity (0.5h / h / 2h), density test, covariate smoothness
12. For PSM: SMD table before/after, common-support trimmed result, entropy-balance version
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |