Skip to content

DAAF-Contribution-Community/daaf / .claude/skills/plm

R panel data models with plm: within (FE), random (RE), between, first difference, Fama-MacBeth. Hausman test, panel unit root tests. Includes estimatr for robust estimation and lme4 for mixed effects. Use when execution language is R. Python equivalent: linearmodels. For high- dimensional FE use fixest; for cross-sectional OLS/GLM use r-stats.

FileKindSize
SKILL.mdskill md15.5 KB
references/diagnostics.mdmarkdown9.6 KB
references/gotchas.mdmarkdown11.2 KB
references/iv.mdmarkdown9.7 KB
references/mixed-effects.mdmarkdown11.1 KB
references/models.mdmarkdown12.2 KB
references/quickstart.mdmarkdown11.8 KB
references/robust.mdmarkdown9.3 KB

references/diagnostics.md

View on GitHub
# Panel Diagnostics

Reference for panel-specific diagnostic tests in plm. Covers the Hausman test
(FE vs RE), poolability tests, serial correlation tests, cross-sectional
dependence tests, and panel unit root tests.

## Contents

- [Hausman Test (FE vs RE)](#hausman-test-fe-vs-re)
- [Poolability Test (F-Test for FE)](#poolability-test-f-test-for-fe)
- [Serial Correlation Tests](#serial-correlation-tests)
- [Cross-Sectional Dependence Tests](#cross-sectional-dependence-tests)
- [Panel Unit Root Tests](#panel-unit-root-tests)
- [Diagnostic Workflow](#diagnostic-workflow)
- [References](#references)

---

## Hausman Test (FE vs RE)

The Hausman test compares FE and RE estimators. Under H0 (RE assumptions
hold), both are consistent but RE is efficient. Under H1, only FE is
consistent.

```r
library(plm)
data("Grunfeld", package = "plm")
pdf <- pdata.frame(Grunfeld, index = c("firm", "year"))

fit_fe <- plm(inv ~ value + capital, data = pdf, model = "within")
fit_re <- plm(inv ~ value + capital, data = pdf, model = "random")

# Hausman test
ht <- phtest(fit_fe, fit_re)
print(ht)
# Low p-value: reject RE, use FE
# High p-value: RE is consistent, prefer RE for efficiency
```

### Interpreting the Result

| p-value | Decision | Reasoning |
|---------|----------|-----------|
| < 0.05 | Use FE | RE assumption (effects uncorrelated with regressors) rejected |
| >= 0.05 | RE is acceptable | Cannot reject that effects are uncorrelated |

### Common Issues with phtest()

**Negative chi-squared statistic**: Can occur when the variance of the
difference (V_FE - V_RE) is not positive semi-definite. This is a
finite-sample problem.

```r
# If phtest() gives a warning about non-positive definite matrix,
# use the auxiliary regression version:
phtest(inv ~ value + capital, data = pdf, method = "aux")
```

**Two-way effects**: When using `effect = "twoways"`, both models must use
the same effect specification:

```r
fit_fe_2w <- plm(y ~ x, data = pdf, model = "within", effect = "twoways")
fit_re_2w <- plm(y ~ x, data = pdf, model = "random", effect = "twoways")
phtest(fit_fe_2w, fit_re_2w)
```

### Comparison with linearmodels

linearmodels (Python) does NOT have a built-in Hausman test. The plm
`phtest()` is one of plm's key advantages over linearmodels for model
selection.

---

## Poolability Test (F-Test for FE)

Tests whether entity fixed effects are jointly significant. H0: all entity
effects are zero (pooled OLS is appropriate). Rejection supports FE.

```r
fit_fe <- plm(inv ~ value + capital, data = pdf, model = "within")
fit_pool <- plm(inv ~ value + capital, data = pdf, model = "pooling")

# F-test for individual effects
pFtest(fit_fe, fit_pool)
# Low p-value: entity effects matter, reject pooled OLS
```

### Honda LM Test (Alternative)

```r
# Breusch-Pagan LM test for individual effects
plmtest(fit_pool, type = "bp")

# Honda test (one-sided, more powerful)
plmtest(fit_pool, type = "honda")

# Gourieroux, Holly & Monfort test (two-way effects; chi-bar-squared
# distribution). Only valid with effect = "twoways".
plmtest(fit_pool, type = "ghm", effect = "twoways")
```

| Test | H0 | When to Use |
|------|-----|-------------|
| `pFtest()` | All entity effects = 0 | Standard F-test (requires within model) |
| `plmtest(type = "bp")` | Var(entity effects) = 0 | Breusch-Pagan LM (uses pooling model) |
| `plmtest(type = "honda")` | Var(entity effects) = 0 | One-sided, more powerful than BP |

---

## Serial Correlation Tests

### Breusch-Godfrey / Wooldridge Test

Tests for serial correlation in panel model residuals.

```r
# Breusch-Godfrey test for serial correlation
pbgtest(fit_fe)
# Low p-value: serial correlation is present

# Specify lag order
pbgtest(fit_fe, order = 2)
```

### Wooldridge Tests

```r
# Wooldridge test for AR(1) serial correlation in FE panels
pwartest(fit_fe)

# Wooldridge first-difference-based test for serial correlation
# (the FD variant is a separate function, not pwartest; it takes the
# formula or an "fd" model -- NOT a within model)
pwfdtest(inv ~ value + capital, data = pdf)
```

Note: `plmtest()` is NOT a serial correlation test — it is the LM test for
unobserved individual/time effects (see the poolability section above).
`plmtest(effect = "time")` tests for time effects, not serial correlation.

### Durbin-Watson for Panel

```r
# Panel Durbin-Watson
pdwtest(fit_fe)
# Statistic near 2: no serial correlation
# Near 0 or 4: positive or negative serial correlation
```

---

## Cross-Sectional Dependence Tests

Tests whether residuals are correlated across entities within the same time
period. Important because cross-sectional dependence invalidates standard
clustered SEs.

### Pesaran CD Test

```r
# Pesaran CD test (works well with large N, any T)
pcdtest(fit_fe, test = "cd")
# Low p-value: cross-sectional dependence detected
# -> Consider Driscoll-Kraay SEs (vcovSCC)
```

### Scaled LM Test

```r
# Scaled LM test for cross-sectional dependence (Pesaran 2004)
pcdtest(fit_fe, test = "sclm")

# Bias-corrected scaled LM (Baltagi, Feng & Kao 2012)
pcdtest(fit_fe, test = "bcsclm")
```

Valid `test=` values are `"cd"`, `"sclm"`, `"bcsclm"`, `"lm"`, `"rho"`, and
`"absrho"` — there is no Friedman option in `pcdtest()`.

### Breusch-Pagan LM Test

```r
# LM test (requires T > N; not suitable for large panels)
pcdtest(fit_fe, test = "lm")
```

| Test | Requirements | Best For |
|------|-------------|----------|
| Pesaran CD (`"cd"`) | Any N, T | Large panels (default choice) |
| Scaled LM (`"sclm"`, `"bcsclm"`) | Moderate N relative to T | Large-T panels; bcsclm corrects small-sample bias |
| LM (Breusch-Pagan, `"lm"`) | T > N | Small panels only |

### When Cross-Sectional Dependence Is Detected

If `pcdtest()` rejects independence, use Driscoll-Kraay SEs:

```r
library(lmtest)
coeftest(fit_fe, vcov = vcovSCC(fit_fe))
```

---

## Panel Unit Root Tests

Tests whether panel variables are stationary. Important for avoiding
spurious regressions in panel data with long T.

### purtest() -- Multiple Panel Unit Root Tests

```r
# Im-Pesaran-Shin (IPS) test
# NOTE: on short-T panels (Grunfeld has T = 20) the default lag search overshoots
# and the Fortran back-end errors with "NA/NaN/Inf in foreign function call (arg 6)".
# Cap the lag search with pmax to keep the augmentation feasible.
purtest(pdf$inv, test = "ips", exo = "intercept", pmax = 4)

# Levin-Lin-Chu (LLC) test  (same pmax guard on short panels)
purtest(pdf$inv, test = "levinlin", exo = "intercept", pmax = 4)

# Maddala-Wu (Fisher-type) test  (same pmax guard on short panels)
purtest(pdf$inv, test = "madwu", exo = "intercept", pmax = 4)

# Hadri test (H0: stationarity, opposite null) -- no lag search, runs as-is
purtest(pdf$inv, test = "hadri", exo = "intercept")
```

> **Short-T lag caps.** `ips`, `levinlin`, and `madwu` augment each series with
> lagged differences; with a small number of time periods the default lag search
> (`lags = "AIC"`) can request more lags than the series supports and the
> underlying Fortran routine fails. Passing `pmax = 4` (or a small integer
> `lags = 2`) bounds the search and lets the test run. `hadri` performs no lag
> augmentation and is unaffected.

| Test | H0 | H1 | Best When |
|------|-----|-----|-----------|
| `"ips"` | All panels have unit root | Some panels are stationary | Heterogeneous panels |
| `"levinlin"` | Common unit root | All panels are stationary | Homogeneous panels |
| `"madwu"` | All panels have unit root | Some panels are stationary | Unbalanced panels |
| `"hadri"` | All panels are stationary | Some panels have unit root | Reverse null |

### cipstest() -- Pesaran CIPS Test

```r
# Cross-sectionally augmented IPS (CIPS) -- robust to cross-sect dependence
cipstest(pdf$inv, type = "drift")
```

The CIPS test is preferable when cross-sectional dependence is present
(which is common in macro and financial panels).

---

## Diagnostic Workflow

A recommended sequence of diagnostic tests for panel analysis:

```r
library(plm)
library(lmtest)

# 1. Set up panel
pdf <- pdata.frame(df, index = c("entity_id", "year"))

# 2. Estimate models
fit_pool <- plm(y ~ x1 + x2, data = pdf, model = "pooling")
fit_fe   <- plm(y ~ x1 + x2, data = pdf, model = "within")
fit_re   <- plm(y ~ x1 + x2, data = pdf, model = "random")

# 3. Poolability: are entity effects significant?
pFtest(fit_fe, fit_pool)
# If reject: entity effects matter, proceed to FE vs RE

# 4. Hausman: FE or RE?
phtest(fit_fe, fit_re)
# If reject: use FE
# If fail to reject: RE is acceptable

# 5. Serial correlation
pbgtest(fit_fe)
# If reject: serial correlation present
# -> Use clustered or Newey-West SEs

# 6. Cross-sectional dependence
pcdtest(fit_fe, test = "cd")
# If reject: cross-sectional dependence present
# -> Use Driscoll-Kraay SEs (vcovSCC)

# 7. Unit roots (if T is large)
# On short panels cap the lag search with pmax (see Panel Unit Root Tests above).
purtest(pdf$y, test = "ips", exo = "intercept", pmax = 4)
# If fail to reject: non-stationarity, consider first-differencing

# 8. Final model with appropriate SEs
# (based on diagnostic results)
coeftest(fit_fe, vcov = vcovHC(fit_fe, cluster = "group"))
```

---

## References

- Hausman, J.A. (1978). "Specification Tests in Econometrics." Econometrica,
  46(6), 1251-1271.
- Pesaran, M.H. (2004). "General Diagnostic Tests for Cross Section
  Dependence in Panels." Cambridge Working Papers in Economics, 0435.
- Im, K.S., Pesaran, M.H., & Shin, Y. (2003). "Testing for Unit Roots in
  Heterogeneous Panels." Journal of Econometrics, 115(1), 53-74.
- Breusch, T.S. & Pagan, A.R. (1980). "The Lagrange Multiplier Test and Its
  Applications to Model Specification in Econometrics." Review of Economic
  Studies, 47(1), 239-253.
- Wooldridge, J.M. (2010). Econometric Analysis of Cross Section and Panel
  Data. 2nd ed. MIT Press.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall