Skip to content

brycewang-stanford/Auto-Empirical-Research-Skills / skills/11-James-Traina-compound-science/skills/reproducible-pipelines

reproducible-pipelines

Verified

This skill covers reproducible research pipelines and replication packages. Use when the user is setting up a research project directory structure, configuring workflow managers (Make, Snakemake, DVC), managing computational environments, preparing replication packages for journal submission, or debugging reproducibility failures. Triggers on "reproducible", "replication package", "Makefile", "Snakemake", "DVC", "pipeline", "workflow manager", "data versioning", "conda environment", "Docker", "seed management", "AEA data editor", "replication", "project structure", or "submission checklist".

references/environment-and-seeds.md

View on GitHub
# Environment Management, Seeds, and Results Caching

## Environment Management

### Conda (Recommended for Python/R Mixed Projects)

```yaml
# environment.yml
name: my-project
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.11.7        # Pin exact version
  - numpy=1.26.4
  - pandas=2.2.0
  - scipy=1.12.0
  - statsmodels=0.14.1
  - scikit-learn=1.4.0
  - matplotlib=3.8.3
  - pip:
    - linearmodels==6.0
    - pyblp==1.1.0
    - rdrobust==1.1.1
```

```bash
# Create environment
conda env create -f environment.yml

# Export exact versions (for reproducibility)
conda env export --no-builds > environment.lock.yml

# Recreate exact environment
conda env create -f environment.lock.yml
```

**Best practices:**
- Pin **exact** versions in the lock file (not `>=` or `~=`)
- Use `conda-forge` channel for most scientific packages
- Test on a clean machine (or CI) to verify the environment file is complete
- `environment.yml` is for human editing; `environment.lock.yml` is the machine-exact specification

### pip + venv (Lighter Weight)

```bash
# Create virtual environment
python -m venv .venv
source .venv/bin/activate

# Install and freeze
pip install numpy==1.26.4 pandas==2.2.0 statsmodels==0.14.1
pip freeze > requirements.txt

# Recreate
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

### renv (For R Projects)

```r
# Initialize renv in project
renv::init()

# Install packages (recorded in renv.lock)
install.packages("fixest")
install.packages("did")

# Snapshot current state
renv::snapshot()

# Collaborator restores exact environment
renv::restore()
```

### Docker (Maximum Reproducibility)

Use Docker when the computational environment itself must be exactly reproducible (OS-level dependencies, system libraries).

```dockerfile
# Dockerfile
FROM continuumio/miniconda3:24.1.2-0

WORKDIR /project

# Copy environment specification first (for caching)
COPY environment.yml .
RUN conda env create -f environment.yml

# Activate environment in subsequent commands
SHELL ["conda", "run", "-n", "my-project", "/bin/bash", "-c"]

# Copy project files
COPY . .

# Default: run the full pipeline
CMD ["make", "all"]
```

```bash
# Build and run
docker build -t my-project .
docker run -v $(pwd)/output:/project/output my-project

# Or run interactively
docker run -it -v $(pwd):/project my-project bash
```

## Random Seed Management

Every stochastic operation must be seeded and logged.

```python
# config.py — Central seed management
import numpy as np
import random
import os

MASTER_SEED = 20240215  # Date-based seeds are easy to document

def set_all_seeds(seed=MASTER_SEED):
    """Set seeds for all random number generators."""
    np.random.seed(seed)
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)

    # If using PyTorch
    try:
        import torch
        torch.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
        torch.backends.cudnn.deterministic = True
    except ImportError:
        pass

def get_rng(seed=None):
    """Create an independent RNG for a specific task.

    Use this instead of global np.random to avoid seed contamination
    between different parts of the pipeline.
    """
    if seed is None:
        seed = MASTER_SEED
    return np.random.default_rng(seed)
```

```python
# In estimation code
from config import get_rng, MASTER_SEED

# Each bootstrap/simulation gets a deterministic, independent seed
rng = get_rng(MASTER_SEED + 1)  # +1 for bootstrap, +2 for simulation, etc.

bootstrap_estimates = []
for b in range(n_bootstrap):
    idx = rng.choice(n, size=n, replace=True)
    # ... estimate on bootstrap sample
```

**Rules:**
1. **One master seed** defined in a config file, documented in README
2. **Derived seeds** for different pipeline stages (bootstrap, simulation, sample splits)
3. **Use `np.random.default_rng()`** not `np.random.seed()` — the new API creates independent generators that don't interfere with each other
4. **Log the seed** in output metadata: `results['seed'] = MASTER_SEED`
5. **Test reproducibility**: run the pipeline twice and diff the outputs

## Results Caching

Avoid re-running expensive computations during development.

```python
import joblib
from pathlib import Path

def cached_computation(func, cache_key, cache_dir="output/cache", **kwargs):
    """Cache expensive computations with dependency-aware keys."""
    cache_path = Path(cache_dir) / f"{cache_key}.joblib"
    cache_path.parent.mkdir(parents=True, exist_ok=True)

    if cache_path.exists():
        return joblib.load(cache_path)

    result = func(**kwargs)
    joblib.dump(result, cache_path)
    return result

# Usage
estimates = cached_computation(
    run_estimation,
    cache_key="main_2sls_v3",  # version the cache key when code changes
    data=df, instruments=['z1', 'z2']
)
```

**Important:** Caching is for development speed only. The final replication run must execute everything from scratch (`make clean && make all`).
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 323kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall