Skip to content

brycewang-stanford/Auto-Empirical-Research-Skills / skills/27-dariia-m-my_claude_skills/paper_verification

academic-paper-verify

Verified

Thoroughly verify all code, tables, figures, modeling decisions, and quantitative claims in an academic paper against its source R scripts and output files. Use this skill whenever you need to audit, replicate, or verify an academic research paper - including cross-checking LaTeX tables against R output, validating econometric modeling choices, ensuring sample sizes are consistent, building a verification manifest, and running automated replication tests. Trigger this skill for any mention of: paper verification, replication check, table audit, code-paper consistency, reproducing results, verifying estimates, checking coefficients, or any variant of "does the paper match the code."

FileKindSize
SKILL.mdskill md11.4 KB
common-pitfalls.mdmarkdown7.0 KB
phase-details.mdmarkdown10.3 KB
replication-script-template.mdmarkdown8.5 KB

replication-script-template.md

View on GitHub
# Replication Script Template

Use this template when building `tests/verify_replication.R` in Phase 6.
Adapt it to the specific project structure.

---

## Full Template

```r
###############################################################################
# verify_replication.R
# Automated verification of paper claims against code output
# Generated by academic-paper-verify skill
###############################################################################

# --- Setup -------------------------------------------------------------------
library(jsonlite)

cat("=== Paper Replication Verification ===\n")
cat("Started:", format(Sys.time(), "%Y-%m-%d %H:%M:%S"), "\n\n")

# Load manifest
manifest_path <- "verification_manifest.json"
if (!file.exists(manifest_path)) {
  stop("Manifest not found at: ", manifest_path,
       "\nRun the manifest builder first.")
}
manifest <- fromJSON(manifest_path)
cat("Loaded manifest with", length(manifest$claims$id), "claims to verify.\n\n")

# --- Results collector -------------------------------------------------------
results <- data.frame(
  claim_id = character(),
  type = character(),
  expected = character(),
  actual = character(),
  tolerance = numeric(),
  status = character(),     # PASS, FAIL, ERROR, SKIP
  message = character(),
  stringsAsFactors = FALSE
)

add_result <- function(claim_id, type, expected, actual,
                       tolerance = NA, status, message = "") {
  results <<- rbind(results, data.frame(
    claim_id = claim_id,
    type = type,
    expected = as.character(expected),
    actual = as.character(actual),
    tolerance = tolerance,
    status = status,
    message = message,
    stringsAsFactors = FALSE
  ))
}

# --- Helper functions --------------------------------------------------------
check_numeric <- function(claim_id, actual, expected, tol = 1e-3, type = "value") {
  if (is.null(actual) || is.na(actual)) {
    add_result(claim_id, type, expected, "NA", tol, "ERROR",
               "Could not extract actual value")
    return(FALSE)
  }
  if (abs(as.numeric(actual) - as.numeric(expected)) <= tol) {
    add_result(claim_id, type, expected, actual, tol, "PASS", "")
    return(TRUE)
  } else {
    add_result(claim_id, type, expected, actual, tol, "FAIL",
               paste0("Difference: ", abs(as.numeric(actual) - as.numeric(expected))))
    return(FALSE)
  }
}

check_exact <- function(claim_id, actual, expected, type = "count") {
  if (is.null(actual) || is.na(actual)) {
    add_result(claim_id, type, expected, "NA", 0, "ERROR",
               "Could not extract actual value")
    return(FALSE)
  }
  if (as.character(actual) == as.character(expected)) {
    add_result(claim_id, type, expected, actual, 0, "PASS", "")
    return(TRUE)
  } else {
    add_result(claim_id, type, expected, actual, 0, "FAIL",
               paste0("Expected exactly ", expected, ", got ", actual))
    return(FALSE)
  }
}

safe_source <- function(script_path) {
  cat("  Sourcing:", script_path, "...")
  if (!file.exists(script_path)) {
    cat(" NOT FOUND\n")
    return(FALSE)
  }
  tryCatch({
    source(script_path, local = FALSE)
    cat(" OK\n")
    return(TRUE)
  }, error = function(e) {
    cat(" ERROR:", conditionMessage(e), "\n")
    return(FALSE)
  })
}

safe_extract_coef <- function(model, varname) {
  tryCatch({
    coefs <- coef(model)
    if (varname %in% names(coefs)) return(coefs[[varname]])
    # Try partial matching for interaction terms, transformations
    matches <- grep(varname, names(coefs), fixed = TRUE)
    if (length(matches) == 1) return(coefs[matches])
    return(NA)
  }, error = function(e) return(NA))
}

safe_extract_se <- function(model, varname, vcov_type = NULL) {
  tryCatch({
    if (!is.null(vcov_type)) {
      # Use sandwich / clubSandwich for robust/clustered SEs
      if (requireNamespace("sandwich", quietly = TRUE)) {
        vcov_mat <- sandwich::vcovHC(model, type = vcov_type)
        se <- sqrt(diag(vcov_mat))
      } else {
        se <- sqrt(diag(vcov(model)))
      }
    } else {
      se <- sqrt(diag(vcov(model)))
    }
    if (varname %in% names(se)) return(se[[varname]])
    matches <- grep(varname, names(se), fixed = TRUE)
    if (length(matches) == 1) return(se[matches])
    return(NA)
  }, error = function(e) return(NA))
}

# --- Verification blocks -----------------------------------------------------
# Each block corresponds to one script or one table.
# Adapt this section entirely to the specific project.

cat("\n--- Verifying Table 1 ---\n")
# Example block:
# if (safe_source("code/02_main_regression.R")) {
#   # The script should have created a model object, e.g. model_main
#   if (exists("model_main")) {
#     check_numeric("T1_R1_C1", safe_extract_coef(model_main, "treatment"),
#                   manifest$claims$expected_value[manifest$claims$id == "T1_R1_C1"])
#     check_exact("T1_N", nobs(model_main),
#                 manifest$claims$expected_value[manifest$claims$id == "T1_N"])
#   } else {
#     add_result("T1_ALL", "table", NA, NA, NA, "ERROR",
#                "model_main not found after sourcing script")
#   }
# } else {
#   add_result("T1_ALL", "table", NA, NA, NA, "SKIP",
#              "Script could not be sourced")
# }

# --- Generate reports --------------------------------------------------------
cat("\n\n=== Verification Complete ===\n")
cat("Total claims:", nrow(results), "\n")
cat("PASS: ", sum(results$status == "PASS"), "\n")
cat("FAIL: ", sum(results$status == "FAIL"), "\n")
cat("ERROR:", sum(results$status == "ERROR"), "\n")
cat("SKIP: ", sum(results$status == "SKIP"), "\n")

# Save structured results
write_json(results, "tests/replication_results.json", pretty = TRUE)

# Generate human-readable summary
summary_lines <- c(
  "# Replication Verification Summary",
  "",
  paste("Generated:", format(Sys.time(), "%Y-%m-%d %H:%M:%S")),
  "",
  "## Results",
  "",
  paste("- Total claims checked:", nrow(results)),
  paste("- PASS:", sum(results$status == "PASS")),
  paste("- FAIL:", sum(results$status == "FAIL")),
  paste("- ERROR:", sum(results$status == "ERROR")),
  paste("- SKIP:", sum(results$status == "SKIP")),
  ""
)

if (sum(results$status == "FAIL") > 0) {
  summary_lines <- c(summary_lines,
    "## Failures",
    ""
  )
  fails <- results[results$status == "FAIL", ]
  for (i in seq_len(nrow(fails))) {
    summary_lines <- c(summary_lines,
      paste0("### ", fails$claim_id[i]),
      paste0("- Type: ", fails$type[i]),
      paste0("- Expected: ", fails$expected[i]),
      paste0("- Actual: ", fails$actual[i]),
      paste0("- Message: ", fails$message[i]),
      ""
    )
  }
}

if (sum(results$status == "ERROR") > 0) {
  summary_lines <- c(summary_lines,
    "## Errors",
    ""
  )
  errs <- results[results$status == "ERROR", ]
  for (i in seq_len(nrow(errs))) {
    summary_lines <- c(summary_lines,
      paste0("### ", errs$claim_id[i]),
      paste0("- Message: ", errs$message[i]),
      ""
    )
  }
}

writeLines(summary_lines, "tests/replication_summary.md")
cat("\nResults saved to tests/replication_results.json")
cat("\nSummary saved to tests/replication_summary.md\n")
```

## Adaptation Notes

This template is a skeleton. When building the actual script for a specific project:

1. **Replace the example verification block** with real blocks for each table and script
   in the project.

2. **Handle the script execution model** - some projects are designed so each script
   is self-contained (reads data, runs analysis, saves output). Others assume a shared
   global environment built up by running scripts in sequence. Adapt accordingly:
   - Self-contained scripts: `safe_source()` each one independently
   - Sequential scripts: source them in order into the global environment

3. **Match the SE extraction to the project** - if the project uses `fixest::feols`,
   use `fixest::se()` and `fixest::coef()` instead of base R equivalents. If it uses
   `lfe::felm`, use the appropriate extraction methods.

4. **Handle package-specific model objects** - different packages store results differently:
   - `lm` / `glm`: `coef()`, `vcov()`, `nobs()`
   - `fixest::feols`: `coef()`, `se()`, `fitstat()` for diagnostics
   - `plm`: `coef()`, `vcov()` with appropriate method
   - `ivreg`: `coef()`, `vcov()`, check first-stage via `summary()`
   - `lfe::felm`: `$coefficients`, `$se`, `$N`

5. **Add tolerance overrides** per claim if needed. Some claims (like bootstrapped CIs)
   need wider tolerance. Others (like sample sizes) need exact matching.

6. **Consider memory** - if the project processes large datasets, sourcing all scripts
   sequentially may run out of memory. Add `gc()` calls between blocks and consider
   running in a fresh R session for each major block.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 196kUpdated Claude CodeCodexVerifiedInstall