brycewang-stanford/Auto-Empirical-Research-Skills / skills/27-dariia-m-my_claude_skills/paper_verification
academic-paper-verify
VerifiedThoroughly verify all code, tables, figures, modeling decisions, and quantitative claims in an academic paper against its source R scripts and output files. Use this skill whenever you need to audit, replicate, or verify an academic research paper - including cross-checking LaTeX tables against R output, validating econometric modeling choices, ensuring sample sizes are consistent, building a verification manifest, and running automated replication tests. Trigger this skill for any mention of: paper verification, replication check, table audit, code-paper consistency, reproducing results, verifying estimates, checking coefficients, or any variant of "does the paper match the code."
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 11.4 KB |
| common-pitfalls.md | markdown | 7.0 KB |
| phase-details.md | markdown | 10.3 KB |
| replication-script-template.md | markdown | 8.5 KB |
replication-script-template.md
View on GitHub# Replication Script Template
Use this template when building `tests/verify_replication.R` in Phase 6.
Adapt it to the specific project structure.
---
## Full Template
```r
###############################################################################
# verify_replication.R
# Automated verification of paper claims against code output
# Generated by academic-paper-verify skill
###############################################################################
# --- Setup -------------------------------------------------------------------
library(jsonlite)
cat("=== Paper Replication Verification ===\n")
cat("Started:", format(Sys.time(), "%Y-%m-%d %H:%M:%S"), "\n\n")
# Load manifest
manifest_path <- "verification_manifest.json"
if (!file.exists(manifest_path)) {
stop("Manifest not found at: ", manifest_path,
"\nRun the manifest builder first.")
}
manifest <- fromJSON(manifest_path)
cat("Loaded manifest with", length(manifest$claims$id), "claims to verify.\n\n")
# --- Results collector -------------------------------------------------------
results <- data.frame(
claim_id = character(),
type = character(),
expected = character(),
actual = character(),
tolerance = numeric(),
status = character(), # PASS, FAIL, ERROR, SKIP
message = character(),
stringsAsFactors = FALSE
)
add_result <- function(claim_id, type, expected, actual,
tolerance = NA, status, message = "") {
results <<- rbind(results, data.frame(
claim_id = claim_id,
type = type,
expected = as.character(expected),
actual = as.character(actual),
tolerance = tolerance,
status = status,
message = message,
stringsAsFactors = FALSE
))
}
# --- Helper functions --------------------------------------------------------
check_numeric <- function(claim_id, actual, expected, tol = 1e-3, type = "value") {
if (is.null(actual) || is.na(actual)) {
add_result(claim_id, type, expected, "NA", tol, "ERROR",
"Could not extract actual value")
return(FALSE)
}
if (abs(as.numeric(actual) - as.numeric(expected)) <= tol) {
add_result(claim_id, type, expected, actual, tol, "PASS", "")
return(TRUE)
} else {
add_result(claim_id, type, expected, actual, tol, "FAIL",
paste0("Difference: ", abs(as.numeric(actual) - as.numeric(expected))))
return(FALSE)
}
}
check_exact <- function(claim_id, actual, expected, type = "count") {
if (is.null(actual) || is.na(actual)) {
add_result(claim_id, type, expected, "NA", 0, "ERROR",
"Could not extract actual value")
return(FALSE)
}
if (as.character(actual) == as.character(expected)) {
add_result(claim_id, type, expected, actual, 0, "PASS", "")
return(TRUE)
} else {
add_result(claim_id, type, expected, actual, 0, "FAIL",
paste0("Expected exactly ", expected, ", got ", actual))
return(FALSE)
}
}
safe_source <- function(script_path) {
cat(" Sourcing:", script_path, "...")
if (!file.exists(script_path)) {
cat(" NOT FOUND\n")
return(FALSE)
}
tryCatch({
source(script_path, local = FALSE)
cat(" OK\n")
return(TRUE)
}, error = function(e) {
cat(" ERROR:", conditionMessage(e), "\n")
return(FALSE)
})
}
safe_extract_coef <- function(model, varname) {
tryCatch({
coefs <- coef(model)
if (varname %in% names(coefs)) return(coefs[[varname]])
# Try partial matching for interaction terms, transformations
matches <- grep(varname, names(coefs), fixed = TRUE)
if (length(matches) == 1) return(coefs[matches])
return(NA)
}, error = function(e) return(NA))
}
safe_extract_se <- function(model, varname, vcov_type = NULL) {
tryCatch({
if (!is.null(vcov_type)) {
# Use sandwich / clubSandwich for robust/clustered SEs
if (requireNamespace("sandwich", quietly = TRUE)) {
vcov_mat <- sandwich::vcovHC(model, type = vcov_type)
se <- sqrt(diag(vcov_mat))
} else {
se <- sqrt(diag(vcov(model)))
}
} else {
se <- sqrt(diag(vcov(model)))
}
if (varname %in% names(se)) return(se[[varname]])
matches <- grep(varname, names(se), fixed = TRUE)
if (length(matches) == 1) return(se[matches])
return(NA)
}, error = function(e) return(NA))
}
# --- Verification blocks -----------------------------------------------------
# Each block corresponds to one script or one table.
# Adapt this section entirely to the specific project.
cat("\n--- Verifying Table 1 ---\n")
# Example block:
# if (safe_source("code/02_main_regression.R")) {
# # The script should have created a model object, e.g. model_main
# if (exists("model_main")) {
# check_numeric("T1_R1_C1", safe_extract_coef(model_main, "treatment"),
# manifest$claims$expected_value[manifest$claims$id == "T1_R1_C1"])
# check_exact("T1_N", nobs(model_main),
# manifest$claims$expected_value[manifest$claims$id == "T1_N"])
# } else {
# add_result("T1_ALL", "table", NA, NA, NA, "ERROR",
# "model_main not found after sourcing script")
# }
# } else {
# add_result("T1_ALL", "table", NA, NA, NA, "SKIP",
# "Script could not be sourced")
# }
# --- Generate reports --------------------------------------------------------
cat("\n\n=== Verification Complete ===\n")
cat("Total claims:", nrow(results), "\n")
cat("PASS: ", sum(results$status == "PASS"), "\n")
cat("FAIL: ", sum(results$status == "FAIL"), "\n")
cat("ERROR:", sum(results$status == "ERROR"), "\n")
cat("SKIP: ", sum(results$status == "SKIP"), "\n")
# Save structured results
write_json(results, "tests/replication_results.json", pretty = TRUE)
# Generate human-readable summary
summary_lines <- c(
"# Replication Verification Summary",
"",
paste("Generated:", format(Sys.time(), "%Y-%m-%d %H:%M:%S")),
"",
"## Results",
"",
paste("- Total claims checked:", nrow(results)),
paste("- PASS:", sum(results$status == "PASS")),
paste("- FAIL:", sum(results$status == "FAIL")),
paste("- ERROR:", sum(results$status == "ERROR")),
paste("- SKIP:", sum(results$status == "SKIP")),
""
)
if (sum(results$status == "FAIL") > 0) {
summary_lines <- c(summary_lines,
"## Failures",
""
)
fails <- results[results$status == "FAIL", ]
for (i in seq_len(nrow(fails))) {
summary_lines <- c(summary_lines,
paste0("### ", fails$claim_id[i]),
paste0("- Type: ", fails$type[i]),
paste0("- Expected: ", fails$expected[i]),
paste0("- Actual: ", fails$actual[i]),
paste0("- Message: ", fails$message[i]),
""
)
}
}
if (sum(results$status == "ERROR") > 0) {
summary_lines <- c(summary_lines,
"## Errors",
""
)
errs <- results[results$status == "ERROR", ]
for (i in seq_len(nrow(errs))) {
summary_lines <- c(summary_lines,
paste0("### ", errs$claim_id[i]),
paste0("- Message: ", errs$message[i]),
""
)
}
}
writeLines(summary_lines, "tests/replication_summary.md")
cat("\nResults saved to tests/replication_results.json")
cat("\nSummary saved to tests/replication_summary.md\n")
```
## Adaptation Notes
This template is a skeleton. When building the actual script for a specific project:
1. **Replace the example verification block** with real blocks for each table and script
in the project.
2. **Handle the script execution model** - some projects are designed so each script
is self-contained (reads data, runs analysis, saves output). Others assume a shared
global environment built up by running scripts in sequence. Adapt accordingly:
- Self-contained scripts: `safe_source()` each one independently
- Sequential scripts: source them in order into the global environment
3. **Match the SE extraction to the project** - if the project uses `fixest::feols`,
use `fixest::se()` and `fixest::coef()` instead of base R equivalents. If it uses
`lfe::felm`, use the appropriate extraction methods.
4. **Handle package-specific model objects** - different packages store results differently:
- `lm` / `glm`: `coef()`, `vcov()`, `nobs()`
- `fixest::feols`: `coef()`, `se()`, `fitstat()` for diagnostics
- `plm`: `coef()`, `vcov()` with appropriate method
- `ivreg`: `coef()`, `vcov()`, check first-stage via `summary()`
- `lfe::felm`: `$coefficients`, `$se`, `$N`
5. **Add tolerance overrides** per claim if needed. Some claims (like bootstrapped CIs)
need wider tolerance. Others (like sample sizes) need exact matching.
6. **Consider memory** - if the project processes large datasets, sourcing all scripts
sequentially may run out of memory. Add `gc()` calls between blocks and consider
running in a fresh R session for each major block.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |