Skip to content

brycewang-stanford/Auto-Empirical-Research-Skills / skills/35-bahayonghang-academic-writing-skills/skills/paper-audit

paper-audit

Verified

Deep-review-first audit for Chinese and English academic papers across LaTeX, Typst, and PDF formats. Use whenever the user wants reviewer-style paper critique, pre-submission readiness checks, pass/fail gate decisions, structured revision roadmaps, or re-audits of revised manuscripts. Trigger even if the user only says "review my paper", "check if this is ready to submit", "audit this PDF", "simulate peer review", "find the biggest problems in this manuscript", or "re-check whether I fixed the review issues". Do not use for direct source editing or compilation-heavy repair; route those to the format-specific writing skills instead.

FileKindSize
SKILL.mdskill md14.3 KB
agents/claims_evidence_reviewer_agent.mdmarkdown354 B
agents/committee_editor_agent.mdmarkdown1.9 KB
agents/committee_literature_agent.mdmarkdown1.9 KB
agents/committee_logic_agent.mdmarkdown1.5 KB
agents/committee_methodology_agent.mdmarkdown1.8 KB
agents/committee_theory_agent.mdmarkdown1.9 KB
agents/critical_reviewer_agent.mdmarkdown8.4 KB
agents/domain_reviewer_agent.mdmarkdown7.1 KB
agents/editor_in_chief_agent.mdmarkdown5.2 KB
agents/evaluation_fairness_reviewer_agent.mdmarkdown371 B
agents/literature_reviewer_agent.mdmarkdown3.6 KB
agents/methodology_reviewer_agent.mdmarkdown7.2 KB
agents/notation_consistency_reviewer_agent.mdmarkdown398 B
agents/prior_art_reviewer_agent.mdmarkdown2.9 KB
agents/section_reviewer_agent.mdmarkdown365 B
agents/self_consistency_reviewer_agent.mdmarkdown420 B
agents/synthesis_agent.mdmarkdown951 B
evals/evals.jsonconfig12.0 KB
evals/fixtures/deep_review_fixture.texother699 B
evals/fixtures/gate_ieee_fixture.texother617 B
evals/fixtures/polish_fixture.texother396 B
evals/fixtures/previous_final_issues.jsonconfig1.1 KB
evals/fixtures/previous_review_report.mdmarkdown205 B
evals/fixtures/quick_audit_fixture.texother538 B
examples/gate_example.mdmarkdown3.5 KB
examples/review_example.mdmarkdown2.5 KB
examples/self_check_example.mdmarkdown2.9 KB
references/AUDIT_GUIDE.mdmarkdown3.4 KB
references/CHANGELOG.mdmarkdown530 B
references/CHECKLIST.mdmarkdown6.0 KB
references/CONSOLIDATION_RULES.mdmarkdown907 B
references/DEEP_REVIEW_CRITERIA.mdmarkdown2.8 KB
references/FORBIDDEN_TERMS.mdmarkdown6.4 KB
references/ISSUE_SCHEMA.mdmarkdown928 B
references/LITERATURE_GROUNDING_GUIDE.mdmarkdown3.3 KB
references/POLISH_GUIDE.mdmarkdown2.3 KB
references/QUALITATIVE_STANDARDS.mdmarkdown5.5 KB
references/QUICK_REFERENCE.mdmarkdown1.1 KB
references/REVIEW_CRITERIA.mdmarkdown2.7 KB
references/REVIEW_LANE_GUIDE.mdmarkdown828 B
references/SCHOLAR_EVAL_GUIDE.mdmarkdown6.4 KB
references/SCORING_SYSTEMS.mdmarkdown1000 B
references/SUBAGENT_TEMPLATES.mdmarkdown987 B
references/TROUBLESHOOTING.mdmarkdown1.4 KB
references/VENUE_RULES.mdmarkdown764 B
references/editorial_decision_standards.mdmarkdown6.7 KB
references/quality_rubrics.mdmarkdown10.8 KB
scripts/audit.pyscript78.4 KB
scripts/build_claim_map.pyscript3.3 KB
scripts/check_citations.pyscript5.6 KB
scripts/check_references.pyscript19.9 KB
scripts/consolidate_review_findings.pyscript5.2 KB
scripts/detect_language.pyscript1.9 KB
scripts/diff_review_issues.pyscript2.0 KB
scripts/literature_compare.pyscript11.4 KB
scripts/literature_search.pyscript22.0 KB
scripts/models/scoring_model.jsonconfig737 B
scripts/parsers.pyscript3.3 KB
scripts/pdf_parser.pyscript11.4 KB
scripts/prepare_review_workspace.pyscript6.8 KB
scripts/render_deep_review_report.pyscript3.5 KB
scripts/report_generator.pyscript36.2 KB
scripts/scholar_eval.pyscript13.7 KB
scripts/scoring_model.pyscript6.2 KB
scripts/verify_quotes.pyscript1.6 KB
scripts/visual_check.pyscript10.7 KB
templates/audit_report_template.mdmarkdown2.8 KB
templates/final_issues_template.jsonconfig498 B
templates/review_report_template.mdmarkdown678 B
templates/revision_roadmap_template.mdmarkdown3.1 KB

evals/evals.json

View on GitHub
{
  "skill_name": "paper-audit",
  "evals": [
    {
      "id": 1,
      "prompt": "Run a quick audit on paper.tex for a NeurIPS submission and tell me what blocks submission versus what is only a quality improvement.",
      "expected_output": "Route to quick-audit, keep blockers first, and produce a compact readiness report rather than a full reviewer-style issue bundle.",
      "files": ["evals/fixtures/quick_audit_fixture.tex"],
      "assertions": [
        {"type": "contains", "text": "quick-audit", "description": "canonical quick-audit mode selected"},
        {"type": "regex", "pattern": "(block|Block|blocking)", "description": "blocking issues discussed"},
        {"type": "regex", "pattern": "\\[(Script|LLM)\\]", "description": "script/llm provenance visible"},
        {"type": "not_contains", "text": "Deep Review Report", "description": "quick audit does not drift into deep-review report format"}
      ]
    },
    {
      "id": 2,
      "prompt": "Review this manuscript like a real conference reviewer. I care more about the biggest validity threats than about grammar. Give me a structured revision roadmap and point me to the generated artifact files.",
      "expected_output": "Route to deep-review, produce a deep-review report, surface artifact paths, and emphasize major/moderate/minor findings plus a revision roadmap.",
      "files": ["evals/fixtures/deep_review_fixture.tex"],
      "assertions": [
        {"type": "contains", "text": "Deep Review Report", "description": "deep-review report rendered"},
        {"type": "contains", "text": "deep-review", "description": "canonical deep-review mode surfaced"},
        {"type": "regex", "pattern": "(Major Issues|Moderate Issues|Minor Issues)", "description": "severity-grouped issue sections present"},
        {"type": "regex", "pattern": "(Artifacts|artifact_dir|final_issues\\.json|review_report\\.md)", "description": "artifact references surfaced"},
        {"type": "regex", "pattern": "(Revision Roadmap|Priority 1)", "description": "roadmap present"},
        {"type": "regex", "pattern": "(final_issues\\.json|review_report\\.md)", "description": "required deep-review artifacts named explicitly"}
      ]
    },
    {
      "id": 3,
      "prompt": "Deep-review this paper and tell me whether the abstract and conclusion claims actually match the evidence in the experiments. I want quote-backed findings with lane/source metadata.",
      "expected_output": "Use a claims-vs-evidence framing, highlight quote-backed findings, and surface structured reviewer metadata rather than a simple checklist dump.",
      "files": ["evals/fixtures/deep_review_fixture.tex"],
      "assertions": [
        {"type": "contains", "text": "deep-review", "description": "canonical deep-review mode surfaced"},
        {"type": "regex", "pattern": "(claims|evidence|overclaim|unsupported)", "description": "claims-vs-evidence reasoning present"},
        {"type": "regex", "pattern": "(Quote|quote)", "description": "quote-backed issue output present"},
        {"type": "regex", "pattern": "(review_lane|claims_vs_evidence|source_kind|Quote Verified)", "description": "structured metadata visible"},
        {"type": "regex", "pattern": "(source_section|Related Sections)", "description": "section anchoring metadata visible"}
      ]
    },
    {
      "id": 4,
      "prompt": "Review this methods-heavy paper and check for notation drift, equation-to-text mismatch, and numbers that do not reconcile between the table and appendix. Show section anchors.",
      "expected_output": "Emphasize notation and numeric consistency, not just generic methodology commentary, and include section anchoring.",
      "files": ["evals/fixtures/deep_review_fixture.tex"],
      "assertions": [
        {"type": "contains", "text": "deep-review", "description": "canonical deep-review mode surfaced"},
        {"type": "regex", "pattern": "(notation|equation|numeric|appendix|reconcile)", "description": "cross-section consistency focus present"},
        {"type": "regex", "pattern": "(source_section|Section|Related Sections)", "description": "section anchoring present"},
        {"type": "regex", "pattern": "(review_lane|notation_and_numeric_consistency)", "description": "lane metadata surfaced"}
      ]
    },
    {
      "id": 5,
      "prompt": "Gate this IEEE paper and separate true submission blockers from advisory pseudocode recommendations like line numbers or shorter comments.",
      "expected_output": "Return PASS or FAIL and clearly distinguish blockers from recommendations.",
      "files": ["evals/fixtures/gate_ieee_fixture.tex"],
      "assertions": [
        {"type": "contains", "text": "Verdict", "description": "verdict shown"},
        {"type": "regex", "pattern": "(PASS|FAIL)", "description": "pass/fail value present"},
        {"type": "regex", "pattern": "(Block|Blocking|must fix)", "description": "blocking section present"},
        {"type": "regex", "pattern": "(advisory|recommendation|recommended)", "description": "advisory distinction present"},
        {"type": "not_contains", "text": "Deep Review Report", "description": "gate output stays in gate format"}
      ]
    },
    {
      "id": 6,
      "prompt": "Re-audit this revision against my previous report and tell me which root-cause issues are fully fixed, partially fixed, still open, or newly introduced.",
      "expected_output": "Use re-audit language with issue-level status classification, not a fresh audit only.",
      "files": [
        "evals/fixtures/deep_review_fixture.tex",
        "evals/fixtures/previous_final_issues.json",
        "evals/fixtures/previous_review_report.md"
      ],
      "assertions": [
        {"type": "contains", "text": "Re-Audit Report", "description": "re-audit report rendered"},
        {"type": "regex", "pattern": "(FULLY_ADDRESSED|PARTIALLY_ADDRESSED|NOT_ADDRESSED|NEW)", "description": "status categories present"},
        {"type": "regex", "pattern": "(root_cause_key|root cause)", "description": "root-cause comparison language present"}
      ]
    },
    {
      "id": 7,
      "prompt": "Run a deep review with literature grounding and tell me if the novelty claim is actually defended against the closest prior work.",
      "expected_output": "Use prior-art and novelty grounding logic, not just missing-reference detection.",
      "files": ["evals/fixtures/deep_review_fixture.tex"],
      "assertions": [
        {"type": "contains", "text": "deep-review", "description": "canonical deep-review mode surfaced"},
        {"type": "regex", "pattern": "(prior.art|prior-art|novelty|closest prior work|grounding)", "description": "prior-art reasoning present"},
        {"type": "regex", "pattern": "\\[(Script|LLM)\\]", "description": "provenance present"},
        {"type": "regex", "pattern": "(review_lane|prior-art|novelty)", "description": "review-lane level framing preserved"}
      ]
    },
    {
      "id": 8,
      "prompt": "Polish this paper only if the precheck says there are no blockers that would make polishing premature.",
      "expected_output": "Route to polish mode, evaluate safety first, and stop when blockers exist.",
      "files": ["evals/fixtures/polish_fixture.tex"],
      "assertions": [
        {"type": "contains", "text": "polish", "description": "polish mode acknowledged"},
        {"type": "regex", "pattern": "(precheck|blocker|safe)", "description": "precheck safety logic present"},
        {"type": "not_contains", "text": "Deep Review Report", "description": "polish flow does not drift into deep-review report output"}
      ]
    },
    {
      "id": 9,
      "prompt": "Gate this submission with an editor-in-chief screening first. Tell me if a busy top-venue EIC would desk-reject this paper based on the title, abstract, and introduction pitch quality alone.",
      "expected_output": "Run gate mode with EIC screening pass. Evaluate pitch quality, venue fit, fatal flaws, and presentation baseline before the detailed checklist.",
      "files": ["evals/fixtures/gate_ieee_fixture.tex"],
      "assertions": [
        {"type": "regex", "pattern": "(EIC|editor.in.chief|desk.reject|screening)", "description": "EIC screening language present"},
        {"type": "regex", "pattern": "(pitch|Pitch)", "description": "pitch quality assessment present"},
        {"type": "regex", "pattern": "(PASS|FAIL|Pass to Review|Desk Reject|Conditional Pass)", "description": "EIC verdict present"},
        {"type": "regex", "pattern": "(Verdict|verdict|score)", "description": "gate verdict or score shown"}
      ]
    },
    {
      "id": 10,
      "prompt": "Deep-review this qualitative research paper. Pay special attention to whether the coding process is transparent, whether data saturation is discussed, and whether there is adequate researcher reflexivity. Also check the theoretical contribution depth.",
      "expected_output": "Route to deep-review and apply qualitative methodology checks (B6-B10) plus theory contribution assessment (A5-A7). Flag missing SRQR elements.",
      "files": ["evals/fixtures/deep_review_fixture.tex"],
      "assertions": [
        {"type": "contains", "text": "deep-review", "description": "canonical deep-review mode surfaced"},
        {"type": "regex", "pattern": "(coding|saturation|reflexivity|triangulation|sampling)", "description": "qualitative methodology checks present"},
        {"type": "regex", "pattern": "(theor|contribution|concept|framework)", "description": "theory contribution assessment present"},
        {"type": "regex", "pattern": "(Major Issues|Moderate Issues|Minor Issues)", "description": "severity-grouped findings present"}
      ]
    },
    {
      "id": 11,
      "prompt": "Deep-review this paper using the Academic Pre-Review Committee (Editor -> Theory -> Literature -> Methodology -> Logic). Be harsh, quote-backed, and point me to the committee artifacts in the deep-review workspace.",
      "expected_output": "Run deep-review with committee routing, produce committee artifacts, and embed a committee section in the deep-review report when available.",
      "files": ["evals/fixtures/deep_review_fixture.tex"],
      "assertions": [
        {"type": "regex", "pattern": "(Academic Pre-Review Committee|Committee)", "description": "committee section present"},
        {"type": "regex", "pattern": "(Editor|Theory|Literature|Methodology|Logic)", "description": "committee roles present"}
      ]
    },
    {
      "id": 12,
      "prompt": "Deep-review this paper but focus ONLY on methodology transparency (SRQR). I want MUST-FIX vs SHOULD-FIX items and quote-backed findings. Use --focus methodology if needed.",
      "expected_output": "Route to deep-review and produce a methodology-focused committee output with MUST-FIX vs SHOULD-FIX separation and SRQR deltas.",
      "files": ["evals/fixtures/deep_review_fixture.tex"],
      "assertions": [
        {"type": "regex", "pattern": "(MUST-FIX|SHOULD-FIX|SRQR)", "description": "methodology transparency structure present"}
      ]
    },
    {
      "id": 13,
      "prompt": "Deep-review this paper but focus ONLY on theory contribution. Give me exactly 3 fatal theory holes with quote+location, plus 2-4 concrete moves to fix. Use --focus theory if needed.",
      "expected_output": "Route to deep-review and produce a theory-focused committee output with 3 fatal theory holes and concrete fixes.",
      "files": ["evals/fixtures/deep_review_fixture.tex"],
      "assertions": [
        {"type": "regex", "pattern": "(3 Fatal Theory Holes|Fatal Theory Holes|Theory Contribution)", "description": "theory review structure present"}
      ]
    },
    {
      "id": 14,
      "prompt": "Deep-review this paper but focus ONLY on the logic chain. I want a logic chain diagnostic diagram (Mermaid flowchart is fine) and quote-backed breakpoints. Use --focus logic if needed.",
      "expected_output": "Route to deep-review and produce a logic-chain-focused committee output including a diagnostic diagram and quoted breakpoints.",
      "files": ["evals/fixtures/deep_review_fixture.tex"],
      "assertions": [
        {"type": "regex", "pattern": "(mermaid|flowchart|Logic Chain)", "description": "logic chain diagnostic present"}
      ]
    }
  ]
}
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 196kUpdated Claude CodeCodexVerifiedInstall