Skip to content

brycewang-stanford/Auto-Empirical-Research-Skills / skills/35-bahayonghang-academic-writing-skills/skills/paper-audit

paper-audit

Verified

Deep-review-first audit for Chinese and English academic papers across LaTeX, Typst, and PDF formats. Use whenever the user wants reviewer-style paper critique, pre-submission readiness checks, pass/fail gate decisions, structured revision roadmaps, or re-audits of revised manuscripts. Trigger even if the user only says "review my paper", "check if this is ready to submit", "audit this PDF", "simulate peer review", "find the biggest problems in this manuscript", or "re-check whether I fixed the review issues". Do not use for direct source editing or compilation-heavy repair; route those to the format-specific writing skills instead.

FileKindSize
SKILL.mdskill md14.3 KB
agents/claims_evidence_reviewer_agent.mdmarkdown354 B
agents/committee_editor_agent.mdmarkdown1.9 KB
agents/committee_literature_agent.mdmarkdown1.9 KB
agents/committee_logic_agent.mdmarkdown1.5 KB
agents/committee_methodology_agent.mdmarkdown1.8 KB
agents/committee_theory_agent.mdmarkdown1.9 KB
agents/critical_reviewer_agent.mdmarkdown8.4 KB
agents/domain_reviewer_agent.mdmarkdown7.1 KB
agents/editor_in_chief_agent.mdmarkdown5.2 KB
agents/evaluation_fairness_reviewer_agent.mdmarkdown371 B
agents/literature_reviewer_agent.mdmarkdown3.6 KB
agents/methodology_reviewer_agent.mdmarkdown7.2 KB
agents/notation_consistency_reviewer_agent.mdmarkdown398 B
agents/prior_art_reviewer_agent.mdmarkdown2.9 KB
agents/section_reviewer_agent.mdmarkdown365 B
agents/self_consistency_reviewer_agent.mdmarkdown420 B
agents/synthesis_agent.mdmarkdown951 B
evals/evals.jsonconfig12.0 KB
evals/fixtures/deep_review_fixture.texother699 B
evals/fixtures/gate_ieee_fixture.texother617 B
evals/fixtures/polish_fixture.texother396 B
evals/fixtures/previous_final_issues.jsonconfig1.1 KB
evals/fixtures/previous_review_report.mdmarkdown205 B
evals/fixtures/quick_audit_fixture.texother538 B
examples/gate_example.mdmarkdown3.5 KB
examples/review_example.mdmarkdown2.5 KB
examples/self_check_example.mdmarkdown2.9 KB
references/AUDIT_GUIDE.mdmarkdown3.4 KB
references/CHANGELOG.mdmarkdown530 B
references/CHECKLIST.mdmarkdown6.0 KB
references/CONSOLIDATION_RULES.mdmarkdown907 B
references/DEEP_REVIEW_CRITERIA.mdmarkdown2.8 KB
references/FORBIDDEN_TERMS.mdmarkdown6.4 KB
references/ISSUE_SCHEMA.mdmarkdown928 B
references/LITERATURE_GROUNDING_GUIDE.mdmarkdown3.3 KB
references/POLISH_GUIDE.mdmarkdown2.3 KB
references/QUALITATIVE_STANDARDS.mdmarkdown5.5 KB
references/QUICK_REFERENCE.mdmarkdown1.1 KB
references/REVIEW_CRITERIA.mdmarkdown2.7 KB
references/REVIEW_LANE_GUIDE.mdmarkdown828 B
references/SCHOLAR_EVAL_GUIDE.mdmarkdown6.4 KB
references/SCORING_SYSTEMS.mdmarkdown1000 B
references/SUBAGENT_TEMPLATES.mdmarkdown987 B
references/TROUBLESHOOTING.mdmarkdown1.4 KB
references/VENUE_RULES.mdmarkdown764 B
references/editorial_decision_standards.mdmarkdown6.7 KB
references/quality_rubrics.mdmarkdown10.8 KB
scripts/audit.pyscript78.4 KB
scripts/build_claim_map.pyscript3.3 KB
scripts/check_citations.pyscript5.6 KB
scripts/check_references.pyscript19.9 KB
scripts/consolidate_review_findings.pyscript5.2 KB
scripts/detect_language.pyscript1.9 KB
scripts/diff_review_issues.pyscript2.0 KB
scripts/literature_compare.pyscript11.4 KB
scripts/literature_search.pyscript22.0 KB
scripts/models/scoring_model.jsonconfig737 B
scripts/parsers.pyscript3.3 KB
scripts/pdf_parser.pyscript11.4 KB
scripts/prepare_review_workspace.pyscript6.8 KB
scripts/render_deep_review_report.pyscript3.5 KB
scripts/report_generator.pyscript36.2 KB
scripts/scholar_eval.pyscript13.7 KB
scripts/scoring_model.pyscript6.2 KB
scripts/verify_quotes.pyscript1.6 KB
scripts/visual_check.pyscript10.7 KB
templates/audit_report_template.mdmarkdown2.8 KB
templates/final_issues_template.jsonconfig498 B
templates/review_report_template.mdmarkdown678 B
templates/revision_roadmap_template.mdmarkdown3.1 KB

references/LITERATURE_GROUNDING_GUIDE.md

View on GitHub
# Literature Grounding Scoring Guide

New 9th dimension for ScholarEval v3.0: evaluates how well a paper is grounded in existing literature.

## Overview

Literature Grounding measures the extent to which a paper:
1. Covers relevant prior work comprehensively
2. Cites recent developments in the field
3. Accurately positions itself relative to existing literature
4. Identifies and fills genuine research gaps

## Scoring Rubric (1-10)

### Score Components

Literature Grounding uses a **mixed** source (Script + LLM):

- **Script component** (from `literature_compare.py`): Automated coverage analysis
  - Coverage ratio (cited papers found in search results)
  - Recency score (fraction of recent papers cited)
  - Missing references penalty
  - Freshness distribution

- **LLM component** (from Domain Reviewer): Qualitative assessment
  - Thematic organization quality
  - Critical analysis depth
  - Research gap derivation
  - Citation context accuracy

### Scoring Criteria

| Score | Level | Behavioral Indicators |
|-------|-------|----------------------|
| 9-10 | Excellent | Comprehensive coverage of seminal and recent works; thematically organized; clear gap derivation; no important missing references |
| 7-8 | Good | Solid coverage with minor gaps; mostly thematic organization; adequate gap identification; 1-2 missing recent papers |
| 5-6 | Fair | Reasonable coverage but notable gaps; some enumeration instead of thematic analysis; weak gap derivation; several missing references |
| 3-4 | Poor | Incomplete coverage; predominantly enumerated; no clear gap derivation; many important references missing |
| 1-2 | Failing | Minimal literature review; critical foundational works missing; no connection between literature and contribution |

### Script Score Computation

The script component (`compute_literature_grounding_score`) uses these weights:

| Factor | Weight | Description |
|--------|--------|-------------|
| Coverage ratio | 40% | `cited_and_found / total_found` -- higher is better |
| Recency | 20% | Fraction of results from last 3 years |
| Missing refs penalty | 30% | Penalizes `found_not_cited` important papers |
| Freshness | 10% | Distribution of publication years |

### LLM Evaluation Prompt

> Evaluate the literature grounding of this paper. Consider:
> - Does the paper cite foundational works with correct attribution?
> - Are key papers from the last 3 years covered?
> - Is the literature organized thematically with critical analysis?
> - Is there an explicit research gap connecting literature to this paper's contribution?
> - Are there obvious omissions in the related work section?

### Merge Rule

Final score = average(script_partial, llm_score) when both available.
Use whichever is available when only one source has data.

## Relationship to Domain Reviewer

The Domain Reviewer agent provides the LLM component of Literature Grounding through its existing literature assessment criteria (A1-A4):
- A1: Thematic vs enumerated organization
- A2: Critical analysis completeness
- A3: Research gap derivation
- A4: Citation density funnel

## Integration

- CLI flag: `--literature-search` enables the script component
- Without `--literature-search`, Literature Grounding shows "N/A (awaiting literature search)"
- In review mode, the Domain Reviewer always provides the LLM component
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 196kUpdated Claude CodeCodexVerifiedInstall