brycewang-stanford/Auto-Empirical-Research-Skills / skills/35-bahayonghang-academic-writing-skills/skills/paper-audit
paper-audit
VerifiedDeep-review-first audit for Chinese and English academic papers across LaTeX, Typst, and PDF formats. Use whenever the user wants reviewer-style paper critique, pre-submission readiness checks, pass/fail gate decisions, structured revision roadmaps, or re-audits of revised manuscripts. Trigger even if the user only says "review my paper", "check if this is ready to submit", "audit this PDF", "simulate peer review", "find the biggest problems in this manuscript", or "re-check whether I fixed the review issues". Do not use for direct source editing or compilation-heavy repair; route those to the format-specific writing skills instead.
references/editorial_decision_standards.md
View on GitHub# Editorial Decision Standards
Reference document for the Synthesis Agent. Defines consensus classification, arbitration rules, score divergence handling, and decision mapping.
---
## 1. Consensus Labels
| Label | Definition | Threshold | Handling |
|-------|-----------|-----------|----------|
| `[CONSENSUS-ALL]` | All 3 agent reviewers flag the same issue | 3/3 agree | Priority 1 — author MUST address |
| `[CONSENSUS-MAJORITY]` | 2 of 3 agent reviewers flag the same issue | 2/3 agree | Priority 1 or 2 (severity-dependent) |
| `[SPLIT]` | No majority — fundamental disagreement | <2/3 | Requires arbitration; document all positions |
| `[AUTOMATED-ONLY]` | Issue found only by Phase 0 scripts | Script-detected | Include as-is; severity from script |
### Matching Criteria
Two findings are considered "the same issue" when:
1. They reference the **same location** (section, figure, table) in the paper, AND
2. They describe the **same underlying problem** (even if phrased differently)
Do NOT match findings that merely share a category (e.g., two unrelated "overclaim" findings in different sections are separate issues).
---
## 2. Arbitration Rules
When reviewers disagree (SPLIT), the Synthesis Agent resolves using these principles in strict priority order:
### Priority 1: Evidence Principle
The position backed by **specific textual evidence** (quotes, data points, citations) outweighs positions based on general impressions.
**Example**: Methodology Reviewer says "Table 3 shows p < 0.05" vs. Critical Reviewer says "results seem unconvincing" — Methodology Reviewer's evidence-backed position wins.
### Priority 2: Expertise Principle
On domain-specific disputes, weight the relevant specialist higher:
| Dispute Type | Primary Authority | Rationale |
|-------------|-------------------|-----------|
| Research design, statistics | Methodology Reviewer | Technical methods expertise |
| Literature coverage, novelty | Domain Reviewer | Field knowledge |
| Logical argument, overclaims | Critical Reviewer | Argumentation expertise |
| Cross-cutting (scope unclear) | Equal weight | Use Evidence Principle |
### Priority 3: Conservative Principle
When evidence and expertise are balanced, lean toward the **more cautious** (more critical) assessment. Rationale: it is safer to flag a potential issue for the author to address than to dismiss a genuine concern.
### Documentation Requirement
Every arbitration MUST be documented with this structure:
```markdown
#### Arbitration: [Issue Title]
- **Disagreement**: [Concise description]
- **Position A** ([Reviewer]): [Their view, with quote from their report]
- **Position B** ([Reviewer]): [Their view, with quote from their report]
- **Principle Applied**: [Evidence / Expertise / Conservative]
- **Resolution**: [Decision and rationale]
- **Confidence**: [High / Medium / Low]
```
---
## 3. Score Divergence Handling
### Detection Threshold
A **score divergence** occurs when two reviewers rate the same dimension with a gap > 2.0 points.
### Handling Protocol
| Gap Size | Action |
|----------|--------|
| <= 1.0 | Average directly, no comment needed |
| 1.1 - 2.0 | Average and note the spread in the report |
| > 2.0 | **Mandatory explanation**: investigate cause, apply arbitration, document rationale for final score |
### Investigation Steps for Large Divergence
1. Identify which specific findings drove each reviewer's score
2. Check if one reviewer considered evidence the other missed
3. Apply the Arbitration Rules (Section 2) to the underlying disagreement
4. Set the final score with documented rationale (do NOT simply average)
### Score Merging Table
| Dimension | Primary Source | Secondary Source | Merge Rule |
|-----------|---------------|-----------------|------------|
| Soundness | Methodology Reviewer | Critical Reviewer | Average; flag if gap > 2 |
| Reproducibility | Methodology Reviewer | — | Direct use |
| Novelty | Domain Reviewer | — | Direct use |
| Significance | Domain Reviewer | — | Direct use |
| 4-dim NeurIPS (Quality, Clarity, Significance, Originality) | Phase 0 script | — | Direct use (objective, deduction-based) |
---
## 4. Decision Matrix
### For Review Mode (Advisory)
| Consensus | Average Score (8-dim) | Recommendation | Typical Action |
|-----------|----------------------|----------------|---------------|
| CONSENSUS-ALL on critical flaw | Any | **Reject** | Fundamental rework needed |
| CONSENSUS-ALL, no critical | >= 7.0 | **Accept** | Minor revisions at most |
| CONSENSUS-ALL, no critical | 5.0 - 6.9 | **Revise & Resubmit** | Address specific weaknesses |
| CONSENSUS-ALL, no critical | < 5.0 | **Reject** | Significant quality concerns |
| CONSENSUS-MAJORITY | >= 7.0 | **Conditional Accept** | Address majority concerns |
| CONSENSUS-MAJORITY | 5.0 - 6.9 | **Major Revision** | Substantial revision needed |
| CONSENSUS-MAJORITY | < 5.0 | **Reject** | Multiple serious issues |
| SPLIT | >= 7.0 | **Discuss** | Highlight areas of disagreement |
| SPLIT | < 7.0 | **Major Revision** | Err on cautious side |
### For 4-Dimension Scale (1-6)
| Overall Score | Label | Interpretation |
|---------------|-------|----------------|
| 5.5 - 6.0 | Exceptional | Top-tier quality, ready for submission |
| 4.5 - 5.4 | Strong | Minor issues only |
| 3.5 - 4.4 | Adequate | Noticeable issues, revision recommended |
| 2.5 - 3.4 | Weak | Significant revision required |
| 1.0 - 2.4 | Insufficient | Fundamental problems |
---
## 5. Report Confidence Indicators
The Synthesis Agent should indicate confidence in the final assessment:
| Indicator | When to Use |
|-----------|------------|
| **High Confidence** | CONSENSUS-ALL on most findings, small score spreads, clear evidence |
| **Medium Confidence** | CONSENSUS-MAJORITY with some SPLIT, moderate score spreads |
| **Low Confidence** | Multiple SPLIT findings, large score divergences, limited evidence |
Include the confidence indicator in the Overall Assessment section of the final report.
---
## 6. Edge Cases
### When All Reviewers Miss an Automated Finding
Phase 0 automated findings that no agent reviewer addresses should be included verbatim in the final report. Do not suppress automated findings simply because agents did not comment on them.
### When an Agent Finding Contradicts Automated Data
If an agent reviewer's assessment contradicts objective automated data (e.g., agent says "citations are complete" but BIB module found undefined references), the automated data takes precedence for factual matters.
### When Critical Reviewer Flags a CRITICAL Issue
Critical Reviewer CRITICAL findings are **never suppressed**, even if other reviewers disagree. They must appear in the final report and be addressed in the Revision Roadmap as Priority 1. The Synthesis Agent may note that other reviewers disagree, but the finding itself must remain.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |