Skip to content

Lambenthan/empiricalwiki / i18n/en/skills/novelty

novelty

Verified

Multi-source novelty verification — WebSearch + Semantic Scholar + wiki + Review LLM cross-verify — outputs novelty score and recommendations

/novelty

Verify the novelty of a research idea or method using multiple sources. Searches WebSearch, Semantic Scholar, existing wiki work, and arXiv recent preprints, then Review LLM cross-verifies. Outputs a novelty score (1-5), closest prior work, differentiation points, and next-step recommendations. Can be used standalone or called by /ideate Phase 4.

Inputs

  • target: one of the following:
    • free-text description of the idea (a paragraph or a few sentences)
    • slug of an ideas/ page in the wiki (e.g. sparse-lora-for-edge-devices)
    • paper title or arXiv URL (check novelty of that paper's method)
  • --quick: fast mode, skip Review LLM cross-verify (Step 3), search only
  • --verbose: output full search results, not just summaries

Outputs

  • Novelty Report (output to terminal, not written to wiki):
    • Novelty Score (1-5)
    • List of closest prior work (top 3-5)
    • Differentiation points versus each prior work
    • Review LLM cross-verify assessment (unless --quick)
    • Recommended action: proceed / modify / abandon
  • This skill is a read-only query — it does not modify any wiki content

Wiki Interaction

Reads

  • wiki/papers/*.md — search existing papers for similar methods
  • wiki/concepts/*.md — check concept overlap
  • wiki/ideas/*.md — check for duplication with existing ideas (especially failure_reason of failed ideas)
  • wiki/claims/*.md — check the current status of claims the idea depends on
  • wiki/graph/context_brief.md — global context to assist search

Writes

  • None. Novelty check is a pure query operation; it does not modify the wiki.

Graph edges created

  • None.

Workflow

Precondition: confirm working directory is the wiki project root (containing wiki/, raw/, tools/).

Step 1: Extract Method Signature

  1. If target is a slug: read wiki/ideas/{slug}.md, extract title, Hypothesis, Approach sketch
  2. If target is free text: use directly
  3. If target is an arXiv URL: download the abstract, extract method description
  4. Extract the "method signature" from the target — the core elements of the method:
    • What: what it does (task / goal)
    • How: the method used (technical approach)
    • Why novel: claimed innovation
  5. Generate 3-5 core keywords for subsequent searches

Step 2: Multi-Source Search

Execute the following searches in parallel (use Agent tool for concurrency):

Source A — Web Search (5+ queries):

  1. Direct query: "<method-name>" + "<task>" — exact phrase search
  2. Component query: <component-1> + <component-2> + <domain> — component combination search
  3. Survey query: "survey" OR "review" + <task-area> + 2024 2025
  4. Competitor query: <alternative-approach> + <same-task>
  5. Recent query: <method-keywords> + arXiv + 2025 2026

Source B — Semantic Scholar + DeepXiv:

python3 tools/fetch_s2.py search "<method-keywords>" --limit 20
python3 tools/fetch_deepxiv.py search "<method-keywords>" --mode hybrid --limit 20

Merge results from both sources (deduplicate by arxiv_id). DeepXiv's hybrid semantic search finds semantically similar work that S2 keyword search may miss.

  • Fetch details and TLDR for top 5 results:
python3 tools/fetch_s2.py paper <s2_id>
python3 tools/fetch_deepxiv.py brief <arxiv_id>

Use DeepXiv brief TLDRs to quickly judge method similarity. If DeepXiv is unavailable: fall back to S2 search only (original behavior).

Source C — Wiki Internal Search:

  1. Scan Key idea and Method sections of all pages in wiki/papers/
  2. Scan Definition and Variants sections of wiki/concepts/
  3. Scan all content in wiki/ideas/, with special attention to:
    • ideas with status = failed and their failure_reason (anti-repetition)
    • ideas with status = proposed/in_progress (avoid internal duplication)
  4. Read wiki/graph/context_brief.md for global perspective

Source D — Recent arXiv Preprints:

  • Use WebSearch: site:arxiv.org <method-keywords> 2025 2026

Step 3: Review LLM Cross-Verify

(Skip if --quick)

Submit the following to Review LLM for independent assessment:

mcp__llm-review__chat:
  system: "You are a senior ML researcher assessing the novelty of a proposed method.
           Be rigorous: if the method is essentially a recombination of known techniques
           with minor changes, score it low. Only score 4-5 if there is a genuinely new
           insight or formulation."
  message: |
    ## Proposed Method
    {method signature from Step 1}

    ## Existing Similar Work Found
    {top 5 similar works from Step 2, with title + one-line summary}

    ## Questions
    1. Is this method genuinely novel, or a minor variation of existing work?
    2. What is the closest existing work and what's the real difference?
    3. Novelty score 1-5 with justification.
    4. If score <= 2, what modification could increase novelty?

Step 4: Generate Novelty Report

Synthesize Step 2 search results and Step 3 Review LLM assessment into a structured report:

# Novelty Report: {idea title}

## Score: {1-5}/5 — {label}

| Score | Label | Meaning |
|-------|-------|---------|
| 1 | Published | Highly similar published work exists |
| 2 | Very Similar | Very similar method exists, only minor differences |
| 3 | Incremental | Clear incremental contribution over existing work |
| 4 | Novel Combination | Creatively combines existing techniques, producing new insight |
| 5 | Fundamentally New | Proposes an entirely new paradigm or formulation |

## Closest Prior Work

1. **{title}** ({year}) — {one-sentence description of the similarity}
   - Difference: {key distinction between this method and the prior work}
   - Wiki link: [[slug]] (if it exists)
2. ...

## Review LLM Assessment
{summary of Review LLM's independent judgment}

## Anti-repetition Check
- Failed ideas in wiki: {list relevant failed ideas with failure_reason}
- In-progress ideas in wiki: {list potentially overlapping ideas}

## Recommendation
- **{proceed / modify / abandon}**
- Rationale: {one paragraph}
- If modify: suggested differentiation directions: {specific suggestions}

Scoring rules (composite judgment):

  • Take the lower of Claude's search-based score and Review LLM's score (conservative principle)
  • If wiki contains a failed idea whose failure_reason overlaps with this idea → lower score by 1
  • If wiki contains a highly overlapping in_progress idea → mark as abandon (internal duplication)

Constraints

  • Do not modify the wiki: novelty check is a pure query; all results are output to terminal only
  • Conservative scoring: underestimate novelty rather than overestimate to avoid wasting effort on known work
  • Must check failed ideas: ideas with status=failed in wiki/ideas/ are important anti-repetition signals
  • Search coverage: at least 5 distinct WebSearch queries + Semantic Scholar + wiki internal search
  • Review LLM independence: do not include Claude's own novelty judgment when submitting to Review LLM; let Review LLM assess independently
  • Cite real sources: all prior work listed in the report must be real (returned by WebSearch/S2); do not fabricate

Error Handling

  • WebSearch unavailable: skip Sources A and D, rely only on S2 + wiki search; note limited coverage in report
  • Semantic Scholar API unavailable: skip S2 portion, use DeepXiv + WebSearch as compensation
  • DeepXiv API unavailable: skip DeepXiv portion, rely on S2 + WebSearch (fall back to original behavior)
  • Review LLM unavailable: skip Step 3; annotate report with "Review LLM cross-verify unavailable, single-model assessment only"
  • Wiki empty: proceed with external searches normally; annotate wiki internal search section with "wiki empty"
  • idea slug not found: prompt user to check the slug, list available slugs in wiki/ideas/

Dependencies

Tools(via Bash)

  • python3 tools/fetch_s2.py search "<query>" --limit 20 — Semantic Scholar keyword search
  • python3 tools/fetch_s2.py paper <s2_id> — fetch paper details
  • python3 tools/fetch_deepxiv.py search "<query>" --mode hybrid --limit 20 — DeepXiv semantic search
  • python3 tools/fetch_deepxiv.py brief <arxiv_id> — fetch paper TLDR for similarity judgment

MCP Servers

  • mcp__llm-review__chat — Review LLM cross-verify (Step 3)

Claude Code Native

  • WebSearch — multi-query web search (Step 2 Sources A + D)
  • Agent tool — parallel execution of multi-source search (Step 2)

Shared References

  • .claude/skills/shared-references/cross-model-review.md (created in Phase 2, Review LLM independence principle)
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall