Skip to content

Lambenthan/empiricalwiki / i18n/en/skills/daily-arxiv

daily-arxiv

Verified

Daily arXiv pull, relevance filtering, auto-ingest of high-priority papers, and SOTA update detection

/daily-arxiv

Pulls new papers from arXiv RSS daily, automatically assesses relevance based on research directions and concepts in the wiki, calls /ingest to fully incorporate highly relevant papers into the wiki, detects SOTA updates, and generates a digest log. Supports cron-scheduled automatic execution as well as manual triggering.

Inputs

  • --hours N: pull papers from the last N hours (default 24)
  • --max-ingest N: maximum papers to ingest per run (default 5, prevents wiki overload)
  • --dry-run: generate digest only, do not execute ingest
  • --categories: override default arXiv categories (default: cs.LG cs.CV cs.CL cs.AI stat.ML)

Outputs

  • raw/discovered/{slug}/ or raw/discovered/{slug}.pdf — fetched source artifact for each auto-ingested paper
  • wiki/papers/{slug}.md — highly relevant paper pages (created via /ingest)
  • Corresponding concepts/, people/, claims/ pages (created via /ingest)
  • Updated wiki/topics/*.md — SOTA tracker annotations (if SOTA update detected)
  • Updated wiki/graph/ — edges.jsonl, context_brief.md, open_questions.md (maintained via /ingest)
  • Updated wiki/index.md and wiki/log.md

Wiki Interaction

Reads

  • wiki/topics/*.md — extract Overview keywords and SOTA tracker, used for relevance scoring and SOTA detection
  • wiki/concepts/*.md — extract Definition keywords, assist relevance scoring
  • wiki/index.md — check whether a paper is already collected (deduplicate by arXiv URL)
  • wiki/papers/*.md — check whether an arxiv ID already exists
  • wiki/graph/open_questions.md — prioritize ingesting papers that fill knowledge gaps

Writes

  • wiki/papers/{slug}.md — CREATE via /ingest
  • wiki/concepts/{slug}.md — CREATE/EDIT via /ingest
  • wiki/people/{slug}.md — CREATE/EDIT via /ingest
  • wiki/claims/{slug}.md — CREATE/EDIT via /ingest
  • wiki/topics/{slug}.md — EDIT (SOTA tracker annotations)
  • wiki/graph/edges.jsonl — APPEND via /ingest
  • wiki/graph/context_brief.md — REBUILD (once at the end)
  • wiki/graph/open_questions.md — REBUILD (once at the end)
  • wiki/index.md — EDIT via /ingest
  • wiki/log.md — APPEND

Graph edges created

  • All edges created by /ingest (paper → concept, paper → claim, etc.)

Workflow

Pre-conditions: confirm the working directory is the wiki project root (directory containing wiki/, raw/, tools/). Set WIKI_ROOT=wiki/.

Step 1: Pull arXiv RSS + Trending Papers

  1. Run fetch_arxiv.py to get the new paper list:
    python3 tools/fetch_arxiv.py --hours <hours> -o /tmp/arxiv_feed.json
    
  2. Fetch DeepXiv trending papers (past 7 days):
    python3 tools/fetch_deepxiv.py trending --days 7 --limit 20
    
    Merge trending papers into the candidate list (deduplicated by arxiv_id); trending papers receive extra attention in subsequent scoring. If DeepXiv is unavailable: skip this sub-step, use RSS results only.
  3. Parse results to obtain the paper list (title, abstract, authors, arxiv_url, arxiv_id, category)
  4. Deduplication: read wiki/index.md, skip papers whose arXiv URL is already in the wiki. Also check existing arxiv IDs in the wiki/papers/ directory.
  5. If no new papers, skip directly to Step 6 to generate an empty digest.

Step 2: Build Relevance Context + DeepXiv Enhancement

  1. Read wiki/topics/*.md and extract for each topic:
    • Core keywords from the Overview paragraph
    • Open problems / Research gaps list
    • Current best results from the SOTA tracker
  2. Read wiki/concepts/*.md and extract for each concept:
    • Key terms from the Definition paragraph
    • tags list
  3. Read wiki/graph/open_questions.md for the current knowledge gap list
  4. Synthesize a "research direction summary" (≤ 2000 characters) containing: core topics, active concepts, gaps to fill
  5. DeepXiv TLDR enhancement (optional): for each new paper, fetch an AI summary and keywords to improve scoring quality:
    python3 tools/fetch_deepxiv.py brief <arxiv_id>
    
    Supplement the original abstract with the returned tldr and keywords to help the LLM judge relevance more precisely. If DeepXiv is unavailable: use only the RSS original title + abstract for scoring (fallback to original behavior).

Step 3: Relevance Scoring

For each new paper, LLM assesses relevance based on title and abstract vs. the research direction summary:

ScoreMeaningAction
3Highly relevant: significant advance in a core directionAuto-ingest
2Moderately relevant: worth noting but not coreList in digest, do not auto-ingest
1Weakly relevant: for reference onlyCollapsed listing
0Not relevantSkip

Bonus rules (can promote a score of 2 to 3):

  • Paper directly addresses a knowledge gap in open_questions.md → +1
  • Paper's benchmark may update the SOTA tracker → +1 (capped at 3)

Batch scoring: submit all papers' title+abstract to the LLM in a single call and return scores as JSON. Avoid per-paper calls.

Step 4: Auto-Ingest High-Priority Papers (with checkpoint resume)

  1. Filter papers with relevance = 3, sorted by the following priority:
    • Papers that fill gap_map gaps first
    • Papers with higher citation counts first (if abstract mentions SOTA results)
  2. Load checkpoint (skip already-completed papers if one exists):
    python3 tools/research_wiki.py checkpoint-load wiki/ "daily-arxiv-{date}"
    
  3. Take the first --max-ingest papers (default 5). For each selected paper:
    • Download the source artifact into raw/discovered/ first:
      python3 tools/init_discovery.py download --raw-root raw --arxiv-id <arxiv_id> --title "<title>"
      
    • Pass the returned canonical_ingest_path from raw/discovered/ into /ingest, not the bare arXiv URL
    • /ingest completes the full wiki incorporation flow (paper + concepts + people + claims + cross-refs + graph)
    • After each success, record checkpoint:
      python3 tools/research_wiki.py checkpoint-save wiki/ "daily-arxiv-{date}" "{arxiv_id}"
      
    • On failure, mark and continue:
      python3 tools/research_wiki.py checkpoint-save wiki/ "daily-arxiv-{date}" "{arxiv_id}" --failed
      
  4. If --dry-run, skip both the raw/discovered/ download and the actual ingest; mark "would ingest" in the digest
  5. After all done, clear checkpoint:
    python3 tools/research_wiki.py checkpoint-clear wiki/ "daily-arxiv-{date}"
    

Step 5: SOTA Detection and Update

  1. For each paper ingested in Step 4, check the benchmark numbers in its Results section
  2. Compare benchmarks against the ## SOTA tracker in the corresponding wiki/topics/ page
  3. If the paper's results beat the current SOTA record:
    • Append/update an entry in the topic page's ## SOTA tracker:
      - **{benchmark_name}**: {score} ← [[{paper-slug}]] ({year}) [previously: {old_score}]
      
    • Set sota_updated for that topic to today's date
  4. If SOTA updates are detected, highlight them in the digest

Step 6: Generate Digest and Write to Log

  1. Rebuild graph derived files (only if any ingest happened):

    python3 tools/research_wiki.py rebuild-context-brief wiki/
    python3 tools/research_wiki.py rebuild-open-questions wiki/
    
  2. Append digest to wiki/log.md:

    python3 tools/research_wiki.py log wiki/ "daily-arxiv | {N_ingested} ingested, {N_relevant} relevant / {N_total} total"
    
  3. Append detailed digest below the current day's log entry:

    ### High Priority (ingested)
    - [[paper-slug]] — {title} ({one-line insight})
    
    ### Worth Watching (relevance = 2)
    - {title} — {arxiv_url} — {one-line summary}
    
    ### Trending This Week (from DeepXiv)
    - {title} — {arxiv_id} — {tweets} tweets, {views} views
    
    ### SOTA Updates
    - {topic}: {benchmark} new record by [[paper-slug]]
    
    <details>
    <summary>Weakly Relevant ({K} papers)</summary>
    
    - {title} — {arxiv_url}
    
    </details>
    

Step 7: Report to User

Output summary:

  • Total papers scanned / count after deduplication
  • Distribution across relevance levels
  • List of ingested papers (with slug links)
  • List of SOTA updates (if any)
  • Recommended manual ingest candidates (top 3 most notable from relevance = 2)
  • Next run time reminder

Constraints

  • Only ingest papers with relevance >= 3: leave the rest for user judgment, do not auto-create wiki pages
  • At most --max-ingest papers per run (default 5): prevents single-run wiki overload
  • /daily-arxiv is raw-read-only except raw/discovered/ for auto-ingested papers: never write to raw/papers/, raw/tmp/, raw/notes/, or raw/web/
  • graph/ maintained via tools only: do not manually edit graph files
  • Bidirectional links: guaranteed by /ingest
  • Deduplication must be strict: double-check by both arxiv_url and arxiv_id
  • Batch scoring: one LLM call to score all papers, no per-paper calls
  • Digest stays concise: see individual papers pages for details; at most one line per paper in the digest
  • log.md is append-only: use python3 tools/research_wiki.py log to append

Error Handling

  • DeepXiv API unavailable: fall back to pure RSS mode (original behavior). Trending section omitted from digest; scoring uses only raw RSS data. Note DeepXiv unavailability in the report.
  • RSS fetch fails: report network error, suggest user check network and retry. Do not modify the wiki.
  • Partial ingest failures: keep completed ingests, mark failed papers in the report, suggest user manually /ingest <url>.
  • wiki directory does not exist: prompt user to run /init first.
  • Empty RSS results: normal situation (fewer papers on holidays/weekends), generate empty digest without error.
  • SOTA comparison fails: if benchmark format does not match, skip and note in report.

Dependencies

Skills(via Skill tool)

  • /ingest — full paper incorporation flow (called in Step 4)

Tools(via Bash)

  • python3 tools/fetch_arxiv.py --hours <N> -o <path> — pull arXiv RSS
  • python3 tools/fetch_deepxiv.py trending --days 7 --limit 20 — fetch trending papers
  • python3 tools/fetch_deepxiv.py brief <arxiv_id> — fetch paper TLDR and keywords
  • python3 tools/init_discovery.py download --raw-root raw --arxiv-id <id> --title "<title>" — download selected papers into raw/discovered/
  • python3 tools/research_wiki.py rebuild-context-brief wiki/ — rebuild compressed context
  • python3 tools/research_wiki.py rebuild-open-questions wiki/ — rebuild knowledge gap map
  • python3 tools/research_wiki.py log wiki/ "<message>" — append log

External APIs

  • arXiv RSS (via tools/fetch_arxiv.py)
  • DeepXiv API (via tools/fetch_deepxiv.py, optional; graceful fallback when unavailable)

Scheduling

  • Can be scheduled for daily automatic execution via CronCreate:
    CronCreate: schedule "/daily-arxiv" daily at 08:00
    
add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks.Stars: 115kUpdated Claude CodeCodexVerifiedInstall