Lambenthan/empiricalwiki / i18n/en/skills/ingest
ingest
VerifiedIngest a paper into the wiki — creates pages (papers + concepts + people + claims) and builds all cross-references and graph edges. Trigger whenever the user says "ingest", "add this paper", drops a `.pdf` / `.tex` / arXiv URL, or asks to fold a paper into the knowledge base.
/ingest
Turn one paper into a fully wired set of wiki pages. Emit well-formed entities and correct cross-references; leave semantic audits (backlink symmetry, dangling nodes, field-value policing) for /check.
Use these local references on demand:
references/pdf-preprocessing.md— arXiv-ID recovery, tex fetching, prepare-paper handoff for direct PDF dropsreferences/dedup-policy.md— merge-vs-create decision rule for concepts and claims, and the line that separates/ingestshape checks from/checksemantic auditsreferences/cross-references.md— forward/reverse link matrix and paper-to-paper edge-type selectionreferences/init-mode.md— manifest-driven handoff from/initand parallel-safety conventionsreferences/error-handling.md— source parse, API, and slug-collision fallbacks
Open docs/runtime-page-templates.en.md before drafting any wiki page frontmatter or body sections, and docs/runtime-support-files.en.md for index.md, log.md, and graph/ formats.
Inputs
source: one of — arXiv URL (e.g.https://arxiv.org/abs/2106.09685), local.tex, local.pdf, or acanonical_ingest_pathhanded off by/initvia.checkpoints/init-sources.json(seereferences/init-mode.md)--discover(optional, default off): after the final report, invoke/discover --anchor <this-paper's-arxiv-id>and append the shortlist to the report as "Related papers you may want to ingest next". Never auto-ingests the suggestions. Skipped automatically in INIT MODE. Treat this as a user-owned flag: do not set it based on repo state.
Outputs
- One fully-wired paper page plus linked entities (concepts, claims, people)
- Graph edges and citations appended via
tools/research_wiki.py - Terminal summary with page counts and suggested follow-up ingests
Wiki Interaction
Reads
wiki/index.mdfor existing slugs and tagswiki/papers/*.mdto detect an already-ingested paperwiki/concepts/*.mdandwiki/foundations/*.mdfor dedup matcheswiki/claims/*.mdfor dedup matcheswiki/people/*.mdfor existing authorswiki/topics/*.mdto place the paper under existing topicswiki/graph/open_questions.mdto notice when the paper addresses a known gap
Writes
wiki/papers/{slug}.md— CREATEwiki/concepts/{slug}.md— CREATE (new) or EDIT (appendkey_papers, aliases, variants)wiki/claims/{slug}.md— CREATE (new) or EDIT (appendevidenceentry)wiki/people/{slug}.md— CREATE (importance ≥ 4 only) or EDIT (appendKey papers)wiki/topics/{slug}.md— EDIT only (no CREATE from/ingest)wiki/graph/edges.jsonl— APPEND via toolwiki/graph/citations.jsonl— APPEND via toolwiki/graph/context_brief.md— REBUILD (skipped in INIT MODE)wiki/graph/open_questions.md— REBUILD (skipped in INIT MODE)wiki/index.md— APPENDwiki/log.md— APPEND via tool
Graph edges created
paper → concept:introduces_concept/uses_concept/extends_concept/critiques_conceptwithconfidencepaper → foundation:derived_from(foundation is terminal; no reverse link)paper → claim:supports/contradictspaper → paper:same_problem_as/similar_method_to/complementary_to/builds_on/compares_against/improves_on/challenges/surveyswithconfidence- bibliographic
paper → paper:citesingraph/citations.jsonl
tools/research_wiki.py add-edge rejects missing confidence/evidence for
paper-paper and paper-concept semantic edges, and rejects legacy
paper-to-concept or paper-to-paper types on new writes.
Workflow
Pre-condition: working directory contains wiki/, raw/, and tools/. Resolve the Python interpreter once and reuse it:
# Find the project root via git so worktree subagents can still locate .venv.
# .venv is gitignored, so a subagent whose cwd is ../.worktrees/<branch>/
# doesn't have one — without this lookup it falls back to system python3 and
# misses the .env-loaded API keys plus the installed deps (deepxiv-sdk etc.).
# git rev-parse --git-common-dir returns the main repo's .git regardless of
# which worktree the shell is in; its parent is the project root.
GIT_COMMON_DIR=$(git rev-parse --git-common-dir 2>/dev/null || true)
PROJECT_ROOT=""
if [ -n "$GIT_COMMON_DIR" ]; then
PROJECT_ROOT=$(cd "$(dirname "$GIT_COMMON_DIR")" 2>/dev/null && pwd)
fi
if [ -x "$PROJECT_ROOT/.venv/bin/python" ]; then PYTHON_BIN="$PROJECT_ROOT/.venv/bin/python"
elif [ -x "$PROJECT_ROOT/.venv/Scripts/python.exe" ]; then PYTHON_BIN="$PROJECT_ROOT/.venv/Scripts/python.exe"
elif [ -x .venv/bin/python ]; then PYTHON_BIN=.venv/bin/python
elif [ -x .venv/Scripts/python.exe ]; then PYTHON_BIN=.venv/Scripts/python.exe
else PYTHON_BIN=python3
fi
export PYTHON_BIN
Step 1: Resolve the source
- If
/initpassed acanonical_ingest_path, enter INIT MODE and consume that path verbatim. Do not rescanraw/. Seereferences/init-mode.md. - If the source is an arXiv URL, extract the arXiv ID, use
"$PYTHON_BIN" tools/fetch_s2.py paper <arxiv-id>to recover the title when possible, then run"$PYTHON_BIN" tools/init_discovery.py download --raw-root raw --arxiv-id <arxiv-id> --title "<title-or-arxiv-id>". Continue from the returnedcanonical_ingest_path. The helper tries arXiv source first and falls back to PDF; do not callfetch_arxiv.pyfor a single paper because it is RSS-only. - If the source is a local
.tex, use it directly. - If the source is a local
.pdf, run the preprocessing pipeline inreferences/pdf-preprocessing.mdto produce a prepared.texunderraw/tmp/before continuing.
Raw persistence rule: never copy or duplicate a file already under raw/discovered/, raw/tmp/, or raw/papers/ into a different raw subtree.
Step 2: Paper identity and enrichment
-
Generate the paper slug:
"$PYTHON_BIN" tools/research_wiki.py slug "<paper-title>" -
Stop-if-exists: if
wiki/papers/{slug}.mdalready exists and the arXiv ID or title matches, report and exit. If they differ, resolve the collision perreferences/error-handling.md. -
When an arXiv ID is available, query Semantic Scholar:
"$PYTHON_BIN" tools/fetch_s2.py paper <arxiv-id>Use the result for
venue,year,s2_id, citation count, and the evidence behind theimportancescore (1-5). -
Optional DeepXiv enrichment, when available. Skip silently if it fails:
"$PYTHON_BIN" tools/fetch_deepxiv.py brief <arxiv-id> "$PYTHON_BIN" tools/fetch_deepxiv.py head <arxiv-id> "$PYTHON_BIN" tools/fetch_deepxiv.py social <arxiv-id>briefseeds the Key-idea section;headsanity-checks your tex parsing against the section structure;socialis an auxiliary importance signal.
Step 3: Write the paper page
Open docs/runtime-page-templates.en.md for the paper template. Fill every required frontmatter field; leave cited_by empty for now (step 5 backfills it).
Before writing, run a shape check on the frontmatter you are about to emit — no more than this:
- every required key is present and non-empty
importance∈ {1,2,3,4,5};statuson claims ∈ the documented set;maturityon concepts ∈ the documented set; claimconfidence∈ [0,1]- YAML parses
The shape check is intentionally narrow. Backlink symmetry, dangling-node detection, and cross-entity consistency are /check's job, not this skill's.
Body sections to populate: Problem, Key idea, Method, Results, Limitations, Open questions, My take, Related.
Step 4: Concepts, claims, people
Follow references/dedup-policy.md. In short:
- For each candidate concept or claim, call the matching
find-similar-*tool first. - Prefer merging into the top result. Create a new page only when the tool returns no acceptable candidate and the paper's importance justifies it.
- For each entity you write or edit, write the reverse link in the same turn. The obligation matrix lives in
references/cross-references.md. - Create a
wiki/people/{slug}.mdonly for papers with importance ≥ 4. Otherwise append to existing author pages only.
Step 5: Paper-to-paper edges and cited_by
Skip this whole step in INIT MODE — the parent /init handles it at fan-in.
"$PYTHON_BIN" tools/fetch_s2.py references <arxiv-id>
"$PYTHON_BIN" tools/fetch_s2.py citations <arxiv-id>
- For each reference whose arXiv ID or title resolves to an existing
wiki/papers/{slug}.md, add a bibliographiccitesrow tograph/citations.jsonl. - Add a semantic paper-to-paper edge in
graph/edges.jsonlonly when the source text gives a clear cue. Edge-type selection is inreferences/cross-references.md. If no semantic relation cleanly fits, keep only thecitesrow. - For each citation already in the wiki, append the citer's slug to this paper's
cited_by. - Surface unmatched high-citation references in the final report so the user can decide whether to follow up with another
/ingest.
Step 6: Topics and index
- Match the paper's domain and tags against existing
wiki/topics/*.md. For each match:- importance ≥ 4 → append to the topic's
## Seminal works - importance < 4 → append under
## SOTA trackeror## Recent workby year - if the paper directly addresses a listed open problem, annotate that line on the topic page
- importance ≥ 4 → append to the topic's
- Do not create new topic pages from
/ingest— topic creation belongs to/initand/edit. - Append new or edited page entries to
wiki/index.mdunder their category headings. Seedocs/runtime-support-files.en.mdfor the exact format.
Step 7: Log and rebuild
"$PYTHON_BIN" tools/research_wiki.py log wiki/ "ingest | added papers/<slug> | updated: <list>"
Unless in INIT MODE:
"$PYTHON_BIN" tools/research_wiki.py rebuild-context-brief wiki/
"$PYTHON_BIN" tools/research_wiki.py rebuild-open-questions wiki/
Step 8: Report
Emit one compact summary covering: pages created, pages updated, graph edges added, contradictions surfaced (if any), and high-citation references not yet in the wiki (suggested follow-up ingests). Close with:
Wiki: +1 paper, +{N} claims, +{M} concepts, +{K} edges
Step 9: Optional discovery (only if --discover is set)
Skip this step unless the user explicitly passed --discover. Also skip it in INIT MODE — /init's parent process decides whether to run discovery at fan-in, not individual subagents.
When active, invoke /discover with the just-ingested paper as the single anchor:
"$PYTHON_BIN" tools/discover.py from-anchors \
--id <arxiv-id-of-this-paper> \
--wiki-root wiki \
--limit 10 \
--output-checkpoint .checkpoints/ \
--markdown
Append the markdown output to the report under a heading like "Related papers you may want to ingest next". Do not auto-ingest anything from the shortlist — the user picks. If discovery fails (S2 outage, all channels empty), note the failure in one line and continue — a failed /discover must not fail an otherwise successful /ingest.
Constraints
raw/papers/,raw/notes/,raw/web/are user-owned and read-only. Direct local/ingestmay add prepared sidecars underraw/tmp/; direct arXiv ingests may write fetched source artifacts underraw/discovered/. INIT MODE treats all ofraw/as read-only.wiki/graph/is tool-owned. Edit only throughtools/research_wiki.py.- Slugs always come from
tools/research_wiki.py slug. Never hand-craft. - Every forward link writes its reverse link in the same turn — the wiki's bidirectional-link invariant. The only exception is links to
wiki/foundations/, which are terminal. - In INIT MODE, do not write reverse links into pages that already exist (created by a sibling worktree or scaffold). Record the relationship via
tools/research_wiki.py add-edgeonly; the parent/initbackfills reverse links during fan-in. - Source priority:
.tex>.pdf> vision API fallback. Never ingest from a PDF when a usable.texis available. - Ingest is conservative about new entities:
- importance < 4: at most 1 new concept and 1 new claim per paper
- importance ≥ 4: at most 3 new concepts and 2 new claims per paper
- Any further candidates must be merged into their nearest
find-similar-*result, or left out for/checkto flag. Rationale and matching rules:references/dedup-policy.md.
/ingestruns a shape check on its own output (required keys, enum ranges, YAML parses) and stops there. Backlink symmetry, dangling nodes, and full semantic audits belong to/check. Do not re-implement them here.- Assume another
/ingestmay run concurrently in a sibling worktree. All shared-file writes (graph/edges.jsonl,graph/citations.jsonl,index.md,log.md) must go throughtools/research_wiki.pyor use append-only semantics. Seereferences/init-mode.md. - In INIT MODE, skip
fetch_s2.py citations,fetch_s2.py references, and therebuild-*commands — the parent/initruns them once after fan-in.
Error Handling
See references/error-handling.md. Highlights: source parse failures cascade tex → PDF → vision API → user handoff; S2 outages default importance to 3 and skip citation backfill; DeepXiv outages skip enrichment silently; slug collisions append a numeric suffix.
Dependencies
Tools (via Bash)
"$PYTHON_BIN" tools/research_wiki.py slug "<title>""$PYTHON_BIN" tools/research_wiki.py find-similar-concept wiki/ "<title>" --aliases "<a,b,c>""$PYTHON_BIN" tools/research_wiki.py find-similar-claim wiki/ "<title>" --tags "<a,b,c>""$PYTHON_BIN" tools/research_wiki.py add-edge wiki/ --from <id> --to <id> --type <type> --evidence "<text>" [--confidence high|medium|low]--confidence high|medium|lowis required for paper-paper and paper-concept semantic edges.
"$PYTHON_BIN" tools/research_wiki.py add-citation wiki/ --from papers/<citing> --to papers/<cited> --source semantic_scholar"$PYTHON_BIN" tools/research_wiki.py log wiki/ "<message>""$PYTHON_BIN" tools/research_wiki.py rebuild-context-brief wiki/"$PYTHON_BIN" tools/research_wiki.py rebuild-open-questions wiki/"$PYTHON_BIN" tools/prepare_paper_source.py --raw-root raw --source <local-path> [--title "<recovered-title>"] [--arxiv-id "<recovered-arxiv-id>"]"$PYTHON_BIN" tools/init_discovery.py download --raw-root raw --arxiv-id <id> --title "<title-or-id>"— single-paper arXiv source/PDF download intoraw/discovered/"$PYTHON_BIN" tools/fetch_s2.py paper|citations|references <arxiv-id>"$PYTHON_BIN" tools/fetch_deepxiv.py brief|head|social <arxiv-id>"$PYTHON_BIN" tools/discover.py from-anchors --id <arxiv-id> --wiki-root wiki --limit 10 --output-checkpoint .checkpoints/ --markdown— only when--discoveris set
Shared References
.claude/skills/shared-references/citation-verification.md
Skills
/init— calls/ingestin parallel subagents via INIT MODE/check— audits wiki state after/ingestcompletes; owns every semantic check/ingestintentionally does not perform/discover— optional follow-up when--discoveris set; produces a shortlist of related papers the user may want to ingest next
External APIs
- Semantic Scholar (via
tools/fetch_s2.py) - DeepXiv (via
tools/fetch_deepxiv.py, optional; graceful fallback) - arXiv (source download)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| canvas-designanthropics/skillsCreate beautiful visual art in .png and .pdf documents using design philosophy. You should use this skill when the user asks to create a poster, piece of art, design, or other static piece. Create original visual designs, never copying existing artists' work to avoid copyright violations. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| docxanthropics/skillsUse this skill whenever the user wants to create, read, edit, or manipulate Word documents (.docx files) or Word templates (.dotx files). Triggers include: any mention of 'Word doc', 'word document', '.docx', '.dotx', or requests to produce professional documents with formatting like tables of contents, headings, page numbers, or letterheads. Also use when extracting or reorganizing content from .docx or .dotx files, inserting or replacing images in documents, performing find-and-replace in Word files, working with tracked changes or comments, or converting content into a polished Word document. If the user asks for a 'report', 'memo', 'letter', 'template', or similar deliverable as a Word or .docx file, use this skill. Do NOT use for PDFs, spreadsheets, Google Docs, or general coding tasks unrelated to document generation. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| pdfanthropics/skillsUse this skill whenever the user wants to do anything with PDF files. This includes reading or extracting text/tables from PDFs, combining or merging multiple PDFs into one, splitting PDFs apart, rotating pages, adding watermarks, creating new PDFs, filling PDF forms, encrypting/decrypting PDFs, extracting images, and OCR on scanned PDFs to make them searchable. If the user mentions a .pdf file or asks to produce one, use this skill. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| pptxanthropics/skillsUse this skill any time a .pptx or .potx file is involved in any way — as input, output, or both. This includes: creating slide decks, pitch decks, or presentations; reading, parsing, or extracting text from any .pptx or .potx file (even if the extracted content will be used elsewhere, like in an email or summary); editing, modifying, or updating existing presentations; combining or splitting slide files; working with templates (.potx), layouts, speaker notes, or comments. Trigger whenever the user mentions "deck," "slides," "presentation," or references a .pptx or .potx filename, regardless of what they plan to do with the content afterward. If a .pptx or .potx file needs to be opened, created, or touched, use this skill. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |