Lambenthan/empiricalwiki / i18n/en/skills/init
init
VerifiedBootstrap ΩmegaWiki from user sources plus optional discovery, then ingest the final paper set in parallel
/init
Build a wiki from
raw/with deterministic source preparation, planner-guided discovery, provisional notes/web scaffolding, and parallel/ingestfan-out/fan-in.
Use these local references on demand:
references/prepare-and-discovery.md— prepare flow, final selection, fetch, and source-manifest rulesreferences/planner-policy.md— planner behavior and LLM trim expectationsreferences/parallel-ingest.md— worktree isolation, subagent prompt contract, merge, and cleanup
Inputs
topic(optional): research direction keywords; omit whenraw/already defines the seed set--no-introduction(optional): disable external discovery; use only when the user explicitly requests it- User-owned sources under
raw/papers/,raw/notes/,raw/web/
Outputs
wiki/scaffold and provisional pages (Summary, topics, ideas, concepts, claims)raw/tmp/andraw/discovered/prepared sources- Final paper pages via parallel
/ingestsubagents .checkpoints/init-*.jsonmanifests for resume and replay- Updated
wiki/index.md,wiki/log.md,wiki/graph/*
Wiki Interaction
Reads
raw/papers/,raw/notes/,raw/web/.checkpoints/init-prepare.jsonand.checkpoints/init-sources.jsonfor resume, planning, and fan-outwiki/index.mdplus existingwiki/topics/,wiki/ideas/,wiki/concepts/,wiki/claims/for duplicate avoidance and scaffold alignment
Writes
wiki/scaffold and provisional pagesraw/tmp/andraw/discovered/wiki/index.md,wiki/log.md,wiki/graph/*.checkpoints/init-prepare.json,.checkpoints/init-plan.json,.checkpoints/init-sources.json, andinit-sessioncheckpoint metadata
Graph edges created
/inititself creates only scaffold-level edges when provisional pages need them- all paper-driven edges are delegated to
/ingest
Workflow
Pre-condition: working directory is the project root containing wiki/, raw/, and tools/. Set WIKI_ROOT=wiki/. Resolve PYTHON_BIN once and reuse it for every Python command during /init so the workflow stays on the interpreter that setup.sh prepared:
# Find the project root via git so worktree subagents can still locate .venv.
# .venv is gitignored, so a subagent whose cwd is ../.worktrees/<branch>/
# doesn't have one — without this lookup it falls back to system python3 and
# misses the .env-loaded API keys plus the installed deps (deepxiv-sdk etc.).
# git rev-parse --git-common-dir returns the main repo's .git regardless of
# which worktree the shell is in; its parent is the project root.
GIT_COMMON_DIR=$(git rev-parse --git-common-dir 2>/dev/null || true)
PROJECT_ROOT=""
if [ -n "$GIT_COMMON_DIR" ]; then
PROJECT_ROOT=$(cd "$(dirname "$GIT_COMMON_DIR")" 2>/dev/null && pwd)
fi
if [ -x "$PROJECT_ROOT/.venv/bin/python" ]; then PYTHON_BIN="$PROJECT_ROOT/.venv/bin/python"
elif [ -x "$PROJECT_ROOT/.venv/Scripts/python.exe" ]; then PYTHON_BIN="$PROJECT_ROOT/.venv/Scripts/python.exe"
elif [ -x .venv/bin/python ]; then PYTHON_BIN=.venv/bin/python
elif [ -x .venv/Scripts/python.exe ]; then PYTHON_BIN=.venv/Scripts/python.exe
else PYTHON_BIN=python3
fi
export PYTHON_BIN
Step 1: Initialize wiki structure
"$PYTHON_BIN" tools/research_wiki.py init wiki/
Create the standard wiki directories, graph/, outputs/, index.md, and log.md. Do not add a second init log entry here.
Step 2: Prepare local inputs into raw/tmp/
"$PYTHON_BIN" tools/init_discovery.py prepare --raw-root raw --pdf-titles-json .checkpoints/init-pdf-titles.json --output-manifest .checkpoints/init-prepare.json
- before running
prepare, inspect each local PDF and write the recovery handoff to.checkpoints/init-pdf-titles.jsonas either{ "raw/papers/foo.pdf": "Recovered Paper Title" }or{ "raw/papers/foo.pdf": { "title": "Recovered Paper Title", "arxiv_id": "2401.00001" } }when a confident arXiv ID is already known - use
"$PYTHON_BIN" tools/prepare_paper_source.py --raw-root raw --source <local-path> [--title "<recovered-title>"] [--arxiv-id "<recovered-arxiv-id>"]for local paper normalization - local PDF recovery order is strict: handed-off arXiv ID or filename/path arXiv ID -> agent-recovered title via Semantic Scholar -> fetched arXiv source -> synthetic
.tex - when the agent supplied a PDF title, treat that title as authoritative for the prepared manifest; fetched/source titles are sanitized fallback metadata only and must not overwrite it
- do not use PDF metadata or body text as arXiv-ID hints during prepare
- metadata or filename titles may remain as provisional display labels only; they are not trusted identity or title-search inputs
- keep notes/web on their original source paths;
/initreads them directly during planning - set each local paper's
canonical_ingest_pathto a preparedraw/tmp/path when available; otherwise fall back to the originalraw/papers/...path - record warnings for failed decode / title recovery / arXiv source fetch rather than aborting
/init - see
references/prepare-and-discovery.mdfor the prepare decision tree and source-preference rules
Step 3: Plan discovery, trim the final set, and write the source manifest
"$PYTHON_BIN" tools/init_discovery.py plan [--topic "<topic>"] --mode auto --raw-root raw --wiki-root wiki --prepared-manifest .checkpoints/init-prepare.json --allow-introduction <true|false> --output-plan .checkpoints/init-plan.json
mode=seededwhen the prepare manifest contains at least one parseable local paper; otherwisemode=bootstrapplanmust read.checkpoints/init-prepare.jsoninstead of rescanningraw/- planner policy is qualitative at the skill layer: favor relevance, freshness, connectivity, and survey coverage
- in seeded mode with limited introduced capacity, avoid over-prioritizing older citation-heavy anchors
- in bootstrap mode, one older canonical anchor may be useful when it improves coverage
- when DeepXiv search is available, use returned
relevance_scorein tool scoring rather than merely noting it in prose - exact ranking weights, shortlist constants, and threshold math belong to
tools/init_discovery.py; treat the tool as the implementation authority and do not restate or override its constants in LLM reasoning - read
.checkpoints/init-plan.jsonand explicitly trim the over-pickedshortlistto a final 8-10 papers total beforefetch - emit an explicit final selection artifact before
fetchwithshortlist_count,final_count, and the exact finalcandidate_idlist in shortlist order - if
final_countfalls outside 8-10, stop and revise the final selection beforefetch, unless--no-introductionis active or the user already supplied more than 10 parseable papers - if
--no-introductionis present, only use this branch when the user explicitly requested local-only behavior; still runfetchwith zero external IDs so it writes.checkpoints/init-sources.json - see
references/planner-policy.mdfor planner behavior, trim expectations, and source-of-truth boundaries
Then run:
"$PYTHON_BIN" tools/init_discovery.py fetch --raw-root raw --plan-json .checkpoints/init-plan.json --prepared-manifest .checkpoints/init-prepare.json --output-sources .checkpoints/init-sources.json --id <candidate-id> --id <candidate-id>
- external papers downloaded by
/initgo toraw/discovered/, neverraw/papers/ - never fetch a paper that is already represented by a prepared local source from
raw/tmp/ .checkpoints/init-sources.jsonis the single source of truth for downstream ingest order
Step 4: Create scaffold pages before paper ingest
Create one wiki/Summary/{area}.md, the needed wiki/topics/{slug}.md, and provisional ideas/, concepts/, and claims/ from notes/web when warranted.
Rules:
- notes/web are authoritative for user intent, not for literature confidence
- every notes/web-derived page must include this exact line immediately after frontmatter:
Provisional note: seeded from raw/notes or raw/web during /init; pending validation from ingested papers.
topics/: create when a direction is explicit or repeatedideas/: create when the user states or strongly implies a research direction or hypothesisconcepts/: create only when the mechanism recurs across notes/web, or appears once in notes/web and once in the final paper setclaims/: create only from explicit assertive statements, never by inference- for notes/web-derived claims, use
status: proposed,confidence: 0.2,source_papers: [], andevidence: [] /prefillis optional background seeding and is not part of/init/initmust not createpeople/pages directly and must not auto-create foundations
Step 5: Parallel paper ingest with worktree isolation
Paper sources for this step come strictly from .checkpoints/init-sources.json:
origin=user_local: canonical prepared.texunderraw/tmp/when available, otherwise fallbackraw/papers/...origin=introduced: fetched dirs or PDFs underraw/discovered/
Parallel ingest contract:
- stash unrelated dirty files before fan-out, then record
stash_ref,base_branch, andbase_commitin checkpoint metadata - commit the freshly created scaffold and init manifests before fan-out so
BASE_COMMITactually contains the pages, manifests, and handoff metadata that subagents must branch from - verify
.gitattributescontainsmerge=unionforwiki/log.md,wiki/graph/edges.jsonl,wiki/graph/citations.jsonl, andwiki/index.mdbefore creating worktrees /initworktree mode must run from a named branch, not detached HEAD- create each worktree from
BASE_COMMIT, not from the already checked-outBASE_BRANCH - subagent prompts must use relative paths only, and the subagent's shell working directory must be the worktree path (
$WT_PATH), not the main repository root - execute
/ingestfor exactly one handed-off source path; do not bypass/ingest - in INIT MODE, consume the handed-off canonical path exactly as provided
- skip
fetch_s2.py citations - skip
fetch_s2.py references - skip per-subagent
rebuild-index - skip per-subagent
rebuild-context-brief - skip per-subagent
rebuild-open-questions - skip conflict-prone topic writes
- commit the ingest result inside the worktree before exiting so fan-in merges a real paper-specific commit instead of an empty branch
- see
references/parallel-ingest.mdfor worktree commands, merge order, fan-in, and cleanup
Step 6: Fan-in, rebuild, and final report
After all subagents complete:
- merge worktree branches sequentially on
BASE_BRANCH - resolve true concept / claim conflicts conservatively: merge, do not multiply near-duplicates
- run:
"$PYTHON_BIN" tools/research_wiki.py dedup-edges wiki/
"$PYTHON_BIN" tools/research_wiki.py dedup-citations wiki/
"$PYTHON_BIN" tools/research_wiki.py rebuild-index wiki/
"$PYTHON_BIN" tools/research_wiki.py rebuild-context-brief wiki/
"$PYTHON_BIN" tools/research_wiki.py rebuild-open-questions wiki/
"$PYTHON_BIN" tools/lint.py --wiki-dir wiki/ --fix
Report separately:
- user-provided papers ingested through prepared
raw/tmp/paths - user-provided papers that fell back to original
raw/papers/paths - discovered papers from
raw/discovered/ - provisional pages seeded from notes/web
- pages created by
/ingest - pages updated by
/ingest - any skipped or failed papers
If stash_ref exists, pop it at the end. If stash pop fails, keep the checkpoint and report the failure.
Constraints
- Do not infer
--no-introductionfrom repository state alone. Use it only when the user explicitly asked to disable external discovery. raw/papers/,raw/notes/, andraw/web/are user-owned inputsraw/tmp/andraw/discovered/are generated handoff areas; direct local/ingestmay also prepare reusable local sidecars underraw/tmp//initmay write external papers only toraw/discovered/;/initand direct local/ingestmay write generated prepared local sources toraw/tmp//prefillis optional background seeding, not part of/init- no skill other than
/prefillmay auto-create foundations /initmust not createpeople/pages directly- notes/web-derived pages are provisional and must carry the exact notice line above
- paper evidence outranks notes/web for claim confidence and concept consolidation
- all paper ingest must run through parallel
/ingestsubagents with worktree isolation - Step 5 must read paper inputs from
.checkpoints/init-sources.json, not by ad hoc folder scanning - exact deterministic planner policy belongs in
tools/init_discovery.py, not in duplicated skill constants
Error Handling
- No parseable paper in
raw/papers/: enter bootstrap mode raw/notes/andraw/web/empty: skip provisional seeding, continue- PDF decode fails during prepare: keep the local source, record the warning in
.checkpoints/init-prepare.json, and fall back to the original path if needed - No confident PDF title is recovered: omit
--title, allow filename/path arXiv-ID recovery only, then fall back directly to synthetic.tex; any metadata-or-filename title is display-only - Chinese content is detected in
raw/notes/orraw/web/: keep going, but preserve a planner warning that note/web extraction and ranking may be less reliable and treat rankings plus provisional pages as lower-confidence - S2 or DeepXiv unavailable: planner falls back to the remaining sources; preserve the warning in the checkpointed plan and note degraded discovery in the report
- External fetch fails for one paper: keep the remaining final set and report the failed download
- Single paper ingest fails: record it via checkpoint, skip it, continue the rest, and list it in the report
- Current checkout is detached HEAD: stop before worktree fan-out and ask the user to switch to or create a named branch first
- stash pop fails: keep checkpoint metadata and report the manual recovery step
Dependencies
Tools (via Bash)
"$PYTHON_BIN" tools/research_wiki.py init wiki/"$PYTHON_BIN" tools/research_wiki.py checkpoint-set-meta wiki/ init-session <key> <value>"$PYTHON_BIN" tools/research_wiki.py checkpoint-save/load/clear wiki/ init-session ..."$PYTHON_BIN" tools/research_wiki.py dedup-edges wiki/"$PYTHON_BIN" tools/research_wiki.py dedup-citations wiki/"$PYTHON_BIN" tools/research_wiki.py rebuild-index wiki/"$PYTHON_BIN" tools/research_wiki.py rebuild-context-brief wiki/"$PYTHON_BIN" tools/research_wiki.py rebuild-open-questions wiki/"$PYTHON_BIN" tools/research_wiki.py log wiki/ "<message>""$PYTHON_BIN" tools/prepare_paper_source.py --raw-root raw --source <local-path> [--title "<recovered-title>"]"$PYTHON_BIN" tools/init_discovery.py prepare --raw-root raw --pdf-titles-json .checkpoints/init-pdf-titles.json --output-manifest .checkpoints/init-prepare.json"$PYTHON_BIN" tools/init_discovery.py plan [--topic "<topic>"] --mode auto --raw-root raw --wiki-root wiki --prepared-manifest .checkpoints/init-prepare.json --allow-introduction <true|false> --output-plan .checkpoints/init-plan.json"$PYTHON_BIN" tools/init_discovery.py fetch --raw-root raw --plan-json .checkpoints/init-plan.json --prepared-manifest .checkpoints/init-prepare.json --output-sources .checkpoints/init-sources.json --id <candidate-id>"$PYTHON_BIN" tools/lint.py --wiki-dir wiki/ --fix
Skills
/ingest— one paper per subagent, in INIT MODE
External APIs used by init_discovery.py
- Semantic Scholar
- DeepXiv (optional)
- arXiv download endpoints
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |