po4yka/llm-wiki-skills / skills/llm-wiki-ecosystem-registry
llm-wiki-ecosystem-registry
VerifiedBuild or refresh a current registry of LLM-Wiki implementations and adjacent open-source frameworks. Use when the user asks what projects exist, which technologies are missing, how OpenWiki/nashsu/vouch/RepoAgent compare, or which ready-made implementation to evaluate.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 5.7 KB |
| references/docs/13-ecosystem-matrix.md | markdown | 13.4 KB |
| references/docs/14-technology-stack.md | markdown | 20.7 KB |
| references/docs/15-implementation-deep-dive.md | markdown | 24.6 KB |
| references/docs/16-retrieval-architecture.md | markdown | 22.4 KB |
| references/docs/17-mcp-api-integration.md | markdown | 19.4 KB |
| references/docs/18-evaluation-methodology.md | markdown | 18.8 KB |
| references/docs/19-security-threat-model.md | markdown | 23.0 KB |
| references/docs/20-ingestion-pipelines.md | markdown | 18.4 KB |
| references/docs/21-publishing-export.md | markdown | 14.6 KB |
| references/templates/source-manifest.yaml | config | 2.2 KB |
references/docs/20-ingestion-pipelines.md
View on GitHub# Ingestion pipelines for LLM-Wiki
> Status: draft
> Current as of: 2026-07-06
> Scope: capture, parsing, normalization, enrichment, chunking, provenance, review and compilation into durable LLM-Wiki artifacts.
## How to use this document
Use this document when designing, reviewing or upgrading an LLM-Wiki ingestion pipeline.
This note is an architecture guide, not a permanent tool benchmark. Before recommending concrete commands or supported-format claims, re-check official upstream docs for current parser support, license, cloud/local behavior, security notes, API changes and version compatibility.
Related skills and docs:
- `skills/llm-wiki-ingestion-stack/SKILL.md`
- `skills/wiki-ingest/SKILL.md`
- `skills/llm-wiki-capture-pipeline/SKILL.md`
- `skills/llm-wiki-security-review/SKILL.md`
- `skills/llm-wiki-retrieval-architect/SKILL.md`
- `docs/16-retrieval-architecture.md`
- `docs/19-security-threat-model.md`
## Executive summary
The safest LLM-Wiki ingestion architecture is **structure-first and provenance-preserving**:
```text
capture -> quarantine -> manifest -> convert/OCR/ASR -> normalize -> enrich -> chunk -> QA -> source page -> wiki draft -> review -> index
```
The main failure mode is not bad embeddings. It is bad conversion:
- dropped tables;
- broken reading order;
- missing page, timestamp, row or message anchors;
- lost attachments;
- flattened code or schema structure;
- missing sensitivity metadata;
- untrusted source instructions leaking into agent behavior.
The default stack by source class:
| Source class | Default path |
| --- | --- |
| Native PDF/Office/HTML | Docling for structure/provenance; MarkItDown for lightweight Markdown conversion. |
| Scanned PDFs/images | OCRmyPDF first; Docling/PaddleOCR/Marker for downstream structure. |
| Web pages | Playwright render capture + Readability extraction; ArchiveBox/SingleFile for durable snapshots. |
| Audio/video | yt-dlp where lawful + Whisper/faster-whisper; keep timestamped segments. |
| Code repositories | tree-sitter/ast-grep for AST-aware chunks; OpenWiki/RepoAgent-style docs for higher-level maps. |
| Emails/chats | mail-parser/Notmuch/official exports; preserve thread/message/attachment structure. |
| Tables/databases | dlt/Airbyte/DuckDB/pandas; preserve schema, query, row-group and source lineage. |
## Source taxonomy
| Source type | Canonical capture unit | Primary risks | Best default representation |
| --- | --- | --- | --- |
| PDF / Office / HTML documents | file or URL | reading-order loss, table loss, layout flattening | structured JSON + Markdown + page anchors |
| Scanned documents | scanned PDF or image | OCR errors, page skew, missing text layer | OCR'd PDF + structured JSON + confidence metadata |
| Web pages | rendered page snapshot | JS-rendered content loss, link rot, boilerplate | archived HTML/PDF/PNG + extracted article text |
| Audio / video | media file or stream | transcription drift, timestamp drift, speaker loss | transcript segments with timestamps |
| Code repositories | repo / commit / file tree | syntactic flattening, stale docs, missing symbols | AST-aware chunks + repo metadata |
| Chats / emails | thread / message / attachment | thread breakup, attachment detachment, privacy risk | thread-aware JSON + message-level metadata |
| Databases / tables | table / query / partition | schema drift, row-level provenance loss | Parquet/JSON + schema + query lineage |
| Images / figures | image / figure / page region | no surrounding context, OCR/table loss | extracted text/caption + bounding boxes + source anchor |
## Reference architecture
```mermaid
flowchart LR
A[Capture source] --> B[Quarantine / staging]
B --> C[Source manifest]
C --> D[Parser / OCR / ASR worker]
D --> E[Normalized structured output]
E --> F[Security and privacy scan]
F --> G[Enrichment and metadata]
G --> H[Structure-aware chunking]
H --> I[Ingestion QA]
I --> J[wiki/sources page]
J --> K[Entity/concept/synthesis draft]
K --> L[Review gate]
L --> M[Indexes: lexical, vector, graph]
```
Durable storage layout:
```text
raw/
sources/<source_id>/original.ext
manifests/<source_id>.yaml
extracted/<source_id>/docling.json
extracted/<source_id>/content.md
extracted/<source_id>/chunks.jsonl
wiki/
sources/<source_id>.md
entities/
concepts/
synthesis/
indexes/
fts/
vectors/
graph/
evals/
ingestion-golden-corpus.yaml
ingestion-fidelity-report.json
```
Rules:
- Raw sources are preserved unless retention policy explicitly says otherwise.
- Manifests are created before synthesis.
- Extraction artifacts are reproducible from raw source + manifest + parser version.
- Production indexes use reviewed or policy-approved content by default.
- Untrusted content remains data; it must not modify agent or tool instructions.
## Tool selection matrix
| Need | First choice | Upgrade / alternate | Main caution |
| --- | --- | --- | --- |
| Broad high-fidelity docs | Docling | Unstructured, Apache Tika, Marker | Verify current supported formats and layout fidelity. |
| Lightweight Markdown | MarkItDown | Pandoc | Not enough for high-fidelity tables/layout by itself. |
| Scanned PDF OCR | OCRmyPDF | Tesseract, PaddleOCR | OCR quality must be sampled. |
| Complex tables/forms/charts | PaddleOCR, Marker | Docling table structure | Licensing/version compatibility can matter. |
| Scientific PDFs/math | Nougat | Marker/Docling with formula extraction | Domain-specialized; do not assume general document fidelity. |
| Rendered web pages | Playwright | Browser-use/harnesses where appropriate | Must archive snapshots for provenance. |
| Web article extraction | Readability/ReadabiliPy | Custom DOM extraction | Weak on apps/dashboards. |
| Durable web archive | ArchiveBox, SingleFile | WARC tooling | More operational complexity; license review for SingleFile. |
| Audio/video | Whisper, faster-whisper | Cloud ASR with policy approval | Keep timestamp provenance and review transcription quality. |
| Video acquisition | yt-dlp | platform export/API where available | Use only where lawful and policy-compliant. |
| Code structure | tree-sitter, ast-grep | language servers, RepoAgent/OpenWiki | Preserve commit/path/symbol metadata. |
| Email | mail-parser, Notmuch | platform export APIs | Preserve thread and attachment links. |
| Chat exports | official export format | custom connector/API | Plan for sensitivity and workspace permissions. |
| Tables/DBs | DuckDB/pandas for local; dlt/Airbyte for pipelines | warehouse-native exports | Preserve schema, query and row-group provenance. |
| PII/private-data redaction | Presidio/scrubadub/custom recognizers | provider-native DLP with policy approval | Redact before external processing or embedding. |
| Secret scanning | gitleaks, detect-secrets, trufflehog | platform secret scanning | Scan raw, wiki, exports and configs. |
| Policy enforcement | OPA/custom policy | app-native RBAC | Enforce before retrieval, not after generation. |
## Pipeline archetypes
### 1. Local-first office/PDF knowledge base
Use for internal docs, reports, slide decks and policy PDFs.
```text
filesystem capture -> Docling/MarkItDown -> optional OCRmyPDF -> heading/page chunks -> local FTS/vector index
```
Recommended defaults:
- Parser: Docling for fidelity; MarkItDown for simple Markdown-first conversion.
- OCR: OCRmyPDF only when scans are present.
- Chunking: heading-aware, page-aware, 300-700 token target.
- Retrieval: SQLite FTS5 + LanceDB or local pgvector.
- Review: source pages first, synthesis later.
### 2. Scanned archive / records pipeline
Use for image-only PDFs, scanned records and paper archives.
```text
scanned PDF -> OCRmyPDF -> Docling/PaddleOCR/Marker -> page/region chunks -> QA -> reviewed source page
```
Required metadata:
- `ocr_engine`;
- `ocr_languages`;
- `ocr_confidence` where available;
- page number;
- bounding box if available;
- scan quality flags;
- review required.
Quality checks:
- empty-page detection;
- OCR gibberish/low confidence;
- page rotation/skew;
- table loss;
- duplicate pages;
- missing page anchors.
### 3. Web capture and normalization
Use for product docs, help centers, research pages and changing websites.
```text
URL list -> Playwright render -> ArchiveBox/SingleFile snapshot -> Readability extraction -> chunk by DOM/heading -> cite archive
```
Required metadata:
- canonical URL;
- fetched URL;
- fetch timestamp;
- HTTP status where available;
- title;
- snapshot path/hash;
- extractor version;
- robots/legal review flag when crawling.
Rule: cite the archived artifact or snapshot ID, not only the normalized text.
### 4. Media and meeting ingestion
Use for audio/video, meeting archives, trainings and webinars.
```text
media capture -> ASR -> timestamp segments -> speaker/section metadata -> source page -> synthesis draft
```
Recommended defaults:
- Acquisition: yt-dlp only where lawful or platform export where available.
- Transcription: Whisper or faster-whisper for local ASR.
- Chunking: by segment/time window, optionally topic boundary.
- Provenance: timestamp ranges must survive into citations.
- Review: sample transcripts before using as high-trust evidence.
### 5. Code + structured data hybrid pipeline
Use for engineering wikis, repo docs, data catalogs and lineage-heavy product knowledge.
```text
repo clone + data export -> tree-sitter/ast-grep + OpenWiki/RepoAgent maps + DuckDB/dlt/Airbyte -> symbol/schema chunks -> wiki docs
```
Required metadata:
- repository;
- branch;
- commit SHA;
- file path;
- symbol path;
- language;
- table name/schema;
- query/source connector;
- row group or partition where applicable.
Rule: do not chunk code by arbitrary token windows when AST/symbol boundaries are available.
### 6. Communication ingestion
Use for emails, Slack/Telegram/Discord exports, issue comments and meeting notes.
```text
platform export -> thread/message parser -> attachment extraction -> redaction/classification -> message/thread chunks -> source page
```
Required metadata:
- workspace/project;
- channel/thread/message IDs;
- author role if policy permits;
- timestamp;
- attachment IDs;
- privacy/sensitivity class;
- export scope and approval.
Rule: preserve thread/message/attachment relationships; do not flatten into one blob.
## Source manifest schema
Every source must have a manifest before generated wiki pages are trusted.
The canonical template is [`../templates/source-manifest.yaml`](../templates/source-manifest.yaml). Do not maintain partial schema copies in docs or skills.
Key field boundaries:
- `captured_by` is the actor class: `human|agent|connector|importer`.
- `capture_method` is the mechanism: `manual|watch-folder|api|export|crawler|repo-scan`.
- Channel names belong in channel-specific metadata, source identifiers, or capture pipeline envelopes, not in `captured_by`.
## Chunk schema
Chunks should be retrieval units, not the source of truth.
```yaml
chunk_id: ""
source_id: ""
source_type: ""
page_path: "wiki/sources/<source_id>.md"
source_anchor:
page_no: null
slide_no: null
time_start_ms: null
time_end_ms: null
message_id: null
thread_id: null
row_group: null
symbol_path: null
bbox: null
section_path: []
text: ""
token_count: 0
hash_sha256: ""
created_by: parser|chunker|agent|human
review_state: draft|reviewed|verified|rejected|quarantined
sensitivity: public|internal|sensitive|regulated|unknown
acl_tags: []
```
## Quality gates
Evaluate conversion, provenance and retrieval separately.
| Gate | Purpose | Example threshold |
| --- | --- | --- |
| Parser success rate | Detect broken converters. | No parser crash on golden PR set. |
| Text coverage | Detect empty or truncated extraction. | >= 0.98 for native text docs. |
| Heading preservation | Detect structure loss. | >= 0.95 on golden docs. |
| Table preservation | Detect table flattening. | >= 0.90 where tables matter. |
| Timestamp coverage | Detect ASR/media provenance loss. | >= 0.99 of transcript segments have timestamps. |
| Thread integrity | Detect chat/email flattening. | 1.0 for required test fixtures. |
| Provenance coverage | Ensure source-to-chunk traceability. | >= 0.99 for all production chunks. |
| Duplicate chunk rate | Detect repeated extraction noise. | <= 0.02. |
| Sensitive-data scan | Prevent unsafe indexing/export. | 0 unreviewed high-risk findings. |
| Retrieval smoke | Ensure new ingestion is findable. | Recall@10 does not regress. |
## Golden corpus strategy
Keep two corpora:
| Corpus | Size | Run when | Contents |
| --- | ---: | --- | --- |
| PR smoke corpus | 10-30 files | every PR | representative small docs, one scan, one web page, one chat/email fixture, one code sample. |
| Nightly corpus | 100-500 files | scheduled | difficult PDFs, scans, tables, charts, web pages, media, multilingual examples, repo snapshots. |
For each fixture, store expected properties, not just expected text:
```yaml
id: "scan_invoice_001"
source: "tests/golden/scan_invoice_001.pdf"
expected:
source_type: scanned_pdf
ocr_required: true
min_pages: 2
anchors:
page: true
max_duplicate_chunk_rate: 0.02
required_fields:
- source_id
- content_sha256
- parser.name
- sensitivity
```
## Security and privacy controls
Ingestion security runs before indexing.
Required controls:
- quarantine untrusted sources before parsing;
- sandbox conversion/OCR/ASR workers;
- prevent parser workers from accessing repo write credentials and model/provider secrets;
- scan extracted text before external processing or embedding;
- classify sensitivity before production retrieval;
- keep raw, extracted, index, trace and export artifacts classified;
- enforce source-domain, tenant, sensitivity, review-state and publication filters before retrieval;
- block public exports without allowlist and redaction report.
Redaction order:
```text
capture -> hash -> convert in sandbox -> scan extracted text -> redact or classify -> write indexes -> review -> export
```
## Incremental sync and deduplication
Use content hashing and source IDs to avoid churn.
Recommended fields:
```yaml
sync:
source_cursor: ""
last_seen_at: "YYYY-MM-DDTHH:MM:SSZ"
previous_sha256: ""
current_sha256: ""
change_type: new|updated|unchanged|deleted|renamed
dedupe_key: ""
canonical_source_id: ""
```
Rules:
- Reuse `source_id` when only metadata changes.
- Re-parse when content hash changes.
- Re-embed only changed chunks when chunk hashes are stable.
- Keep tombstone manifests for deleted upstream sources when citations may still reference them.
- Do not deduplicate across tenants or sensitivity classes without policy approval.
## Operational models
| Mode | Use when | Requirements |
| --- | --- | --- |
| Manual batch | First migration or sensitive corpus. | Report-only dry run, human review. |
| Watched inbox | Personal capture. | Low-risk sources, clear review states. |
| PR-based ingest | Team repo/wiki. | Branch, diff, CODEOWNERS, CI gates. |
| Queue/worker ETL | Production scale. | Retries, idempotency, observability, dead-letter queue. |
| Connector platform | SaaS/db-heavy org. | Source scopes, incremental cursors, access review. |
## Observability
Track:
- ingest queue depth;
- parser success/failure by source type and parser version;
- conversion latency;
- OCR/ASR latency;
- average chunks per source;
- duplicate chunk rate;
- chunks missing provenance anchors;
- scan/redaction findings;
- embedding job failures;
- index refresh lag;
- retrieval recall drift after ingestion changes.
## First 90 days rollout
| Period | Work |
| --- | --- |
| Days 1-14 | Inventory source types, data classes, owners and retention constraints. |
| Days 15-30 | Add manifests, content hashing and raw/source/wiki layout. |
| Days 31-45 | Build golden corpus and ingestion smoke tests. |
| Days 46-60 | Add source-type pipelines for docs, scans, web, media, code and chats/email. |
| Days 61-75 | Add security scans, redaction, parser sandbox and reviewed-only indexing. |
| Days 76-90 | Add incremental sync, dedupe, observability and nightly fidelity regression. |
## Anti-patterns
- Flattening every source into one text field.
- Creating synthesis pages from unmanifested files.
- Discarding raw sources after conversion.
- Embedding raw sensitive captures before classification/redaction.
- Treating OCR/table extraction as trustworthy without sampling.
- Losing page, timestamp, message, row or symbol anchors.
- Using source URLs as durable provenance without archived snapshots.
- Mixing draft/rejected/quarantined content into production indexes.
- Running untrusted parsers with broad filesystem, network or credential access.
- Re-embedding the whole corpus on every small metadata change.
## Source URLs to re-check
- <https://docling-project.github.io/docling/>
- <https://docling-project.github.io/docling/concepts/chunking/>
- <https://github.com/microsoft/markitdown>
- <https://docs.unstructured.io/open-source/introduction/overview>
- <https://tika.apache.org/>
- <https://ocrmypdf.readthedocs.io/>
- <https://tesseract-ocr.github.io/tessdoc/Installation.html>
- <https://paddlepaddle.github.io/PaddleOCR/main/en/index.html>
- <https://github.com/datalab-to/marker>
- <https://github.com/facebookresearch/nougat>
- <https://playwright.dev/>
- <https://github.com/mozilla/readability>
- <https://archivebox.io/>
- <https://github.com/gildas-lormeau/singlefile>
- <https://github.com/openai/whisper>
- <https://github.com/SYSTRAN/faster-whisper>
- <https://github.com/yt-dlp/yt-dlp>
- <https://github.com/tree-sitter/tree-sitter>
- <https://ast-grep.github.io/>
- <https://github.com/langchain-ai/openwiki>
- <https://github.com/openbmb/repoagent>
- <https://github.com/SpamScope/mail-parser>
- <https://notmuchmail.org/getting-started/>
- <https://slack.com/help/articles/201658943-Export-your-workspace-data>
- <https://dlthub.com/docs/intro>
- <https://docs.airbyte.com/>
- <https://duckdb.org/>
- <https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html>
- <https://qdrant.tech/documentation/search/filtering/>
- <https://qdrant.tech/documentation/manage-data/indexing/>
- <https://docs.lancedb.com/quickstart>
- <https://github.com/pgvector/pgvector>
- <https://github.com/data-privacy-stack/presidio>
- <https://github.com/gitleaks/gitleaks>
- <https://github.com/Yelp/detect-secrets>
- <https://docs.semgrep.dev/getting-started/quickstart-ce>
- <https://docs.docker.com/engine/security/seccomp/>
- <https://openpolicyagent.org/docs>
- <https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/>
- <https://github.com/eXascaleInfolab/pytrec_eval>
- <https://www.promptfoo.dev/docs/intro/>
- <https://deepeval.com/docs/getting-started>
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |