moonlight-lupin/agent-skills / research/library-rag
library-rag
UnverifiedSemantic search over a personal library using Nemotron-3-Embed-1B embeddings + sqlite-vec. Index books, documents, any text corpus; query by meaning. Includes EPUB→Markdown conversion and MCP server for auto-available search tools.
| File | Kind | Size |
|---|---|---|
| .gitignore | other | 305 B |
| LICENSE | other | 1.3 KB |
| README.md | markdown | 11.8 KB |
| SKILL.md | skill md | 18.9 KB |
| references/chunking-strategies.md | markdown | 5.8 KB |
| references/epub-conversion.md | markdown | 3.8 KB |
| references/openrouter-embeddings.md | markdown | 4.1 KB |
| references/portable-rag-per-skill.md | markdown | 12.1 KB |
| references/rag-pipeline-review.md | markdown | 9.7 KB |
| requirements-dev.txt | markdown | 167 B |
| requirements.txt | manifest | 286 B |
| scripts/convert_epub_library.py | script | 13.0 KB |
| scripts/convert_pdf_library.py | script | 6.0 KB |
| scripts/mcp_server.py | script | 10.9 KB |
| scripts/rag_common.py | script | 8.7 KB |
| scripts/rag_index.py | script | 23.4 KB |
| scripts/rag_query.py | script | 6.8 KB |
| tests/conftest.py | script | 188 B |
| tests/test_chunking.py | script | 5.0 KB |
| tests/test_convert.py | script | 4.6 KB |
| tests/test_convert_pdf.py | script | 4.1 KB |
| tests/test_index_search.py | script | 4.8 KB |
| tests/test_indexing_atomicity.py | script | 4.8 KB |
| tests/test_prune_vacuum.py | script | 5.5 KB |
| tests/test_rag_common.py | script | 4.9 KB |
references/chunking-strategies.md
View on GitHub# Chunking Strategies for Library RAG
Per-source-type chunking patterns used by `rag_index.py`. When adding a new book
format, model your chunker on the closest match here and register it in `discover_files()`.
## Design principles
1. **Target ~800–1000 chars per chunk** — enough for meaningful embeddings, not so much
that the signal dilutes.
2. **Merge small chunks** — paragraphs shorter than ~200 chars should merge with the next
paragraph, not stand alone.
3. **Preserve natural boundaries** — verse groups, paragraph breaks, section headings.
Never split mid-sentence unless a paragraph is absurdly long.
4. **Chunk overlap (~15%)** — `split_long_text()` carries the last ~15% of each chunk
into the next chunk's prefix, improving recall for content that straddles a boundary.
5. **Attach metadata** — every chunk carries `source_type`, `source_book`, `source_file`,
`book`, `chapter`, `verse_range` or `section_title`, `chunk_index`, `chunk_text`.
The `file_hash` is added by the indexer after chunking, not by the chunker.
6. **Clean for embedding** — `clean_for_embedding()` strips markdown markers (`**bold**`,
`# headings`, `[links](url)`, `<br>`, `**[Table N]**`, `<!-- Page N -->`) before
sending to the API. Chunk text itself retains markdown for readability.
7. **L2-normalize vectors** — `normalize_vec()` is called at store time (`float_to_blob`)
and query time (`get_embedding`) so the cosine similarity formula is exact.
## Chunker patterns
### Paragraph merge (general-purpose)
For plain text or markdown without strong heading structure. Merge consecutive
paragraphs into ~max_chars chunks, respecting `MIN_CHUNK_CHARS`:
```python
def merge_paragraphs(paras, max_chars=MAX_CHUNK_CHARS, min_chars=MIN_CHUNK_CHARS):
chunks = []
current = []
current_len = 0
for para in paras:
para = para.strip()
if not para:
continue
if current_len + len(para) > max_chars and current:
text = '\n\n'.join(current)
if len(text) >= min_chars or len(current) == 1:
chunks.append(text)
current = [para]
current_len = len(para)
else:
current.append(para)
current_len += len(para) + 2
if current:
text = '\n\n'.join(current)
if len(text) >= 10:
chunks.append(text)
return chunks
```
### Heading-based (structured books)
For markdown with `##` and `###` section headings. Split by headings, keep tables
attached to their section, fall back to paragraph merge when no headings found.
**Critical pitfall:** Some books use **single newlines for line wrapping**, not
paragraph breaks. The chunker MUST normalize single newlines to spaces before
splitting by double-newline paragraph breaks:
```python
normalized = re.sub(r'(?<!\n)\n(?!\n)', ' ', text_content) # single NL → space
```
Without this, the entire chapter becomes one giant chunk because there are only
2-3 real `\n\n` breaks in a 170K-char file.
### Page-aware (PDF-extracted content)
For markdown extracted from PDFs with `<!-- Page N -->` markers. Parse markers
to stamp each chunk with its page number. See `references/portable-rag-per-skill.md`
for the full pattern.
## Adding a new source type
1. Write a `chunk_<type>(file_path)` function that returns a list of chunk dicts.
2. Each chunk dict needs: `source_type`, `source_book`, `source_file`, `book`, `chapter`,
`verse_range` or `section_title`, `chunk_index`, `chunk_text`.
Do NOT include `file_hash` — the indexer adds it.
3. Register the file discovery in `discover_files()` — add a glob pattern and map it
to your chunker.
4. Test with `--dry-run --source <type>` to verify chunk counts are reasonable
(not 1 per file, not 1000 per file).
5. Run the indexer incrementally — only new files get embedded.
## Common pitfalls
- **sqlite-vec extension must be loaded BEFORE any vec0 DDL**: `init_db()` loads the
extension first, then drops/creates tables. Reversing this order causes
`sqlite3.OperationalError: no such module: vec0`.
- **`file_hash` missing from chunks**: The indexer adds `file_hash` to each chunk dict
after chunking but before calling `store_batch()`. If you call `store_batch()` directly
(e.g. from the MCP `add_book` tool), ensure `file_hash` is set.
- **Single-newline wrapping**: Some text sources use single `\n` for line wrapping, not
paragraph breaks. Normalize to spaces before splitting on `\n\n`.
- **OpenRouter / NIM batch limits**: Batch size of 32 works reliably. Larger batches may
timeout on slow connections.
- **Page citations lost without marker parsing**: PDF extractors insert `<!-- Page N -->`
comments at page boundaries, but if the chunker doesn't parse them, all chunks degrade
to chapter-level citations. Use `split_by_pages()` to split text by markers and stamp
each chunk with its page. See `references/portable-rag-per-skill.md` for the pattern.
- **Similarity formula assumes unit-normalized vectors**: `1 - distance²/2` is only
exact cosine similarity if both stored and query vectors are L2-normalized. Nemotron-3-Embed-1B
(and the legacy bge-m3 path) often return near-unit-norm vectors, but normalize explicitly
at store and query time to guard against model/API changes. See `normalize_vec()` in
`rag_common.py` (used by both indexer and query).
- **`clean_for_embedding` must strip PDF/table artifacts**: In addition to markdown
markers, strip `<br>` (table cell breaks), `**[Table N]**` (table markers), and
`<!-- Page N -->` (page markers) from the text sent to the embedding API. These
are noise that degrade vector quality. Chunk text in the DB retains them for display.
- **`MIN_CHUNK_CHARS` defined but unused**: If a min threshold is defined, wire it in
via a `merge_tiny_chunks()` post-processing step. Without it, near-empty chunks
dilute search results.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |