moonlight-lupin/agent-skills / research/library-rag
library-rag
UnverifiedSemantic search over a personal library using Nemotron-3-Embed-1B embeddings + sqlite-vec. Index books, documents, any text corpus; query by meaning. Includes EPUB→Markdown conversion and MCP server for auto-available search tools.
| File | Kind | Size |
|---|---|---|
| .gitignore | other | 305 B |
| LICENSE | other | 1.3 KB |
| README.md | markdown | 11.8 KB |
| SKILL.md | skill md | 18.9 KB |
| references/chunking-strategies.md | markdown | 5.8 KB |
| references/epub-conversion.md | markdown | 3.8 KB |
| references/openrouter-embeddings.md | markdown | 4.1 KB |
| references/portable-rag-per-skill.md | markdown | 12.1 KB |
| references/rag-pipeline-review.md | markdown | 9.7 KB |
| requirements-dev.txt | markdown | 167 B |
| requirements.txt | manifest | 286 B |
| scripts/convert_epub_library.py | script | 13.0 KB |
| scripts/convert_pdf_library.py | script | 6.0 KB |
| scripts/mcp_server.py | script | 10.9 KB |
| scripts/rag_common.py | script | 8.7 KB |
| scripts/rag_index.py | script | 23.4 KB |
| scripts/rag_query.py | script | 6.8 KB |
| tests/conftest.py | script | 188 B |
| tests/test_chunking.py | script | 5.0 KB |
| tests/test_convert.py | script | 4.6 KB |
| tests/test_convert_pdf.py | script | 4.1 KB |
| tests/test_index_search.py | script | 4.8 KB |
| tests/test_indexing_atomicity.py | script | 4.8 KB |
| tests/test_prune_vacuum.py | script | 5.5 KB |
| tests/test_rag_common.py | script | 4.9 KB |
references/epub-conversion.md
View on GitHub# EPUB Conversion Notes
## EPUB to text extraction
### Extraction script
```python
import ebooklib
from ebooklib import epub
from bs4 import BeautifulSoup
import os
def extract_epub_to_text(epub_path, output_path):
"""Extract full text from an EPUB and save as .txt."""
book = epub.read_epub(epub_path, options={"ignore_ncx": True})
chunks = []
for item in book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
soup = BeautifulSoup(item.get_content(), "html.parser")
text = soup.get_text(separator="\n", strip=True)
if text.strip():
chunks.append(text)
full_text = "\n\n".join(chunks)
os.makedirs(os.path.dirname(output_path), exist_ok=True)
with open(output_path, "w", encoding="utf-8") as f:
f.write(full_text)
return len(full_text)
```
Dependencies: `pip install ebooklib beautifulsoup4` (use `--break-system-packages` on PEP 668 systems or use a venv).
### Key options
- `ignore_ncx=True` — skip the NCX navigation spine (toc.ncx) which duplicates content from the HTML items.
- `ebooklib.ITEM_DOCUMENT` — extracts only HTML content items (chapters, sections). Filters out images, stylesheets, and metadata.
- `BeautifulSoup.get_text(separator="\n", strip=True)` — clean line-separated text suitable for Markdown generation or direct RAG indexing.
## Structured Markdown conversion (chapter splitting)
After text extraction, the raw `.txt` must be parsed into per-chapter Markdown files.
The conversion script is at `scripts/convert_epub_library.py`.
### Per-chapter Markdown structure
Each chapter becomes `chapters/chNN-slug.md` with YAML frontmatter:
```yaml
---
title: "Chapter Title"
author: Author Name
year: 2024
book: Book Title
chapter: 5
slug: ch05-chapter-title
source: book.txt
license: Private copy — copyrighted work, converted for personal study use only
---
```
### Pitfalls encountered and fixed
1. **Multiple TOC ghosts**: Large EPUBs produce multiple table-of-contents blocks in the extracted text (short TOC, extended TOC, page-number lists). You MUST skip all of them when finding chapter headings in body text. Identify TOC line ranges and exclude them from chapter-heading searches.
2. **CHAPTER N vs N. Title**: EPUBs use different heading patterns in TOC vs body. Always grep for the BODY pattern (e.g. `^Chapter \d+$` or `^CHAPTER \d+$`), not the TOC pattern. Use the TOC only to map chapter numbers to titles.
3. **Title + subtitle leak into YAML**: EPUB headings often include a title line followed by subtitle text. When building YAML frontmatter, use ONLY the main title. Post-process by removing indented continuation lines between `title:` and the next YAML field.
4. **Slug length**: Truncate slugs to 60 chars (preserving `chNN-` prefix) to avoid `OSError: File name too long`.
5. **Page-number blocks**: Some EPUBs extract huge blocks of standalone page numbers. Detect by long runs of purely-numeric lines and skip.
6. **Additional notes and appendices**: Reference works often have special sections between or after chapters. Handle them as separate sections with their own slug prefixes (`additional-note-`, `appendix-NN-`).
7. **Post-processing pass is essential**: After initial generation, ALWAYS verify:
- Every `.md` file starts with `---` (valid YAML frontmatter)
- Every file has `title:`, `year:`, `slug:` fields
- No multi-line values leaked into the YAML block
- Slug lengths are under the filesystem limit
- Content is non-empty for each chapter file
## Telegram limitation
Telegram gateway rejects `.epub` as an unsupported document type. Workarounds:
1. **ZIP the EPUB** — send as `.zip`, agent can unzip and process.
2. **Local file reference** — if the file is already on the machine, reference it by path directly.
3. **Pre-convert** — extract text on the sending device and send `.txt` or `.md` instead.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |