Skip to content

K-Dense-AI/scientific-agent-skills / skills/open-notebook

open-notebook

Verified

Organizes research with the self-hosted Open Notebook alternative to NotebookLM. Supports source ingestion (PDFs, web pages, audio, video, and Office documents), cited document chat, text and vector search, notes, custom transformations, and multi-speaker podcasts. Use when automating Open Notebook through its REST API or configuring its local or cloud AI providers, including OpenAI, Anthropic, Google, Ollama, Groq, and Mistral.

references/architecture.md

View on GitHub
# Open Notebook Architecture

Reviewed 2026-09-30 using
[application wiring](https://github.com/lfnovo/open-notebook/blob/v1.14.0/api/main.py),
[storage paths](https://github.com/lfnovo/open-notebook/blob/v1.14.0/open_notebook/config.py),
and [source installation](https://github.com/lfnovo/open-notebook/blob/v1.14.0/docs/1-INSTALLATION/from-source.md).
These are source-level findings, not live deployment verification.

## Runtime components

- **Next.js/React frontend:** Docker exposes port 8502; source development uses 3000.
- **FastAPI backend:** default port 5055, business routes below `/api`, live schemas
  at `/openapi.json`. Pydantic models validate known fields, but extra request fields
  may be silently ignored. Use the deployed schema when relying on a filter.
- **SurrealDB v2:** notebook/source/note/model/credential records, graph relations,
  full-text/vector retrieval, background command records, and schema migrations.
  The upstream Compose file persists its RocksDB data under `/mydata`.
- **Background worker:** `surreal-commands-worker` processes extraction, embeddings,
  and podcast work. It is a separate process even when bundled in the app container.
  API health alone cannot demonstrate processing readiness.
- **LangChain/LangGraph:** AI chains and conversation state. Chat checkpoint state
  also uses SQLite under the application data directory, not just SurrealDB.
- **Esperanto and processing libraries:** connect to independently configured
  language, embedding, transcription, speech, and extraction providers.

## Ingestion and evidence

1. A request specifies link/upload/text content and notebook associations.
2. The backend validates content type and path/URL policy, creates a source, and
   either runs processing synchronously or submits a background command.
3. Extraction produces text; optional transformations produce insights; requested
   embeddings enable vector retrieval.
4. Poll status and retrieve the source to inspect text, errors, and embedding
   coverage. A successful queue submission does not show completion.

PDF extraction, OCR, tables, and transcripts can lose important scientific detail.
Compare extracted measurements, units, and section boundaries with the original
before using an AI summary as evidence. Preserve source IDs and selected context
with analysis outputs; generated notes are derivative material, not independent data.

## Chat versus Ask

Notebook chat first builds a context object from explicit source/note selections,
then submits that object with a message to `/api/chat/execute`. The endpoint returns
JSON message history. An empty context configuration has special meaning: it
includes all notebook items in short form. Explicit ID selections and verification
of returned content prevent accidental reliance on missing/full-text-free context.
The context builder can skip individual unavailable records without failing.

Ask searches the knowledge base with an embedding model and uses strategy, answer,
and final-answer language models. It can return JSON or an SSE stream depending on
the route. The release v1.14.0 Search/Ask API is global. Current main adds notebook
scoping; a field silently ignored by an older server cannot enforce that boundary.

## Podcasts and persistence

Podcast jobs combine an episode profile, one multi-speaker profile, supplied
content, language models for scripts, and speech models for audio. They create an
episode and audio under `/app/data/podcasts` in the container. A retry of a failed
episode deletes its record/audio and starts a new job. Inspect the returned job
result for the new episode ID.

Back up SurrealDB **and** `/app/data`, which contains uploads, podcasts, SQLite
checkpoints, and caches. Stored provider keys use the server's
`OPEN_NOTEBOOK_ENCRYPTION_KEY`; changing or losing it can make credentials unreadable.
This is not encryption of every stored document.

## Data boundaries

Self-hosted storage and local AI inference are separate choices. Cloud models,
embedding providers, transcription, speech, or external extraction services can
receive research content. Source ingestion can also fetch public URLs from the
server. Use only intended source URLs and inspect the selected provider for each
stage. The password middleware is instance-wide bearer-password protection; it
does not create per-notebook user permissions.

## Source navigation

| Upstream location | Responsibility |
| --- | --- |
| `api/main.py`, `api/auth.py` | Router mounting, middleware, password checks |
| `api/models.py`, `api/routers/` | Request/response schemas and endpoints |
| `api/podcast_service.py` | Podcast submission and job-result contract |
| `open_notebook/domain/` | Notebook/source/note records and relationships |
| `open_notebook/graphs/` | Chat, Ask, transformation and processing flows |
| `open_notebook/utils/context_builder.py` | Source/note context selection |
| `commands/` | Background jobs |
| `frontend/` | Next.js client |
| `docker-compose.yml` | Services and persistent mounts |
deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall