Skip to content

EliasOulkadi/shokunin / .pack/skills/chromadb

chromadb

Verified

Manage the ChromaDB vector database that stores the ecosystem's persistent memory. Use when user asks to check memory storage, backup memory, search stored entries, delete entries, or reset the vector database. Do NOT use for general question answering about past sessions (use the memory skill for that).

chromadb · Memory Storage

ChromaDB is the vector database that powers Shokunin's persistent AI memory. It stores embeddings for semantic search across sessions. Built on SQLite with an ONNX embedding model for local-first, offline-capable operation.

Workflow

Step 1: Check storage status

python ~/.shokunin/scripts/chroma-helper.py count

Output: total entries, collection count, disk size on disk, last write timestamp.

python ~/.shokunin/scripts/chroma-helper.py stats

Output: entries per type (decision, file, command, checkpoint, session_end), per project, per date range.

Step 2: Search stored entries

python ~/.shokunin/scripts/chroma-helper.py search "query text" "project-name" 10

Searches via vector similarity + BM25 keyword hybrid. Results sorted by relevance with metadata attached.

python ~/.shokunin/scripts/chroma-helper.py search "error handling" "" 20 --type decision

Filter by entry type. Leave project empty to search all projects.

Step 3: List recent entries

python ~/.shokunin/scripts/chroma-helper.py recent 10

Shows the 10 most recently inserted entries across all projects. Use for quick audit.

python ~/.shokunin/scripts/chroma-helper.py recent 50 --project myproject

Filter recent entries to a specific project.

Step 4: Backup the database

$backupDir = "$env:USERPROFILE\.shokunin\backups\chroma-$(Get-Date -Format 'yyyyMMdd-HHmmss')"
New-Item -ItemType Directory -Path $backupDir -Force | Out-Null
Compress-Archive -Path "$env:USERPROFILE\.shokunin\memory\chroma_db" -DestinationPath "$backupDir\chroma_db.zip" -Force

Also back up session logs:

Compress-Archive -Path "$env:USERPROFILE\.shokunin\memory\sessions" -DestinationPath "$backupDir\sessions.zip" -Force

Verify backup integrity:

Add-Type -AssemblyName System.IO.Compression.FileSystem
[System.IO.Compression.ZipFile]::OpenRead("$backupDir\chroma_db.zip").Entries.Count

Step 5: Delete entries by session prefix

python ~/.shokunin/scripts/chroma-helper.py delete "session-prefix"

Deletes all entries whose session_id starts with the given prefix. Non-reversible. Backup first.

python ~/.shokunin/scripts/chroma-helper.py delete --project "old-project"

Delete all entries for a project. Useful when archiving finished projects.

Step 6: Full reset (requires confirmation)

Remove-Item -Recurse -Force "$env:USERPROFILE\.shokunin\memory\chroma_db"
# Restart MCP server to recreate the database

The database is auto-created on next MCP server startup. All sessions, embeddings, and metadata are permanently lost. Only do this when migrating or recovering from corruption.

Storage Details

LocationPurpose
~/.shokunin/memory/chroma_db/ChromaDB persistent files (SQLite + embeddings)
~/.shokunin/memory/sessions/Per-session JSONL logs and Markdown summaries
~/.shokunin/memory/mcp-server.logMCP server log file (auto-rotated at 5MB)
~/.shokunin/backups/Timestamped backup archives
~/.shokunin/scripts/chroma-helper.pyCLI management script

Data Model

ConceptDescription
CollectionChromaDB namespace for embeddings (one per project by default)
EmbeddingVector representation of text for semantic similarity search
MetadataKey-value pairs attached to each entry (type, tags, project, session_id, timestamp)
DocumentThe raw text content stored alongside the embedding
BM25 IndexSparse retrieval index that complements vector search (keyword matching)
Result FusionMerges vector + BM25 results for higher recall (reciprocal rank fusion)
Entry Typesdecision, file, command, preference, checkpoint, session_end, general

Session Management

CommandDescription
python ~/.shokunin/scripts/chroma-helper.py session list [N]List recent sessions with metadata
python ~/.shokunin/scripts/chroma-helper.py session continue <id>Load full session context
python ~/.shokunin/scripts/chroma-helper.py session summary <id>Show session summary
python ~/.shokunin/scripts/chroma-helper.py save "<text>" "<id>" "<type>" "<tags>" "<project>"Save entry

Error Handling

ErrorCauseFix
ChromaDB not foundPackage not installedpip install chromadb
Collection emptyNo data stored yetNormal on first use. Start storing context.
Slow first queryDownloading ONNX modelFirst query downloads ~79MB. Subsequent queries are instant.
Permission deniedFile locked by another processClose other MCP server instances
Corrupted databaseUnexpected shutdown or disk fullRestore from backup or delete and let the system reinitialize
Memory usage highLarge collection over timeRun consolidate_memories to merge old entries
Embedding dimension mismatchModel was upgraded or changedDelete chroma_db/ and reindex from session logs
UUID collisionRace condition in parallel writesChromaDB handles this internally; retry the insert
SQLite disk I/O errorFilesystem full or read-onlyFree disk space (need ~2x DB size); check permissions

Anti-Patterns

PatternProblemFix
Deleting chroma_db/ without backupPermanent data lossAlways backup first (Step 4)
Running multiple MCP serversDatabase locked, data corruptionOnly one MCP server instance at a time
Manual SQL manipulationBreaks ChromaDB invariantsUse chroma-helper.py CLI or MCP tools only
Ignoring session_list filtersSeeing test/healthcheck entriessession_list automatically filters noise
Storing extremely large textBloating vector indexChromaDB truncates at 500 chars
Never consolidatingDatabase grows indefinitelyRun consolidate_memories periodically
Skipping backupsNo recovery from disk failureWeekly backup is configured via Task Scheduler
Using raw SQLite instead of ChromaDB APISchema and index corruptionAlways go through ChromaDB's Python client
Searching without project filter for broad queriesToo many irrelevant resultsAlways filter by project when context is known

Automation

Weekly backup every Sunday via Windows Task Scheduler (pre-configured). Manual backup via Step 4 above. Logs auto-rotate at 5MB to prevent unbounded growth. The consolidate_memories function runs automatically when entry count exceeds 10,000 per project.

Recovery Procedures

Restore from backup

$backup = Get-ChildItem "$env:USERPROFILE\.shokunin\backups" -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 1
Expand-Archive -Path "$backup\chroma_db.zip" -DestinationPath "$env:USERPROFILE\.shokunin\memory\chroma_db" -Force

Rebuild from session logs

If backups are unavailable, session JSONL logs contain all raw text. Replay them:

python ~/.shokunin/scripts/chroma-helper.py rebuild --from-sessions

Verify database health

python ~/.shokunin/scripts/chroma-helper.py verify

Checks: SQLite integrity, embedding count vs document count, orphan metadata, collection schema version.

Checklist

  • ChromaDB client initialized lazily (not at import time)
  • Collection name sanitized (alphanumeric + hyphens only)
  • Embedding dimension matches between writes and queries
  • Backups configured before destructive operations (delete/reset)
  • Search results validated — empty results handled gracefully

Sources

  • ChromaDB documentation (docs.trychroma.com)
  • OpenAI text-embedding-ada-002 (or local ONNX embedder)
  • SQLite documentation (sqlite.org)
  • DuckDB (ChromaDB's internal query engine)
  • MCP Protocol (modelcontextprotocol.io)
  • Shokunin Memory System Architecture (ARCHITECTURE.md)
deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall