Skip to content

specstoryai/getspecstory / lore

SpecStory Lore - mine your SpecStory coding histories (any agent - Claude Code, Codex, Cursor, Gemini, and more) into a persistent corpus, surface your reproducible workflows with corroborated evidence, and interactively forge the chosen ones into skills installed across all your agent harnesses. Use when the user wants to turn past AI coding sessions into reusable skills, asks "what could I make into a skill", "mine my lore", "forge skills from my history", or points at a .specstory/history directory.

FileKindSize
.claude-plugin/plugin.jsonconfig738 B
.gitattributesother342 B
.gitignoreother36 B
AGENTS.mdmarkdown1.6 KB
AS-BUILT-ARCHITECTURE.mdmarkdown27.6 KB
CHANGELOG.mdmarkdown12.7 KB
CLAUDE.mdmarkdown11 B
CONTRIBUTING.mdmarkdown5.1 KB
HOW-IT-WORKS.mdmarkdown20.7 KB
LICENSEother11.1 KB
README.mdmarkdown9.3 KB
SKILL.mdskill md32.9 KB
fixtures/golden/forge-plan.mdmarkdown1.7 KB
fixtures/projA/.specstory/.project.jsonconfig93 B
fixtures/projA/.specstory/history/2025-07-10_09-00-00Z-fix-the-gitignore.mdmarkdown736 B
fixtures/projA/.specstory/history/2026-05-01_10-00-00Z-can-we-run-a.mdmarkdown1.2 KB
fixtures/projA/.specstory/history/2026-05-02_11-00-00Z-can-we-run-a.mdmarkdown1015 B
fixtures/projA/.specstory/history/2026-05-03_12-00-00Z-can-we-run-a.mdmarkdown994 B
fixtures/projA/.specstory/history/2026-05-04_13-00-00Z-check-the-vet.mdmarkdown1.0 KB
fixtures/projA/tools/subpkg/.specstory/history/2026-05-07_10-00-00Z-tune-the-subpkg.mdmarkdown458 B
fixtures/projB/.specstory/.project.jsonconfig93 B
fixtures/projB/.specstory/history/2026-05-05_14-00-00Z-can-we-run-a.mdmarkdown795 B
fixtures/projC/.specstory/.project.jsonconfig93 B
fixtures/projC/.specstory/history/2026-05-10_09-00-00Z-where-is-save-implemented.mdmarkdown4.0 KB
fixtures/projC/.specstory/history/2026-05-11_10-00-00Z-where-is-token-validation.mdmarkdown1.9 KB
fixtures/provider-antigravity/.specstory/history/2026-05-23_09-00-00Z-run-the-linter.mdmarkdown567 B
fixtures/provider-deepseek/.specstory/history/2026-05-22_09-00-00Z-count-source-lines.mdmarkdown562 B
fixtures/provider-droid/.specstory/history/2026-05-21_09-00-00Z-show-git-status.mdmarkdown612 B
fixtures/provider-gemini/.specstory/history/2026-05-20_09-00-00Z-list-the-workspace.mdmarkdown610 B
package.jsonmanifest337 B
scripts/deep-mine.workflow.jsscript6.3 KB
scripts/hooks/validate-plan.mjsscript2.5 KB
scripts/lib/beats.mjsscript22.2 KB
scripts/lib/db.mjsscript3.8 KB
scripts/lib/discover.mjsscript3.3 KB
scripts/lib/forged.mjsscript6.5 KB
scripts/lib/indexer.mjsscript7.4 KB
scripts/lib/parse.mjsscript10.4 KB
scripts/lib/patterns.mjsscript14.4 KB
scripts/lib/report.mjsscript21.8 KB
scripts/mine-skills.mjsscript16.9 KB
scripts/theme-sweep.workflow.jsscript7.5 KB
tests/engine.test.mjsscript44.9 KB

scripts/lib/indexer.mjs

View on GitHub
// indexer.mjs - walk projects, parse new/changed transcripts (via parse.mjs), persist beats.
//
// IDEMPOTENCY CONTRACT:
// - session identity = project_id + '/' + filename (stable across re-runs and dir reorganizations)
// - fingerprint = size + mtime + PARSER_VERSION: a session is skipped only when the file is
//   unchanged AND it was indexed by the current parser. Grown/edited sessions are REPLACED whole
//   (beats + children deleted, re-inserted). Engine upgrades re-parse automatically.
// - `--force` re-indexes everything regardless. `prune` removes sessions whose file is gone.

import { readFileSync, statSync, existsSync, realpathSync } from 'node:fs'
import { basename, relative } from 'node:path'
import { execFileSync } from 'node:child_process'
import { discoverProjects, walkMd, fileDate } from './discover.mjs'
import { parseSessionFile, intentSig, sniffAuthor } from './parse.mjs'
import { PARSER_VERSION, deleteSessionRows } from './db.mjs'

// Authoritative author per transcript: who ADDED the file to git (one batched call per project).
// Histories committed to a shared repo carry their session owner this way; local-only files fall
// through to path-sniffing, then the machine user (an uncommitted transcript on this machine is
// almost certainly the local user's session).
function gitAuthors(historyDir) {
  const map = new Map()
  try {
    const root = execFileSync('git', ['rev-parse', '--show-toplevel'], { cwd: historyDir, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }).trim()
    const rel = relative(root, realpathSync(historyDir))   // realpath: /tmp vs /private/tmp etc.
    const out = execFileSync('git', ['log', '--diff-filter=A', '--format=\x01%an', '--name-only', '--', rel],
      { cwd: root, encoding: 'utf8', maxBuffer: 64 * 1024 * 1024, stdio: ['ignore', 'pipe', 'ignore'] })
    let author = ''
    for (const line of out.split('\n')) {
      if (line.startsWith('\x01')) { author = line.slice(1).trim(); continue }
      const f = line.trim()
      if (f && author && !map.has(basename(f))) map.set(basename(f), author)
    }
  } catch { /* not a git repo, or git unavailable - fall through to sniffing */ }
  return map
}

export function indexCorpus(db, args) {
  const projects = discoverProjects(args)
  if (!projects.length) return { indexed: 0, skippedKnown: 0, skippedBig: 0, projects, error: 'no .specstory history found' }
  const cutoff = args.days > 0 ? new Date(Date.now() - args.days * 864e5).toISOString().slice(0, 10) : null

  const known = new Map(db.prepare('SELECT id, size, mtime, parser FROM sessions').all().map(r => [r.id, r]))
  const insSession = db.prepare('INSERT OR REPLACE INTO sessions(id,project_id,project_name,path,date,agent,size,beats,mtime,parser,author,uuid) VALUES(?,?,?,?,?,?,?,?,?,?,?,?)')
  const insEp = db.prepare('INSERT INTO beats(session_id,ord,start_line,intent_raw,intent_sig,n_tools,tool_mix,files,n_cmds,exit_fails,outcome) VALUES(?,?,?,?,?,?,?,?,?,?,?)')
  const insCmd = db.prepare('INSERT INTO commands(beat_id,ord,head,raw,line) VALUES(?,?,?,?,?)')
  const insGram = db.prepare('INSERT INTO grams(beat_id,n,gram) VALUES(?,?,?)')
  const insMeta = db.prepare('INSERT INTO meta_hits(beat_id,meta_id,quote,line) VALUES(?,?,?,?)')

  let indexed = 0, skippedKnown = 0, skippedBig = 0
  for (const proj of projects) {
    // live feedback to stderr: visible in the agent window, never pollutes stdout/--emit json
    const t0 = Date.now()
    let projNew = 0
    process.stderr.write(`๐Ÿ“œ lore ยท indexing ${proj.name} โ€ฆ\n`)
    const authors = gitAuthors(proj.historyDir)
    for (const path of walkMd(proj.historyDir)) {
      let st
      try { st = statSync(path) } catch { continue }
      const size = st.size, mtime = Math.floor(st.mtimeMs)
      if (size > args.maxBytes) { skippedBig++; continue }
      const sid = proj.id + '/' + basename(path)
      const k = known.get(sid)
      if (!args.force && k && k.size === size && k.mtime === mtime && k.parser === PARSER_VERSION) {
        skippedKnown++; continue   // unchanged file, current parser โ†’ already indexed
      }

      let text
      try { text = readFileSync(path, 'utf8') } catch { continue }
      const date = fileDate(path, text.slice(0, 600))
      if (cutoff && date && date < cutoff) continue

      const { agent, uuid, beats: eps } = parseSessionFile(text)
      indexed++; projNew++
      if (projNew % 100 === 0) process.stderr.write(`   โ€ฆ ${projNew} sessions parsed\n`)
      // author ladder: git add-author (authoritative) > home-dir sniff > machine user
      const author = authors.get(basename(path)) || sniffAuthor(text) || process.env.USER || 'unknown'

      // replace this session's prior rows, then write fresh - ATOMICALLY. Without the transaction,
      // a crash mid-write leaves a fingerprint-matched session row with missing children that the
      // incremental skip would then preserve forever. (Batching also makes indexing much faster.)
      db.exec('BEGIN')
      try {
        deleteSessionRows(db, sid)
        insSession.run(sid, proj.id, proj.name, path, date, agent, size, eps.length, mtime, PARSER_VERSION, author, uuid)
        for (let k2 = 0; k2 < eps.length; k2++) {
          const e = eps[k2]
          const sig = intentSig(e.intent)
          const mix = Object.entries(e.tools).map(([t, c]) => `${t}:${c}`).join(',')
          const res = insEp.run(sid, k2, e.startLine, (e.firstLine || '').slice(0, 200), sig,
            e.nTools, mix, [...e.files].slice(0, 20).join(','), e.cmds.length, e.fails, e.outcome)
          const epId = res.lastInsertRowid
          e.cmds.forEach((c, ord) => insCmd.run(epId, ord, c.head, c.raw, c.line))
          const heads = e.cmds.map(c => c.head)
          const seen = new Set()
          for (let n = 2; n <= 4; n++) for (let s = 0; s + n <= heads.length; s++) {
            const g = heads.slice(s, s + n).join(' โ–ธ ')
            const key = n + '|' + g
            if (!seen.has(key)) { seen.add(key); insGram.run(epId, n, g) }   // dedupe within beat
          }
          for (const m of e.metas) insMeta.run(epId, m.id, m.quote, m.line)
        }
        db.exec('COMMIT')
      } catch (err) {
        try { db.exec('ROLLBACK') } catch { /* not in a tx */ }
        throw err
      }
    }
    process.stderr.write(`   ${proj.name}: +${projNew} new (${((Date.now() - t0) / 1000).toFixed(1)}s)\n`)
  }
  return { indexed, skippedKnown, skippedBig, projects }
}

// prune: drop sessions whose transcript no longer exists on disk (deleted/moved corpora),
// and report duplicate filename groups that exist under multiple project_ids (the
// "project gained a git remote โ†’ new git_id" drift case) so the caller can decide.
export function pruneCorpus(db) {
  const rows = db.prepare('SELECT id, path FROM sessions').all()
  let removed = 0
  for (const r of rows) {
    if (!existsSync(r.path)) {
      deleteSessionRows(db, r.id)
      db.prepare('DELETE FROM sessions WHERE id=?').run(r.id)
      removed++
    }
  }
  const dupes = db.prepare(`
    SELECT path, COUNT(DISTINCT project_id) np, GROUP_CONCAT(DISTINCT project_id) pids
    FROM sessions GROUP BY path HAVING np > 1`).all()
  // the SAME session (by provider UUID) indexed from two places - copied/cloned corpora
  // double-count every pattern and fake "portable" signals
  const contentDupes = db.prepare(`
    SELECT uuid, COUNT(*) n, GROUP_CONCAT(id, ' | ') ids
    FROM sessions WHERE uuid != '' GROUP BY uuid HAVING n > 1`).all()
  return { removed, remaining: rows.length - removed, dupes, contentDupes }
}
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why โ€” repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" โ€” or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself โ€” your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall