Skip to content

donaldfilimon/abi / .agents/skills/abi-wdbx-persistence

abi-wdbx-persistence

Verified

WDBX persistence superpower. JSONL snapshots, CRC32-framed WAL, epoch-gated segment checkpoints, recovery, compaction.

WDBX moved out of this repository on 2026-08-22. It now lives in the sibling repo ../wdbx together with abi-compute, abi-foundation, abi-core, and abi-telemetry; abi consumes them by relative path. Source paths below therefore read ../wdbx/crates/.... Run WDBX-only tests from that repo (./tools/cargo.sh test --locked --manifest-path ../wdbx/Cargo.toml --workspace < /dev/null, run from the abi checkout), and abi's gate (./tools/check.sh) from here.

Under the Abbey System Constitution (docs/superpowers/specs/2026-08-22-abbey-system-constitution.md) WDBX is the provenance-aware episodic substrate, not a vector store. Most of the evidence half of its specification is unimplemented; the measured gap list is in docs/superpowers/specs/2026-08-22-wdbx-conformance-gap-analysis.md. Do not describe an episodic capability as Current on the strength of the vector-store features that do exist.

ABI Superpower: WDBX Persistence

Exposes the WDBX durability layer as a superpower. Layered snapshot + WAL + segment design with runtime recovery and compaction.

Actions

init

Initialize a new durable store:

/abi-wdbx-persistence init --path ./data

verify

Cross-check checkpoint integrity, block-chain validity, WAL frame integrity, and merged checkpoint+WAL chain:

/abi-wdbx-persistence verify --path ./data

compact

Retain newest N segment checkpoints, reclaim older ones:

/abi-wdbx-persistence compact --path ./data --keep 5

recover

Show recovery status (latest checkpoint + WAL delta):

/abi-wdbx-persistence recover --path ./data

wal

Inspect WAL frames:

/abi-wdbx-persistence wal --path ./data

segments

List segment checkpoints and epochs:

/abi-wdbx-persistence segments --path ./data

snapshot

Create JSONL snapshot (compatibility mirror):

/abi-wdbx-persistence snapshot --path ./data

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        WDBX Store                                │
└─────────────────────────────────────────────────────────────────┘
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
       ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
       │  WAL        │ │  Segments   │ │  Snapshot   │
       │  (wal)      │ │ (segments)  │ │ (persist)   │
       │             │ │             │ │  (mirror)    │
       │ CRC32 frames│ │ Manifest +  │ │  JSONL +    │
       │ Append-only │ │ epoch files │ │  SHA-256    │
       │ Replay +    │ │ compactRetain│ │  checksum   │
       │ corruption  │ │ Latest-N    │ │             │
       │ detection   │ │             │ │             │
       └─────────────┘ └─────────────┘ └─────────────┘
              │               │               │
              └───────────────┼───────────────┘
                              ▼
                    ┌─────────────────────┐
                    │  Recovery           │
                    │  (recovery +        │
                    │   durable_store)    │
                    │                     │
                    │  1. Load latest     │
                    │     segment epoch   │
                    │  2. Replay WAL      │
                    │     delta forward   │
                    │  3. Checkpoint      │
                    │     runtime mutations
                    └─────────────────────┘

File Formats

WAL (../wdbx/crates/abi-wdbx/src/wal.rs)

  • CRC32-framed append-only records
  • Supports: putVector, putBlock, putSpatial, putTemporalNode, putTemporalEdge, deleteVector
  • Replay reconstructs state deterministically (reuses persistence.deserialize)
  • Corruption detection: flipped-byte CRC rejection, bad-header rejection

Segments (../wdbx/crates/abi-wdbx/src/segments.rs)

  • Manifest: <base>.manifest — lists epoch checkpoints in order
  • Checkpoints: <base>.seg.<epoch>.jsonl — immutable epoch snapshots
  • loadLatest() — loads newest epoch
  • listEpochs() — lists active epochs
  • reclaimEpochs() — removes old epochs
  • compactRetainingLatest(keep) — retains newest N checkpoints

Snapshot (../wdbx/crates/abi-wdbx/src/persistence.rs + ../wdbx/crates/abi-wdbx/src/persistence.rs)

  • Header: # ABI-WDBX v1
  • Records: minified JSON per record (kv, vector, block, spatial, temporal_node, temporal_edge)
  • Trailer: # checksum:<sha256-hex> covering record body
  • Load rejects ChecksumMismatch on tamper/truncation
  • Checksum-less snapshots loadable (backward compatibility)
  • Integer field overflow → FieldOutOfRange (no panic)
  • Vector ID monotonic restore → CorruptVectorId on mismatch
  • Block timestamp restore → SHA-256 chain reproduces exactly

Recovery Flow (../wdbx/crates/abi-wdbx/src/wal.rs + ../wdbx/crates/abi-wdbx/src/durable.rs)

  1. Load latest segment checkpoint (default runtime baseline)
  2. Replay sidecar WAL delta forward
  3. Checkpoint runtime mutations through segments
  4. Compatibility mirror — monolithic JSONL snapshot still written

CLI commands (wdbx db *) recover WAL-ahead state before read/write:

  • block insert/get → recover → operate
  • query → recover → search
  • db verify → cross-checks WAL replay vs current checkpoint block count

CLI Surface (abi wdbx db)

CommandDescription
db init <path>Create new segment-backed store
db verify <path>Full integrity check (checkpoint + WAL + block chain)
`db compact [--codec nonepq
block insert <path> <profile> <metadata>Write segment checkpoint + WAL
block get <path>Read latest block
query <path> [text] [persona]Stats, hybrid search, or persona-isolated retrieval

Durability Invariants

InvariantEnforced By
Append-only WAL../wdbx/crates/abi-wdbx/src/wal.rs single-writer, CRC frames
Checkpoint immutability../wdbx/crates/abi-wdbx/src/segments.rs epoch files never modified
Snapshot integritySHA-256 trailer, ChecksumMismatch on load
Block chain integritySHA-256 link, verifyBlocks()
Epoch orderingManifest lists epochs sequentially
Vector ID monotonicityCorruptVectorId on restore mismatch
WAL-ahead recovery../wdbx/crates/abi-wdbx/src/wal.rs prefers segment + WAL delta

CLI Access

abi wdbx db init ./data
abi wdbx db verify ./data
abi wdbx db compact ./data 5
abi wdbx block insert ./data abbey '{"note":"test"}'
abi wdbx block get ./data
abi wdbx query ./data "search text" abbey

Build and runtime boundary

abi-wdbx is a normal Rust workspace crate; there is no feat-wdbx switch or FeatureDisabled stub. Persistence is selected at runtime by the requested store path and environment. Tests use scratch paths, ABI_WDBX_PATH=:memory:, or ABI_WDBX_PERSIST=0 and never open the user's live store.

Claim Boundary

Per docs/spec/wdbx-north-star.mdx §3.1 and docs/contracts/external-claims-audit.mdx:

  • ✅ JSONL snapshot + CRC32 WAL + segment checkpoints
  • ✅ Runtime recovery (load epoch + replay WAL)
  • ✅ db verify cross-checks all layers
  • ✅ db compact retains newest N epochs
  • ✅ Compatibility JSONL mirror for tooling
  • ❌ NOT full MVCC visibility (block snapshots + epoch checkpoints only)
  • ❌ NOT cross-process/concurrent checkpoint coordination
  • ❌ NOT at-rest encryption (proposed Phase 4)
  • ❌ NOT signed snapshots (proposed Phase 4)
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall