donaldfilimon/abi / .agents/skills/abi-wdbx-persistence
abi-wdbx-persistence
VerifiedWDBX persistence superpower. JSONL snapshots, CRC32-framed WAL, epoch-gated segment checkpoints, recovery, compaction.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 8.9 KB |
SKILL.md
View on GitHub---
name: abi-wdbx-persistence
description: WDBX persistence superpower. JSONL snapshots, CRC32-framed WAL, epoch-gated segment checkpoints, recovery, compaction.
superpower:
command: "execute"
parameters:
- name: "action"
type: "string"
enum: ["init", "verify", "compact", "recover", "wal", "segments", "snapshot"]
description: "Persistence action"
- name: "path"
type: "string"
description: "Database path"
- name: "keep"
type: "integer"
description: "Segments to keep during compaction"
---
> **WDBX moved out of this repository on 2026-08-22.** It now lives in the
> sibling repo `../wdbx` together with `abi-compute`,
> `abi-foundation`, `abi-core`, and `abi-telemetry`; `abi` consumes them by
> relative path. Source paths below therefore read `../wdbx/crates/...`. Run
> WDBX-only tests from that repo (`./tools/cargo.sh test --locked
> --manifest-path ../wdbx/Cargo.toml --workspace < /dev/null`, run from the abi
> checkout), and `abi`'s gate (`./tools/check.sh`) from here.
>
> Under the Abbey System Constitution
> (`docs/superpowers/specs/2026-08-22-abbey-system-constitution.md`) WDBX is the
> **provenance-aware episodic substrate**, not a vector store. Most of the
> evidence half of its specification is unimplemented; the measured gap list is
> in `docs/superpowers/specs/2026-08-22-wdbx-conformance-gap-analysis.md`. Do not
> describe an episodic capability as Current on the strength of the vector-store
> features that do exist.
# ABI Superpower: WDBX Persistence
Exposes the WDBX durability layer as a superpower. Layered snapshot + WAL + segment design with runtime recovery and compaction.
## Actions
### init
Initialize a new durable store:
```
/abi-wdbx-persistence init --path ./data
```
### verify
Cross-check checkpoint integrity, block-chain validity, WAL frame integrity, and merged checkpoint+WAL chain:
```
/abi-wdbx-persistence verify --path ./data
```
### compact
Retain newest N segment checkpoints, reclaim older ones:
```
/abi-wdbx-persistence compact --path ./data --keep 5
```
### recover
Show recovery status (latest checkpoint + WAL delta):
```
/abi-wdbx-persistence recover --path ./data
```
### wal
Inspect WAL frames:
```
/abi-wdbx-persistence wal --path ./data
```
### segments
List segment checkpoints and epochs:
```
/abi-wdbx-persistence segments --path ./data
```
### snapshot
Create JSONL snapshot (compatibility mirror):
```
/abi-wdbx-persistence snapshot --path ./data
```
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ WDBX Store │
└─────────────────────────────────────────────────────────────────┘
│
┌───────────────┼───────────────┐
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ WAL │ │ Segments │ │ Snapshot │
│ (wal) │ │ (segments) │ │ (persist) │
│ │ │ │ │ (mirror) │
│ CRC32 frames│ │ Manifest + │ │ JSONL + │
│ Append-only │ │ epoch files │ │ SHA-256 │
│ Replay + │ │ compactRetain│ │ checksum │
│ corruption │ │ Latest-N │ │ │
│ detection │ │ │ │ │
└─────────────┘ └─────────────┘ └─────────────┘
│ │ │
└───────────────┼───────────────┘
▼
┌─────────────────────┐
│ Recovery │
│ (recovery + │
│ durable_store) │
│ │
│ 1. Load latest │
│ segment epoch │
│ 2. Replay WAL │
│ delta forward │
│ 3. Checkpoint │
│ runtime mutations
└─────────────────────┘
```
## File Formats
### WAL (`../wdbx/crates/abi-wdbx/src/wal.rs`)
- CRC32-framed append-only records
- Supports: `putVector`, `putBlock`, `putSpatial`, `putTemporalNode`, `putTemporalEdge`, `deleteVector`
- Replay reconstructs state deterministically (reuses `persistence.deserialize`)
- Corruption detection: flipped-byte CRC rejection, bad-header rejection
### Segments (`../wdbx/crates/abi-wdbx/src/segments.rs`)
- Manifest: `<base>.manifest` — lists epoch checkpoints in order
- Checkpoints: `<base>.seg.<epoch>.jsonl` — immutable epoch snapshots
- `loadLatest()` — loads newest epoch
- `listEpochs()` — lists active epochs
- `reclaimEpochs()` — removes old epochs
- `compactRetainingLatest(keep)` — retains newest N checkpoints
### Snapshot (`../wdbx/crates/abi-wdbx/src/persistence.rs` + `../wdbx/crates/abi-wdbx/src/persistence.rs`)
- Header: `# ABI-WDBX v1`
- Records: minified JSON per record (`kv`, `vector`, `block`, `spatial`, `temporal_node`, `temporal_edge`)
- Trailer: `# checksum:<sha256-hex>` covering record body
- Load rejects `ChecksumMismatch` on tamper/truncation
- Checksum-less snapshots loadable (backward compatibility)
- Integer field overflow → `FieldOutOfRange` (no panic)
- Vector ID monotonic restore → `CorruptVectorId` on mismatch
- Block timestamp restore → SHA-256 chain reproduces exactly
## Recovery Flow (`../wdbx/crates/abi-wdbx/src/wal.rs` + `../wdbx/crates/abi-wdbx/src/durable.rs`)
1. **Load latest segment checkpoint** (default runtime baseline)
2. **Replay sidecar WAL** delta forward
3. **Checkpoint runtime mutations** through segments
4. **Compatibility mirror** — monolithic JSONL snapshot still written
CLI commands (`wdbx db *`) recover WAL-ahead state before read/write:
- `block insert/get` → recover → operate
- `query` → recover → search
- `db verify` → cross-checks WAL replay vs current checkpoint block count
## CLI Surface (`abi wdbx db`)
| Command | Description |
|---------|-------------|
| `db init <path>` | Create new segment-backed store |
| `db verify <path>` | Full integrity check (checkpoint + WAL + block chain) |
| `db compact <path> [--codec none|pq|autoencoder]` | Publish and verify an immutable v2 generation; `none` is exact, learned codecs are explicitly lossy and report quality evidence |
| `block insert <path> <profile> <metadata>` | Write segment checkpoint + WAL |
| `block get <path>` | Read latest block |
| `query <path> [text] [persona]` | Stats, hybrid search, or persona-isolated retrieval |
## Durability Invariants
| Invariant | Enforced By |
|-----------|-------------|
| Append-only WAL | `../wdbx/crates/abi-wdbx/src/wal.rs` single-writer, CRC frames |
| Checkpoint immutability | `../wdbx/crates/abi-wdbx/src/segments.rs` epoch files never modified |
| Snapshot integrity | SHA-256 trailer, `ChecksumMismatch` on load |
| Block chain integrity | SHA-256 link, `verifyBlocks()` |
| Epoch ordering | Manifest lists epochs sequentially |
| Vector ID monotonicity | `CorruptVectorId` on restore mismatch |
| WAL-ahead recovery | `../wdbx/crates/abi-wdbx/src/wal.rs` prefers segment + WAL delta |
## CLI Access
```
abi wdbx db init ./data
abi wdbx db verify ./data
abi wdbx db compact ./data 5
abi wdbx block insert ./data abbey '{"note":"test"}'
abi wdbx block get ./data
abi wdbx query ./data "search text" abbey
```
## Build and runtime boundary
`abi-wdbx` is a normal Rust workspace crate; there is no `feat-wdbx` switch or
`FeatureDisabled` stub. Persistence is selected at runtime by the requested
store path and environment. Tests use scratch paths, `ABI_WDBX_PATH=:memory:`,
or `ABI_WDBX_PERSIST=0` and never open the user's live store.
## Claim Boundary
Per `docs/spec/wdbx-north-star.mdx` §3.1 and `docs/contracts/external-claims-audit.mdx`:
- ✅ JSONL snapshot + CRC32 WAL + segment checkpoints
- ✅ Runtime recovery (load epoch + replay WAL)
- ✅ `db verify` cross-checks all layers
- ✅ `db compact` retains newest N epochs
- ✅ Compatibility JSONL mirror for tooling
- ❌ NOT full MVCC visibility (block snapshots + epoch checkpoints only)
- ❌ NOT cross-process/concurrent checkpoint coordination
- ❌ NOT at-rest encryption (proposed Phase 4)
- ❌ NOT signed snapshots (proposed Phase 4)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |