Skip to content

donaldfilimon/abi / .agents/skills/abi-wdbx-persistence

abi-wdbx-persistence

Verified

WDBX persistence superpower. JSONL snapshots, CRC32-framed WAL, epoch-gated segment checkpoints, recovery, compaction.

FileKindSize
SKILL.mdskill md8.9 KB
---
name: abi-wdbx-persistence
description: WDBX persistence superpower. JSONL snapshots, CRC32-framed WAL, epoch-gated segment checkpoints, recovery, compaction.
superpower:
  command: "execute"
  parameters:
    - name: "action"
      type: "string"
      enum: ["init", "verify", "compact", "recover", "wal", "segments", "snapshot"]
      description: "Persistence action"
    - name: "path"
      type: "string"
      description: "Database path"
    - name: "keep"
      type: "integer"
      description: "Segments to keep during compaction"
---

> **WDBX moved out of this repository on 2026-08-22.** It now lives in the
> sibling repo `../wdbx` together with `abi-compute`,
> `abi-foundation`, `abi-core`, and `abi-telemetry`; `abi` consumes them by
> relative path. Source paths below therefore read `../wdbx/crates/...`. Run
> WDBX-only tests from that repo (`./tools/cargo.sh test --locked
> --manifest-path ../wdbx/Cargo.toml --workspace < /dev/null`, run from the abi
> checkout), and `abi`'s gate (`./tools/check.sh`) from here.
>
> Under the Abbey System Constitution
> (`docs/superpowers/specs/2026-08-22-abbey-system-constitution.md`) WDBX is the
> **provenance-aware episodic substrate**, not a vector store. Most of the
> evidence half of its specification is unimplemented; the measured gap list is
> in `docs/superpowers/specs/2026-08-22-wdbx-conformance-gap-analysis.md`. Do not
> describe an episodic capability as Current on the strength of the vector-store
> features that do exist.

# ABI Superpower: WDBX Persistence

Exposes the WDBX durability layer as a superpower. Layered snapshot + WAL + segment design with runtime recovery and compaction.

## Actions

### init
Initialize a new durable store:
```
/abi-wdbx-persistence init --path ./data
```

### verify
Cross-check checkpoint integrity, block-chain validity, WAL frame integrity, and merged checkpoint+WAL chain:
```
/abi-wdbx-persistence verify --path ./data
```

### compact
Retain newest N segment checkpoints, reclaim older ones:
```
/abi-wdbx-persistence compact --path ./data --keep 5
```

### recover
Show recovery status (latest checkpoint + WAL delta):
```
/abi-wdbx-persistence recover --path ./data
```

### wal
Inspect WAL frames:
```
/abi-wdbx-persistence wal --path ./data
```

### segments
List segment checkpoints and epochs:
```
/abi-wdbx-persistence segments --path ./data
```

### snapshot
Create JSONL snapshot (compatibility mirror):
```
/abi-wdbx-persistence snapshot --path ./data
```

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                        WDBX Store                                │
└─────────────────────────────────────────────────────────────────┘
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
       ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
       │  WAL        │ │  Segments   │ │  Snapshot   │
       │  (wal)      │ │ (segments)  │ │ (persist)   │
       │             │ │             │ │  (mirror)    │
       │ CRC32 frames│ │ Manifest +  │ │  JSONL +    │
       │ Append-only │ │ epoch files │ │  SHA-256    │
       │ Replay +    │ │ compactRetain│ │  checksum   │
       │ corruption  │ │ Latest-N    │ │             │
       │ detection   │ │             │ │             │
       └─────────────┘ └─────────────┘ └─────────────┘
              │               │               │
              └───────────────┼───────────────┘
                              ▼
                    ┌─────────────────────┐
                    │  Recovery           │
                    │  (recovery +        │
                    │   durable_store)    │
                    │                     │
                    │  1. Load latest     │
                    │     segment epoch   │
                    │  2. Replay WAL      │
                    │     delta forward   │
                    │  3. Checkpoint      │
                    │     runtime mutations
                    └─────────────────────┘
```

## File Formats

### WAL (`../wdbx/crates/abi-wdbx/src/wal.rs`)
- CRC32-framed append-only records
- Supports: `putVector`, `putBlock`, `putSpatial`, `putTemporalNode`, `putTemporalEdge`, `deleteVector`
- Replay reconstructs state deterministically (reuses `persistence.deserialize`)
- Corruption detection: flipped-byte CRC rejection, bad-header rejection

### Segments (`../wdbx/crates/abi-wdbx/src/segments.rs`)
- Manifest: `<base>.manifest` — lists epoch checkpoints in order
- Checkpoints: `<base>.seg.<epoch>.jsonl` — immutable epoch snapshots
- `loadLatest()` — loads newest epoch
- `listEpochs()` — lists active epochs
- `reclaimEpochs()` — removes old epochs
- `compactRetainingLatest(keep)` — retains newest N checkpoints

### Snapshot (`../wdbx/crates/abi-wdbx/src/persistence.rs` + `../wdbx/crates/abi-wdbx/src/persistence.rs`)
- Header: `# ABI-WDBX v1`
- Records: minified JSON per record (`kv`, `vector`, `block`, `spatial`, `temporal_node`, `temporal_edge`)
- Trailer: `# checksum:<sha256-hex>` covering record body
- Load rejects `ChecksumMismatch` on tamper/truncation
- Checksum-less snapshots loadable (backward compatibility)
- Integer field overflow → `FieldOutOfRange` (no panic)
- Vector ID monotonic restore → `CorruptVectorId` on mismatch
- Block timestamp restore → SHA-256 chain reproduces exactly

## Recovery Flow (`../wdbx/crates/abi-wdbx/src/wal.rs` + `../wdbx/crates/abi-wdbx/src/durable.rs`)

1. **Load latest segment checkpoint** (default runtime baseline)
2. **Replay sidecar WAL** delta forward
3. **Checkpoint runtime mutations** through segments
4. **Compatibility mirror** — monolithic JSONL snapshot still written

CLI commands (`wdbx db *`) recover WAL-ahead state before read/write:
- `block insert/get` → recover → operate
- `query` → recover → search
- `db verify` → cross-checks WAL replay vs current checkpoint block count

## CLI Surface (`abi wdbx db`)

| Command | Description |
|---------|-------------|
| `db init <path>` | Create new segment-backed store |
| `db verify <path>` | Full integrity check (checkpoint + WAL + block chain) |
| `db compact <path> [--codec none|pq|autoencoder]` | Publish and verify an immutable v2 generation; `none` is exact, learned codecs are explicitly lossy and report quality evidence |
| `block insert <path> <profile> <metadata>` | Write segment checkpoint + WAL |
| `block get <path>` | Read latest block |
| `query <path> [text] [persona]` | Stats, hybrid search, or persona-isolated retrieval |

## Durability Invariants

| Invariant | Enforced By |
|-----------|-------------|
| Append-only WAL | `../wdbx/crates/abi-wdbx/src/wal.rs` single-writer, CRC frames |
| Checkpoint immutability | `../wdbx/crates/abi-wdbx/src/segments.rs` epoch files never modified |
| Snapshot integrity | SHA-256 trailer, `ChecksumMismatch` on load |
| Block chain integrity | SHA-256 link, `verifyBlocks()` |
| Epoch ordering | Manifest lists epochs sequentially |
| Vector ID monotonicity | `CorruptVectorId` on restore mismatch |
| WAL-ahead recovery | `../wdbx/crates/abi-wdbx/src/wal.rs` prefers segment + WAL delta |

## CLI Access

```
abi wdbx db init ./data
abi wdbx db verify ./data
abi wdbx db compact ./data 5
abi wdbx block insert ./data abbey '{"note":"test"}'
abi wdbx block get ./data
abi wdbx query ./data "search text" abbey
```

## Build and runtime boundary

`abi-wdbx` is a normal Rust workspace crate; there is no `feat-wdbx` switch or
`FeatureDisabled` stub. Persistence is selected at runtime by the requested
store path and environment. Tests use scratch paths, `ABI_WDBX_PATH=:memory:`,
or `ABI_WDBX_PERSIST=0` and never open the user's live store.

## Claim Boundary

Per `docs/spec/wdbx-north-star.mdx` §3.1 and `docs/contracts/external-claims-audit.mdx`:
- ✅ JSONL snapshot + CRC32 WAL + segment checkpoints
- ✅ Runtime recovery (load epoch + replay WAL)
- ✅ `db verify` cross-checks all layers
- ✅ `db compact` retains newest N epochs
- ✅ Compatibility JSONL mirror for tooling
- ❌ NOT full MVCC visibility (block snapshots + epoch checkpoints only)
- ❌ NOT cross-process/concurrent checkpoint coordination
- ❌ NOT at-rest encryption (proposed Phase 4)
- ❌ NOT signed snapshots (proposed Phase 4)
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall