Disentinel/grafema / .claude/skills/rfdb-write-lock-edge-flush-gotcha
rfdb-write-lock-edge-flush-gotcha
VerifiedRFDB server becomes unresponsive (60s+ timeouts) during enricher addEdges write storms. Use when: (1) RFDB CPU spikes to 60-90% after addEdges calls; (2) subsequent reads time out at 60s; (3) server stays catatonic for 5-15 min after a write storm; (4) Tokio main thread parked (write-lock contention). Root cause: add_edges() calls maybe_auto_flush() which has a memory-pressure path that fires on NODE count — not edge count — and holds the exclusive write lock during disk I/O. Fix: separate maybe_auto_flush_edges() uses byte-limit only.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 4.7 KB |
SKILL.md
View on GitHub---
name: rfdb-write-lock-edge-flush-gotcha
description: |
RFDB server becomes unresponsive (60s+ timeouts) during enricher addEdges write storms.
Use when: (1) RFDB CPU spikes to 60-90% after addEdges calls; (2) subsequent reads time out
at 60s; (3) server stays catatonic for 5-15 min after a write storm; (4) Tokio main thread
parked (write-lock contention). Root cause: add_edges() calls maybe_auto_flush() which has a
memory-pressure path that fires on NODE count — not edge count — and holds the exclusive write
lock during disk I/O. Fix: separate maybe_auto_flush_edges() uses byte-limit only.
author: Claude Code
version: 1.0.0
date: 2026-04-26
---
# RFDB Write Lock Edge-Flush Gotcha (RFD-67)
## Problem
RFDB server saturates during enricher write storms. A single `addEdges` call can hold the
exclusive write lock (`db.engine.write()`) for seconds while flushing data to disk,
blocking ALL reads and other writes during that time.
## Context / Trigger Conditions
- Enrichers (`behaviorEnricher`, `packageApiEnricher`) call `client.addEdges()` repeatedly
- Server was previously used for `grafema analyze` (analysis nodes still in write buffer)
- System memory > 70% used (very common on dev machines)
- First `addEdges` call spikes CPU; subsequent calls time out at 60s
## Root Cause (code trace)
```
client.addEdges(10 edges)
→ with_engine_write() ← acquires EXCLUSIVE write lock
→ engine.add_edges()
→ store.upsert_edges() ← fast, in-memory (~0.01ms)
→ self.maybe_auto_flush() ← ← ← THE BUG
Path 2: memory_pressure > 0.7 && total_write_buffer_nodes() >= 500
→ store.flush_all() ← DISK I/O (seconds) under write lock!
→ lock released
```
`maybe_auto_flush` has two paths:
1. Buffer overflow: `any_shard_needs_flush(node_limit, byte_limit)` — fine
2. **Memory pressure**: `memory_pressure > 0.7 && total_write_buffer_nodes() >= 500` — BUG
Path 2 fires based on **nodes**, not edges. After `grafema analyze`, the write buffer holds
analysis nodes. The enricher's first `addEdges` call triggers a full flush of all those
analysis nodes — the edge write causes a massive node flush.
Files involved:
- `packages/rfdb-server/src/graph/engine_v2.rs` — `add_edges()`, `maybe_auto_flush()`
- `packages/rfdb-server/src/bin/rfdb_server.rs` — `with_engine_write()` (line ~2289)
- `packages/rfdb-server/src/database_manager.rs` — `Database.engine: RwLock<Box<dyn GraphStore>>`
## Fix (applied in RFD-67)
Replace `maybe_auto_flush()` call in `add_edges` with `maybe_auto_flush_edges()`:
```rust
fn maybe_auto_flush_edges(&mut self) {
// Byte-limit only — never fires on memory pressure / node count.
// Prevents edge writes from triggering node flushes under write lock.
if self.store.any_shard_needs_flush(usize::MAX, self.cached_profile.write_buffer_byte_limit) {
if let Err(e) = self.store.flush_all(&mut self.manifest) {
tracing::warn!("auto-flush (edges) failed: {}", e);
}
}
}
```
Key invariant: `add_nodes` keeps `maybe_auto_flush()` (nodes are 120 bytes, OOM risk is real).
`add_edges` uses `maybe_auto_flush_edges()` (edges are 98 bytes, byte limit = 100MB default).
## Verification
Test: `test_edge_writes_do_not_block_reads` in `engine_v2.rs` — wraps engine in RwLock,
runs concurrent writer + reader threads, asserts reads complete in < 1s during edge storm.
## Enricher Contract After Fix
Enrichers must call `Flush` explicitly after their write phase:
```javascript
await enrichBehaviors(client, lookup, { flushBatchSize: 50 });
await client.flush(); // required — edges won't auto-flush during normal write storms
```
Recommended `flushBatchSize`: 50–200 edges per batch. Under 50: excessive socket round-trips.
Over 200: diminishing returns, larger per-request latency.
## Diagnosis Checklist
If RFDB saturates again:
1. `RFDB_VERBOSE=1` — log every request; look for `AddEdges`/`Flush` taking > 1s
2. `GetStats` → `shardDiagnostics.writeBufferNodes` — high = unflushed analysis nodes
3. `samply record` on rfdb-server PID → look for `flush_with_ids` holding write lock
4. Check if enricher calls `client.flush()` after writes
## Notes
The fundamental fix (moving disk I/O off the write-lock path entirely via background flush
thread) is deferred. The current fix eliminates the specific bug: edge writes triggering node
flushes. If enrichers begin emitting both nodes AND edges, the `add_nodes` memory-pressure path
could trigger — at that point a background flush thread becomes necessary.
Safe write throughput: up to ~100K addEdges batches (at 10 edges each) before auto-flush.
That's 1M edges before the byte limit triggers. No enricher should hit this in practice.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |