Skip to content

resonatehq/resonate-skills / durable-execution

durable-execution

Unverified

Conceptual introduction to durable execution — what it is, why you'd want it, and the tradeoffs between rolling your own (just your DB) versus using a framework like Resonate. Read this BEFORE picking an implementation. For the Resonate SDK's concrete Context API (ctx.run, ctx.sleep, ctx.promise, structured concurrency), use the per-SDK skill for your language — resonate-basic-durable-world-usage-{typescript,python,rust,go}.

references/TROUBLESHOOTING.md

View on GitHub
# Troubleshooting Durable Workflows

Five-step triage for when things go wrong.

> **Language note.** The symptoms and fixes here are written against the **TypeScript** SDK (generator functions, `yield*`). The *categories* of failure (non-determinism, missing checkpoints, wrong function shape) are universal, but the concrete tells differ per SDK — Python uses bare `yield`, Rust uses `async fn` + `.await`, Go uses ordinary funcs + `Future.Await`, and none of those will show generator-specific errors. For language-specific debugging, see `resonate-basic-debugging-{typescript,python,rust,go}`.

---

## The 5-Step Debug Protocol

1. **Is the server running?** — `curl -sf http://localhost:9090/metrics && echo "up" || echo "down"`
2. **Is the worker connected?** — Check worker logs for "Worker running" and SSE connection
3. **What state is the promise in?** — Query with `promise.get`
4. **What state is the task in?** — Query with `task.get` or `debug.snap`
5. **Is the target group correct?** — Compare `resonate:target` tag with worker's group

---

## Workflow Hangs (Never Completes)

The most common issue. The workflow dispatched but nothing happens.

### Cause 1: Worker Not Registered for the Function

The server dispatched a task for function `processOrder`, but no worker has registered it.

**Diagnosis:**
```bash
# Check worker logs for registration messages
# Look for: "register: processOrder"
```

**Fix:** Ensure `resonate.register("processOrder", fn)` runs immediately after constructing the `Resonate` client — there's no separate `start()` call; the client begins accepting dispatched tasks as soon as it's constructed, so a function registered later misses early tasks.

### Cause 2: Target Group Mismatch

The promise was created with `"resonate:target": "poll://any@workers"` but the worker connected with `group: "worker"` (singular).

**Diagnosis:**
```bash
# Check the promise's target tag
curl -s -X POST http://localhost:8001 \
  -H "Content-Type: application/json" \
  -d '{"kind":"promise.get","head":{"corrId":"d","version":""},"data":{"id":"YOUR_ID"}}' | jq '.data.promise.tags["resonate:target"]'
```

**Fix:** Make sure the `group` in `new Resonate({ group: "..." })` matches the target in the dispatch call.

### Cause 3: Missing `yield`

```typescript
// WRONG — ctx.run() returns a Descriptor, step never executes
const result = ctx.run(myStep, data);

// CORRECT — yield the Descriptor
const result = yield* ctx.run(myStep, data);
```

**Fix:** Add `yield` before every `ctx.run()`, `ctx.rpc()`, `ctx.sleep()`, and `ctx.promise()`.

### Cause 4: Task Stuck in `acquired` State

A worker acquired the task but crashed before fulfilling it. The task remains `acquired` until the lease expires (default: 15 seconds).

**Diagnosis:**
```bash
curl -s -X POST http://localhost:8001 \
  -H "Content-Type: application/json" \
  -d '{"kind":"debug.snap","head":{"corrId":"d","version":""},"data":{}}' | jq '.data.taskTimeouts'
```

**Fix:** Wait for the lease timeout. The server will transition the task back to `pending` and re-dispatch it. If you need faster recovery, reduce `RESONATE_TASKS__LEASE_TIMEOUT`.

### Cause 5: `async function` Instead of `function*`

```typescript
// WRONG — async functions bypass the durable execution engine
resonate.register("broken", async function (ctx, data) { ... });

// CORRECT — must be a generator function
resonate.register("working", function* (ctx, data) { ... });
```

---

## Promise States and What They Mean

| State | Meaning | Action |
|-------|---------|--------|
| `pending` | Workflow is in progress or hasn't started | Wait, or check if a worker is processing it |
| `resolved` | Workflow completed successfully | Read the value for the result |
| `rejected` | Workflow failed with an error | Read the value for the error message |
| `rejected_canceled` | Workflow was explicitly canceled | Check who/what canceled it |
| `rejected_timedout` | Promise exceeded its `timeoutAt` deadline | Increase timeout or investigate slow steps |

Query a promise:

```bash
curl -s -X POST http://localhost:8001 \
  -H "Content-Type: application/json" \
  -d '{"kind":"promise.get","head":{"corrId":"q","version":""},"data":{"id":"YOUR_PROMISE_ID"}}' | jq '.data.promise'
```

---

## Server Won't Start

| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| "Port already in use" | Another process on port 8001 | `lsof -i :8001` then kill it, or change `RESONATE_SERVER__PORT` |
| Exits immediately | Database path not writable | `mkdir -p $(dirname $RESONATE_STORAGE__SQLITE__PATH)` |
| Auth key error | Invalid PEM file | Regenerate: `openssl genrsa -out priv.pem 2048 && openssl rsa -in priv.pem -pubout -out pub.pem` |

---

## Authentication Errors

### 401 Unauthorized

- **No token:** Add `token: "..."` to the Resonate constructor — a top-level option, not nested under `auth`. An unknown option is silently ignored, so a nested `auth` object produces no error and no credentials.
- **Expired token:** Regenerate with `jwt encode --secret @private_key.pem -A RS256 --exp='+90 days' '{"prefix":""}'`
- **Wrong key:** Token was signed with a different private key than the server's public key

### 403 Forbidden

- **Prefix mismatch:** Token's `prefix` claim doesn't match the promise ID prefix
- **No `prefix` claim:** A token without a `prefix` claim denies all access. Must include `"prefix": ""` for unrestricted access. (A literally empty payload `{}` fails earlier as a 401 — the server also requires an `exp` claim on every token, signed or not.)

---

## Unexpected Replays

Side effects (logging, API calls) re-execute on replay. This is expected behavior.

```typescript
resonate.register("myWorkflow", function* (ctx: Context) {
  // WARNING: This console.log runs on EVERY replay
  console.log("Starting workflow");

  // This only runs once — result is cached
  const result = yield* ctx.run(function* () {
    console.log("This also replays, but the RESULT is cached");
    return 42;
  });

  return result;
});
```

**Fix:** Move side effects inside `yield* ctx.run()` blocks. Use the outbox pattern for external calls (emails, webhooks). Accept that `console.log` inside generators will re-execute on replay.

---

## `await` Inside Generator (Silent Failure)

```typescript
// WRONG — await bypasses the durable execution engine
resonate.register("broken", function* (ctx: Context) {
  const data = await fetch("https://api.example.com/data");  // NOT durable!
  return data;
});
```

The `await` works on first run but is NOT replayed on crash recovery. The fetch call is invisible to the engine.

**Fix:** Resonate SDK generators are synchronous. For async I/O, use `ctx.rpc()` to dispatch to a dedicated worker, or accept that the operation is not replay-safe.

---

## Non-Deterministic Replay Errors

If you use `Math.random()`, `Date.now()`, or `crypto.randomUUID()` between yields, replay produces different child IDs than the original execution. The engine can't match cached results to the correct steps.

```typescript
// WRONG — different ID on replay
resonate.register("broken", function* (ctx: Context) {
  const id = crypto.randomUUID();  // Different on replay!
  yield* ctx.run(processWithId, id);
});

// CORRECT — derive ID from stable inputs
resonate.register("fixed", function* (ctx: Context, orderId: string) {
  yield* ctx.run(processWithId, `${orderId}-step-1`);
});
```

---

## Server Health Checks

```bash
# Is the server responding?
curl -sf http://localhost:9090/metrics | head -3

# How many requests has it processed?
curl -s http://localhost:9090/metrics | grep resonate_request_total

# Snapshot of all internal state (promises, tasks, messages, timeouts)
curl -s -X POST http://localhost:8001 \
  -H "Content-Type: application/json" \
  -d '{"kind":"debug.snap","head":{"corrId":"snap","version":""},"data":{}}' | jq '.data | keys'
```

---

## Quick Fixes

| Problem | Quick Fix |
|---------|-----------|
| "Same ID returns old result" | Expected — idempotency. Use a new ID for a new execution |
| Worker keeps restarting | Check `journalctl -u resonate -f` for crash logs |
| Tasks re-dispatching repeatedly | Worker not calling `task.fulfill`. Check for unhandled errors |
| HITL promise never resolves | Verify the resolve/reject/cancel call uses the correct promise ID and value encoding |
| Sleep timer doesn't fire | Confirm the server is running and a worker is connected — durable sleeps fire on the server's background processing loop |
| Database locked errors | Only one server process per database file. Kill duplicates |
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall