resonatehq/resonate-skills / durable-execution
durable-execution
UnverifiedConceptual introduction to durable execution — what it is, why you'd want it, and the tradeoffs between rolling your own (just your DB) versus using a framework like Resonate. Read this BEFORE picking an implementation. For the Resonate SDK's concrete Context API (ctx.run, ctx.sleep, ctx.promise, structured concurrency), use the per-SDK skill for your language — resonate-basic-durable-world-usage-{typescript,python,rust,go}.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 19.4 KB |
| references/BAKED-IN.md | markdown | 11.0 KB |
| references/DEPLOYMENT.md | markdown | 9.2 KB |
| references/RESONATE-PATTERNS.md | markdown | 16.6 KB |
| references/RESONATE-QUICKSTART.md | markdown | 7.9 KB |
| references/RESONATE-SDK.md | markdown | 30.2 KB |
| references/TESTING.md | markdown | 13.1 KB |
| references/TROUBLESHOOTING.md | markdown | 8.4 KB |
references/TROUBLESHOOTING.md
View on GitHub# Troubleshooting Durable Workflows
Five-step triage for when things go wrong.
> **Language note.** The symptoms and fixes here are written against the **TypeScript** SDK (generator functions, `yield*`). The *categories* of failure (non-determinism, missing checkpoints, wrong function shape) are universal, but the concrete tells differ per SDK — Python uses bare `yield`, Rust uses `async fn` + `.await`, Go uses ordinary funcs + `Future.Await`, and none of those will show generator-specific errors. For language-specific debugging, see `resonate-basic-debugging-{typescript,python,rust,go}`.
---
## The 5-Step Debug Protocol
1. **Is the server running?** — `curl -sf http://localhost:9090/metrics && echo "up" || echo "down"`
2. **Is the worker connected?** — Check worker logs for "Worker running" and SSE connection
3. **What state is the promise in?** — Query with `promise.get`
4. **What state is the task in?** — Query with `task.get` or `debug.snap`
5. **Is the target group correct?** — Compare `resonate:target` tag with worker's group
---
## Workflow Hangs (Never Completes)
The most common issue. The workflow dispatched but nothing happens.
### Cause 1: Worker Not Registered for the Function
The server dispatched a task for function `processOrder`, but no worker has registered it.
**Diagnosis:**
```bash
# Check worker logs for registration messages
# Look for: "register: processOrder"
```
**Fix:** Ensure `resonate.register("processOrder", fn)` runs immediately after constructing the `Resonate` client — there's no separate `start()` call; the client begins accepting dispatched tasks as soon as it's constructed, so a function registered later misses early tasks.
### Cause 2: Target Group Mismatch
The promise was created with `"resonate:target": "poll://any@workers"` but the worker connected with `group: "worker"` (singular).
**Diagnosis:**
```bash
# Check the promise's target tag
curl -s -X POST http://localhost:8001 \
-H "Content-Type: application/json" \
-d '{"kind":"promise.get","head":{"corrId":"d","version":""},"data":{"id":"YOUR_ID"}}' | jq '.data.promise.tags["resonate:target"]'
```
**Fix:** Make sure the `group` in `new Resonate({ group: "..." })` matches the target in the dispatch call.
### Cause 3: Missing `yield`
```typescript
// WRONG — ctx.run() returns a Descriptor, step never executes
const result = ctx.run(myStep, data);
// CORRECT — yield the Descriptor
const result = yield* ctx.run(myStep, data);
```
**Fix:** Add `yield` before every `ctx.run()`, `ctx.rpc()`, `ctx.sleep()`, and `ctx.promise()`.
### Cause 4: Task Stuck in `acquired` State
A worker acquired the task but crashed before fulfilling it. The task remains `acquired` until the lease expires (default: 15 seconds).
**Diagnosis:**
```bash
curl -s -X POST http://localhost:8001 \
-H "Content-Type: application/json" \
-d '{"kind":"debug.snap","head":{"corrId":"d","version":""},"data":{}}' | jq '.data.taskTimeouts'
```
**Fix:** Wait for the lease timeout. The server will transition the task back to `pending` and re-dispatch it. If you need faster recovery, reduce `RESONATE_TASKS__LEASE_TIMEOUT`.
### Cause 5: `async function` Instead of `function*`
```typescript
// WRONG — async functions bypass the durable execution engine
resonate.register("broken", async function (ctx, data) { ... });
// CORRECT — must be a generator function
resonate.register("working", function* (ctx, data) { ... });
```
---
## Promise States and What They Mean
| State | Meaning | Action |
|-------|---------|--------|
| `pending` | Workflow is in progress or hasn't started | Wait, or check if a worker is processing it |
| `resolved` | Workflow completed successfully | Read the value for the result |
| `rejected` | Workflow failed with an error | Read the value for the error message |
| `rejected_canceled` | Workflow was explicitly canceled | Check who/what canceled it |
| `rejected_timedout` | Promise exceeded its `timeoutAt` deadline | Increase timeout or investigate slow steps |
Query a promise:
```bash
curl -s -X POST http://localhost:8001 \
-H "Content-Type: application/json" \
-d '{"kind":"promise.get","head":{"corrId":"q","version":""},"data":{"id":"YOUR_PROMISE_ID"}}' | jq '.data.promise'
```
---
## Server Won't Start
| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| "Port already in use" | Another process on port 8001 | `lsof -i :8001` then kill it, or change `RESONATE_SERVER__PORT` |
| Exits immediately | Database path not writable | `mkdir -p $(dirname $RESONATE_STORAGE__SQLITE__PATH)` |
| Auth key error | Invalid PEM file | Regenerate: `openssl genrsa -out priv.pem 2048 && openssl rsa -in priv.pem -pubout -out pub.pem` |
---
## Authentication Errors
### 401 Unauthorized
- **No token:** Add `token: "..."` to the Resonate constructor — a top-level option, not nested under `auth`. An unknown option is silently ignored, so a nested `auth` object produces no error and no credentials.
- **Expired token:** Regenerate with `jwt encode --secret @private_key.pem -A RS256 --exp='+90 days' '{"prefix":""}'`
- **Wrong key:** Token was signed with a different private key than the server's public key
### 403 Forbidden
- **Prefix mismatch:** Token's `prefix` claim doesn't match the promise ID prefix
- **No `prefix` claim:** A token without a `prefix` claim denies all access. Must include `"prefix": ""` for unrestricted access. (A literally empty payload `{}` fails earlier as a 401 — the server also requires an `exp` claim on every token, signed or not.)
---
## Unexpected Replays
Side effects (logging, API calls) re-execute on replay. This is expected behavior.
```typescript
resonate.register("myWorkflow", function* (ctx: Context) {
// WARNING: This console.log runs on EVERY replay
console.log("Starting workflow");
// This only runs once — result is cached
const result = yield* ctx.run(function* () {
console.log("This also replays, but the RESULT is cached");
return 42;
});
return result;
});
```
**Fix:** Move side effects inside `yield* ctx.run()` blocks. Use the outbox pattern for external calls (emails, webhooks). Accept that `console.log` inside generators will re-execute on replay.
---
## `await` Inside Generator (Silent Failure)
```typescript
// WRONG — await bypasses the durable execution engine
resonate.register("broken", function* (ctx: Context) {
const data = await fetch("https://api.example.com/data"); // NOT durable!
return data;
});
```
The `await` works on first run but is NOT replayed on crash recovery. The fetch call is invisible to the engine.
**Fix:** Resonate SDK generators are synchronous. For async I/O, use `ctx.rpc()` to dispatch to a dedicated worker, or accept that the operation is not replay-safe.
---
## Non-Deterministic Replay Errors
If you use `Math.random()`, `Date.now()`, or `crypto.randomUUID()` between yields, replay produces different child IDs than the original execution. The engine can't match cached results to the correct steps.
```typescript
// WRONG — different ID on replay
resonate.register("broken", function* (ctx: Context) {
const id = crypto.randomUUID(); // Different on replay!
yield* ctx.run(processWithId, id);
});
// CORRECT — derive ID from stable inputs
resonate.register("fixed", function* (ctx: Context, orderId: string) {
yield* ctx.run(processWithId, `${orderId}-step-1`);
});
```
---
## Server Health Checks
```bash
# Is the server responding?
curl -sf http://localhost:9090/metrics | head -3
# How many requests has it processed?
curl -s http://localhost:9090/metrics | grep resonate_request_total
# Snapshot of all internal state (promises, tasks, messages, timeouts)
curl -s -X POST http://localhost:8001 \
-H "Content-Type: application/json" \
-d '{"kind":"debug.snap","head":{"corrId":"snap","version":""},"data":{}}' | jq '.data | keys'
```
---
## Quick Fixes
| Problem | Quick Fix |
|---------|-----------|
| "Same ID returns old result" | Expected — idempotency. Use a new ID for a new execution |
| Worker keeps restarting | Check `journalctl -u resonate -f` for crash logs |
| Tasks re-dispatching repeatedly | Worker not calling `task.fulfill`. Check for unhandled errors |
| HITL promise never resolves | Verify the resolve/reject/cancel call uses the correct promise ID and value encoding |
| Sleep timer doesn't fire | Confirm the server is running and a worker is connected — durable sleeps fire on the server's background processing loop |
| Database locked errors | Only one server process per database file. Kill duplicates |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |