Systemorph/MeshWeaver / .claude/skills/storm
storm
VerifiedDiagnose and CURE storms that wedge the portal — resubscribe/retry storms, NotFound/DeliveryFailure storms, per-instance-hub accumulation, create-in-render loops. A storm makes the process unresponsive → liveness /healthz times out → the pod is pulled from the Service → ingress returns 502 → SIGKILL + restart. Use when a portal pod restarts/502s, memory climbs unbounded, the log volume explodes, or you are reviewing reactive code that could re-fire on every emission. Storms come from exactly two roots: (1) sync-over-async blocking, (2) uncaught / not-forwarded exceptions or delivery failures. Grounded in ErrorPropagationAndWedges, AccessContextPropagation.md; pairs with the /async skill.
/storm — Storms wedge the portal. Cure the root, never the symptom.
A storm is the same cheap operation re-firing without bound — a resubscribe, a retry, a
NotFound re-post, a thread/hub re-activation, a write that fails and is retried. Individually each
is fast; together they starve the scheduler / exhaust memory / pin the action block. The process
stops responding → the k8s liveness probe /healthz times out → the pod is removed from the
Service endpoints → the ingress has no healthy backend → 502 → liveness SIGKILLs (exit 137)
→ restart. Users see a 502 outage. This must never happen.
Canonical references:
- ErrorPropagationAndWedges — the wedges-to-zero principle this skill enforces.
- /async — the sync-over-async + AccessContext half of every storm.
- AccessContextPropagation.md — lost identity → silent write failure → swallowed → retried → storm.
- MeshNodeStreamCache.md — the existing storm-breaker (negative cache / backoff) to build ON, not around.
The two roots (there are only two)
Every storm traces to one of these. Find which, fix THAT:
-
Sync-over-async / blocking.
async/await/Task<T>/.ToTask()/.FirstAsync()(blocking)/.Wait()/.Result/.GetAwaiter().GetResult()/Observable.FromAsync/SemaphoreSlimon a hub action block, grain turn, or Blazor circuit. It parks the scheduler; the awaited message can never be processed → deadlock that presents as a hang. → fix per /async. -
Uncaught / not-forwarded exceptions or delivery failures. A write fails (often because AccessContext was lost across a
.Subscribe/IIoPool hop → RLS denies), a stream errors, a request getsNotFound/DeliveryFailure— and the error is swallowed (.Catch(Observable. Empty),catch {},_ => { /* ignore */ }) or retried (a 1 s ticker, a watchdog, a resubscribe, a.Retry()) instead of surfaced. The amplifier turns one mishandled error into a flood. (The 2026-06-08 outage was an initial-state retry watchdog; the 2026-06-25 outage was thread-hub accumulation + a swallowed-and-retried heartbeat read.)
The cure (one principle)
Surface errors until you reach a boundary where you can error gracefully — then error gracefully. Do NOT swallow, and do NOT retry a "shouldn't happen" state. Forward the error up the reactive chain (
OnError) until it reaches a real sink: an activity's terminalStatus = Failed, the GUI error area /PortalErrorSink, a thread's output cell, a logged warning at a known boundary. Everything before that sink must forward, not absorb.
Band-aids that are FORBIDDEN as "the fix" (they hide the storm): raising a timeout/pool size/retry
count; a watchdog/poller that resubscribes; .Catch(Observable.Empty); catch {}; a Clear()
"for isolation"; a sleep. See AGENTS.md → "No band-aids".
Diagnose a live/just-happened storm (the procedure)
The portal hung silently while wedged (a deadlocked/starved process stops logging), then SIGTERM at shutdown emits a mass-disposal storm that can rotate the pre-hang logs out of the buffer. Work the timeline:
🚨 Break-glass reads. Restart counts, pod events and the previous container's log are exactly the three questions the control instance's Hosting API does not answer yet (measured 2026-09-08:
Hosting/DeploymentStatusandHosting/LogEntryexist as types with zero instances). Take them read-only throughaz aks command invoke, name them as break-glass in what you write down, and never turn the cure into akubectlwrite — a restart is aRestartHosting/InstanceAction. Policy: OperatingFromThePortal.md.
# 1. Did the pod restart, and how? exit 137 + "failed liveness probe" = wedge (not OOMKilled).
az aks command invoke -g <rg> -n <cluster> --command \
"kubectl get pod <portal-pod> -n <ns> -o jsonpath='{.status.containerStatuses[0].restartCount}{\" \"}{.status.containerStatuses[0].lastState}'; \
kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -30"
# 2. Find the SILENCE = the hang window. Per-minute line counts around the kill.
# Zero lines for minutes, then a huge burst in the final minute = hung, then mass disposal.
az aks command invoke ... --command \
"L=\$(kubectl logs <pod> -n <ns> --previous --timestamps 2>/dev/null); \
for m in 45 46 47 48 49 50 51 52 53 54 55 56; do echo -n \"HH:\$m \"; echo \"\$L\" | grep -c \"THH:\$m\"; done"
# 3. What stormed? Source frequency in the wedge/disposal window; storm counts.
az aks command invoke ... --command \
"L=\$(kubectl logs <pod> -n <ns> --previous 2>/dev/null); \
echo \"\$L\" | grep -E 'fail:|warn:' | grep -oE '(fail|warn): [A-Za-z0-9._]+' | sort | uniq -c | sort -rn | head; \
echo -n 'NotFound: '; echo \"\$L\" | grep -c NotFound; \
echo -n 'distinct sync hubs disposed: '; echo \"\$L\" | grep -oE 'sync/[A-Za-z0-9_-]+' | sort -u | wc -l"
# 4. Distinct vs looping. Is it N things created (creation storm) or 1 thing re-firing (loop)?
az aks command invoke ... --command \
"kubectl logs <pod> -n <ns> --previous 2>/dev/null | grep -oE '<the-repeating-token>' | sort | uniq -c | sort -rn | head"
# 5. Is the CURRENT (restarted) container re-accumulating? memory trend + the repeating block.
az aks command invoke ... --command "kubectl top pod -n <ns>; kubectl logs <pod> -n <ns> --since=60m | grep -A1 '<token>' | head -30"
Read the last lines before the silence (grep -v 'THH:<kill-minute>' | tail -50) — the last
thing logged before it froze is the trigger. If those are rotated out, the mass-disposal volume
(distinct sync/ hubs) tells you the scale of the accumulation, and step 5 tells you whether
it's still happening.
Storm taxonomy — tell → cure
| Storm | Tell in logs / code | Cure |
|---|---|---|
| Resubscribe / initial-state retry | Observable.Interval/Timer or .Catch that re-subscribes the source on a "shouldn't happen" state; repeated SubscribeRequest | Delete the retry; let the error surface. If the initial state is genuinely missing, find why it's dropped/errored. |
| Swallow-and-retry | _ => { /* swallow; next tick retries */ } on a per-tick read; .Catch(Observable.Empty) inside a loop | Forward the error to a sink; drop the path from the retry set on terminal failure; don't re-read a NotFound every tick. |
| NotFound / DeliveryFailure | [ROUTE] NotFound repeating; ownerless satellite (_Thread/_Activity/… with no real owner); a path advertised before its CreateNode completed | Never advertise a node path until Create completes (stamp inside .Subscribe). Anchor satellites under a real owner. Lean on the MeshNodeStreamCache negative cache. |
| Fan-out to a dead subscriber (#2426/#2546) | [ROUTE] Stream-routed delivery to 'portal/…' / 'node/…' has no live subscriber at the change-feed rate for hours after a restart/disconnect; senders are per-node owners; "a NACK has no NACK of its own" | Not a retry — the owner's server-side stream is immortal (only an UnsubscribeRequest disposes it). The router's refusal is stamped DeliveryFailure.TargetUnserved, is delivered to a grain-hosted sender over the grain transport, and the owner EVICTS that subscriber's streams (Workspace.EvictClientSubscriptions). Never a negative cache on the delivery path; the LOG is windowed (DeadTargetRefusalLog). |
| Per-instance hub accumulation | thousands of sync/ hubs at disposal; a per-hub Observable.Interval heartbeat that runs until dispose; memory climbing | Ensure per-instance hubs (thread/round/sync) are disposed when their round/thread reaches terminal state, not only at mesh shutdown. A live timer must not outlive its purpose. |
| Create-in-render | StartThread/SubmitMessage/CreateNode inside a .Select/.Subscribe on GetMeshNodeStream(...) that re-fires every emission | Move creation out of the render/subscription; one-shot it (.Take(1)/init) or make it idempotent (deterministic path), guard with DistinctUntilChanged. |
| Sync-over-async deadlock | /healthz timeout with no error; .Result/.Wait()/.ToTask() on a hub/circuit | Per /async: compose IObservable + Subscribe; push the leaf to IIoPool. |
Where to look first (highest-yield)
Storms cluster where an error crosses an async boundary and isn't forwarded — usually because AccessContext was lost (write fails closed → swallowed → retried). Grep for the amplifiers:
rg -n "Catch\(Observable\.Empty|Catch<[^>]+>\(_ *=> *Observable\.Empty|catch *\{\s*\}|/\* *swallow|/\* *ignore|\.Retry\(" src
rg -n "Observable\.Interval|Observable\.Timer\(|RegisterForDisposal\(_ *=> *sub" src # unbounded tickers — does the sub die with its purpose?
rg -n "\.Subscribe\(\s*_? *=> *\{ *\}\s*,\s*_? *=> *\{\s*\}\)" src # write with both arms empty = swallowed failure
For each hit decide: graceful sink (logged/surfaced at a real boundary → OK) or silent swallow / blind retry (→ fix: surface, or stop retrying a terminal state). When the swallowed thing is a write, check the /async AccessContext rule first — the failure is usually a fail-closed RLS denial from a lost identity, and the real fix is to carry the context, then let any remaining error surface.
Checklist before merging reactive code
- No blocking async on a hub/grain/circuit (
.Result/.Wait()/.ToTask()/.FirstAsync()-blocking/Observable.FromAsync/SemaphoreSlim). → /async - Every
.Subscribewrite has an onError that surfaces or logs at a real boundary — never an empty swallow inside a loop/ticker. - No timer/watchdog/
.Catchthat re-triggers a "shouldn't happen" state. If state is missing, fix the source, don't retry. - Writes on
.Subscribe/IIoPool/Observable.Createhops carry AccessContext (else fail-closed → swallow → storm). - Node paths are advertised only after
CreateNodecompletes; satellites anchor under a real owner. - Per-instance hubs / timers are disposed at terminal state, not only at mesh shutdown.
- If a bound exists (top-N, no-retry, sampling), it's
log()-ged — silent truncation reads as success.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 466k | Updated | Claude CodeCodex | Unverified | Install |
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 466k | Updated | Claude CodeCodex | Unverified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |