Systemorph/MeshWeaver / .claude/skills/sigsegv
sigsegv
VerifiedTriage a host crash — exit=139 SIGSEGV, exit=134 SIGABRT, a createdump core, a trx that says "N passed" over a dead process, a pod crash-looping with lastExit=139. Use whenever a test host or portal dies on a signal instead of failing an assertion. In this codebase a crash is almost never a runtime bug: it is async on a turn-based scheduler, or a disposal whose wait was not a subscription. Read the exit code before naming a cause. Grounded in DebuggingNativeCrashes.md, AlcLeaseRegistry, the #613 teardown inversion.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 28.0 KB |
SKILL.md
View on GitHub---
name: sigsegv
description: 'Triage a host crash — exit=139 SIGSEGV, exit=134 SIGABRT, a createdump core, a trx that says "N passed" over a dead process, a pod crash-looping with lastExit=139. Use whenever a test host or portal dies on a signal instead of failing an assertion. In this codebase a crash is almost never a runtime bug: it is async on a turn-based scheduler, or a disposal whose wait was not a subscription. Read the exit code before naming a cause. Grounded in DebuggingNativeCrashes.md, AlcLeaseRegistry, the #613 teardown inversion.'
user-invocable: true
allowed-tools:
- Read
- Bash
- Grep
- Edit
---
# /sigsegv — Read a host crash as async-or-dispose, not a runtime bug
A host that dies on a **signal** does not fail a test — it fails the shard. No assertion, no stack
in the trx, often no trx at all. The instinct is to suspect the runtime. Resist it: in this
codebase the crashes that were ours came from exactly two places, and both are the same mistake
wearing different clothes —
> **either `async`/`await` is involved, or a disposal-path observable was not properly
> subscribed.**
🚨 **And of those two, it is MOSTLY THE SECOND: a dispose nobody waited for.** Teardown here is
reactive — `Dispose()` *starts* the work and returns; the completion arrives later on `Disposed`,
`DisposalCompleted`, or a `StreamEndedEvent`. So "disposed" and "finished disposing" are different
moments, and every crash in Family B is something acting during the gap: unloading an ALC, freeing a
pool, tearing down a hub, while the previous phase is still running. **Start by asking which wait
was skipped**, not which line faulted.
The gap is dangerous in both directions, which is why it is worth naming rather than filing under
"ordering":
- **Nobody subscribed to the completion**, so teardown ran ahead of it → a use-after-free, and the
crash lands in whatever happened to still be executing (Family B).
- **Somebody raced the completion with a timer** — `Task.WhenAny(done, Task.Delay(…))` — and then
acted on "the wait expired" as if it were "the work finished" (Family A). *Prefer retaining a
resource over acting on a guess.*
- **The completion had nowhere to land.** The waiter is not wrong and not slow: it is holding a
subscription to something already torn down, so it receives no value, no completion and no error,
and only its own timeout ever fires. Seen 2026-08-28: a compile-overlay self-heal recycled an
instance hub with `DisposeRequest` while a live subscriber was mid-render; the terminal
`StreamEndedEvent` was dropped (`no synchronization hub found`) and the render sat 37.7 s of total
silence until its 50 s budget expired. **A burst of work then dead silence is this shape** — see
[/debug](../debug/SKILL.md); it is a missed observation, not a slow path.
A timer race deciding "the wait expired, unload anyway"; a `Dispose()` that blocks; an unload
ordered before the drain; a subscription whose `IDisposable` was discarded; a `Subscribe(onNext)`
with no error arm. **Look there first** — before the dump, before the GC, before "flaky CI".
> Canonical references — the doc is the forensics, this skill is the triage:
> - [DebuggingNativeCrashes.md](../../../src/MeshWeaver.Documentation/Data/Architecture/DebuggingNativeCrashes.md) — fetching and reading a core dump (do not re-derive it here).
> - [WritingTests.md](../../../src/MeshWeaver.Documentation/Data/Architecture/WritingTests.md) → "Reading a CI Failure" — what the run actually carries.
> - [ControlledIoPooling.md](../../../src/MeshWeaver.Documentation/Data/Architecture/ControlledIoPooling.md) · [HubDisposalModel.md](../../../src/MeshWeaver.Documentation/Data/Architecture/HubDisposalModel.md) · [/async](../async/SKILL.md) Rule 1a.
## Step 0 — Is it native at all? `139` is not proof
`139 = 128 + 11`, and `.github/workflows/dotnet-test.yml` classifies it for you
(`[CI] <name> exit=139 SIGNAL SIGSEGV`). But **`createdump` also fires on unhandled MANAGED
exceptions**, and the spew is identical. 2026-08-26: the control instance crash-looping with
`lastExit=139` + a core dump was read as a native fault and a plausible-looking disposal defect was
shipped as "the prod fix". One line of the actual dump said otherwise:
```
[createdump] Unwind: exception type System.TypeInitializationException
```
It was a `MissingMethodException` from a binary-breaking record change. **Get the exception type
before naming a cause:**
```bash
kubectl -n <ns> logs <pod> --tail=2000 | grep -iE "Unwind: exception type|MissingMethod|TypeInitialization"
```
| marker | meaning |
|---|---|
| `exit=139` | SIGSEGV — *or* an unhandled managed exception routed through `createdump`. Check. |
| `exit=134` | SIGABRT — runtime abort / `FailFast`. An `AccessViolationException` on a non-null-but-unmapped pointer lands here. 🚨 **Corrected: that is NOT by itself use-after-unload.** Sighting #18 (2026-09-21) is this exact shape — AV at `0x3001000048`, fail-fast, `134` — in a process with **three** collectible `NodeAssemblyLoadContext`s resident, all Alive, none unloading, zero retired contexts — the bad word is read through a **dangling static base** of a collectible generic instantiation, not through a freed context (fact 6, DebuggingNativeCrashes.md *sighting #18*; its first reading said "one ALC, 133 modules" from a roots census and a module-name count). Read the ALC census as MANAGED state — handles, not roots — before naming a cause. 🚨 And `NT_SIGINFO` on a `134` describes the **abort** (`signo 6`, `si_code 0`, `si_addr 0x0`), not the fault; the page-fault `ucontext` is on the alternate signal stack. |
| `exit=124` / `137` | **not a crash** — CI's 8 m wall-clock cap killed a hang. Never appears locally (macOS ships no `timeout`). |
| `exit=2` | xUnit v3's `AppDomain.UnhandledException` handler calling `Environment.Exit(2)`. The summary prints all green and the trx is clean. Grep the shard log for `FATAL ERROR`, not "catastrophic". |
**The crashing PROJECT is meaningful; the crashing TEST usually is not** — the signal lands wherever
the process happened to be. Where the evidence lives:
- **The trx.** For any exit it cannot explain, the shard runs `.github/scripts/record-host-crash.py`
and writes a `<project>.HOST_CRASHED` failure into it, so no reporter can announce "N passed" over
a dead process (`MeshWeaver.Content.Test` did exactly that, #2495). Captured output is in
`<Output><TextMessages>`, **not** `<StdOut>` — finding `StdOut` empty says nothing.
- **`collected-logs/_meshweaver-test-trace.log`.** The only log surviving a killed host.
`TEST_START` / `TEST_END` window markers and `[FAULT]` records with stacks, joinable by `pid=`.
A `TEST_START` with no `TEST_END` names the test a killed host was stuck in.
🚨 Records are rate-bounded — `grep FAULT-BUDGET` before reasoning from an *absence*.
- **`test-logs/teardown-stragglers.log`** (`test/MeshWeaver.Fixture/TeardownStragglerCapturer.cs`) —
`FirstChanceException` dumps every straggler with a stack, **including the caught ones invisible
everywhere else**. This is the diagnostic for "all tests pass, host exits non-zero". Run the
native host the way CI does (`dotnet <Suite>.dll -trx …` from the bin dir); `dotnet test` is not
equivalent.
## Step 1 — Is it OURS? Two fingerprints, and only one of them is
Both are SIGSEGV. They are different bugs and the dump distinguishes them in one line —
`si_code` and `si_addr` from `NT_SIGINFO`:
| | fault address | reading | verdict |
|---|---|---|---|
| **Use-after-unload** | `si_code = SI_KERNEL`, `si_addr = 0`, faulting register holds a **non-canonical** value (e.g. `rax = 0x0074007300200022` — UTF-16 text where a pointer belonged) | a **#GP on a non-canonical pointer**, not a null deref — freed-and-reused memory | **OURS.** Family A below. |
| **Zeroed MethodTable header** | `si_code = 1` (`SEGV_MAPERR`), `TRAPNO=14`/`ERR=0x4`, `RIP` inside file-backed `libcoreclr`, instruction reading a MethodTable field off a register that is `0` — `si_addr` is the FIELD OFFSET, so `0x0` for `MT->m_dwFlags` (`mov ecx,[rax]`, `RAX = 0`) and **`0x4` for `MT->m_BaseSize`** (`mov 0x4(%rax),%esi`, sighting #10). The register allocation and the frame both vary; the dereference does not | one 8-byte object header reads as exactly zero while its block stays coherent | **CoreCLR / upstream.** Not ours. |
| **A corrupt reference, surfacing in MANAGED code** *(sighting #18)* | exits **134**, not 139. `NT_SIGINFO` is the abort; the real fault is `TRAPNO=14`/`ERR=0x4` with `CR2` **non-null** (`0x3001000048`), `RIP` in **JIT-compiled code**, on an application thread with a full managed stack, and `Unwind: exception type` ×6 in the core | the static *base* (`m_pGCStatics`) of a collectible generic instantiation points at a statics box that no longer exists — a live `MiddlewareDeclaration` sits at the address, so its MethodTable word is what the code read; the type's context is Alive and its `LoaderAllocator.m_slots` holds a box for every sibling instantiation but not this one | **CoreCLR / upstream. A candidate mechanism with a deterministic repro reproduces this state; it is not proven to be #18's cause.** A thread EXIT frees its collectible thread-static handles against the allocator that owns each TLS index NOW, so an index reused after an unload frees an arbitrary slot (here a GC-statics box) of a live context: [CollectibleThreadStaticHandleReuse.md](../../../src/MeshWeaver.Documentation/Data/Architecture/CollectibleThreadStaticHandleReuse.md). Not the #12–#16 unload mechanism. |
🚨 **Within a `si_signo = 11` / `SEGV_MAPERR` record, `si_addr` is NOT part of the fingerprint** — it
is only the offset of whichever MethodTable field the faulting code happened to read. Sighting #10
faults at `si_addr = 0x4` and is the same bug. Match on *"a MethodTable word that is exactly zero"*,
never on the literal address. 🚨 **That rule does not extend to a `SIGABRT` record, and row 3 is why:**
when `si_signo = 6` the `si_addr = 0x0` is the *abort's* and says nothing about any dereference, so
reading it as a MethodTable field offset produces this family's fingerprint out of the wrong event.
Check `si_signo` before you interpret `si_addr` at all.
The second one is the FutuRe family. 🚨 **And the family has a second form, so "no zeroed MethodTable
word ⇒ not this family" is not a verdict.** Sighting #18 (2026-09-21, `10.0.12`) faulted in
`ReplaySubject<__Canon>…Subscription.Dispose()+0x3ac` — managed code, on the hub's own
`MessageService.DrainLoop` thread, inside the current instance's teardown — because a GC-reference
static held `0x7fea1704ccf8`, the MethodTable of the then-DI-container's `MiddlewareDeclaration`
(Autofac, since removed — MeshWeaver#7145), which is in no GC
segment. **Three collectible contexts were resident and none was unloading** (the first reading said
*none at all* — a roots census), so the 2026-09-11 overlap diagnosis and the fixes built on it could
not have prevented it: the static base was dangling while its context lived (fact 6, read twice —
JIT-helper layout off the core and the DAC — agreeing). Full entry, controls and the ClrMD recipe:
DebuggingNativeCrashes.md, *"sighting #18"*. 🚨 **Corrected 2026-09-11: the "collectible-ALC hypothesis
falsified three ways" line that stood here was unsound.** Its three arguments (RIP in file-backed
runtime code; a freed `LoaderAllocator` yields a non-null *unmapped* pointer; a free-list item has no
ALC) exclude only a freed collectible *MethodTable* being dereferenced — the zeroed word is the header
of an ordinary default-context object — and its `alc=1` support counts `AssemblyLoadContext.All`, which
drops a context the moment `Unload()` is called. Read for the MANAGED state (ClrMD, not native SOS),
every retained FutuRe dump (#12–#16) held **3–7 `NodeAssemblyLoadContext`s mid-unload** while the next
instance was built or run, and the zeroed object is the **garbage of a disposed hub** (its DI child
registry, STJ polymorphic metadata, `TypeRegistry`) — see DebuggingNativeCrashes.md, *"the MANAGED view
of sightings #11–#16"*. The GitSync pair (#10/#11, `GetCodeInfo`) is a different branch: no collectible
context at all, an interior stale reference. 🚨 **When an unload does not finish, gcroot the
`LoaderAllocator`, never the context** (the runtime's own strong handle always holds an unloading context).
Measured 2026-09-11: the one strong holder of FutuRe's unfinished unloads was **System.Text.Json's static
`ReflectionEmitCachingMemberAccessor` cache** (a `DynamicMethod` accessor's scope holds the collectible
type; 1 s sliding expiry on a 200 ms timer), so contexts were freed when STJ's timer fired — during the
next mesh. Evicted now on `Unloading` (`JsonMemberAccessorCacheEviction`); the DI container keeps no
such cache (`MeshServiceProvider`). Handle type
`(10)` in gcroot output is `HNDTYPE_WEAK_INTERIOR_POINTER` — weak, not a root. And **do not "fix" it by
disabling concurrent GC** — that was tried
(#1274), changed nothing measurable, and was removed. Read the instruction + registers, not the
function name: the frame moved across `background_sweep` → `plan_phase` → `find_first_object` →
`background_mark_simple1` (2026-09-03) while the fault did not. 🚨 **And sighting #10 (2026-09-06)
left `gc_heap` entirely** — `LCGMethodResolver::GetCodeInfo+0x1f7`, `si_addr = 0x4` because the null
MethodTable is read at offset `+4` — so a `gc_heap` frame is a *symptom* of the family, never its
definition. The portable fingerprint is **a MethodTable word that reads exactly zero**, whoever
dereferences it. 🚨 **And the frame REVISITS — it is not a progression.** Sightings #11/#12
(2026-09-07/08) were read as a pair: one is `GetCodeInfo+0x1f7` with `si_addr = 0x4` on a mutator
inside the JIT, the other is back in `background_sweep()+0xa61` with `si_addr = 0x0` on a dedicated
BGC thread — same runtime binary, 33 hours apart, `[R15]` reading zero in both. So do not read #10's
move out of `gc_heap` as the family migrating toward the LCG/serialization workload; that was a
sample of one. 🚨 **Nor is it a property of one runtime build.** Sighting #15 (2026-09-10) ran on
**`10.0.12`**, libcoreclr build-id `79945f51…` — a different binary from #4–#14's `10.0.11` /
`989b56df…` — so "wait for the next patch" is not a plan. (Its "live, referenced" block was later shown
to be garbage: the 96-byte referrer is itself reachable from no GC root.) 🚨 **Read a dump for who was
ACTIVE, not only for the faulting frame:** the evidence of an overlap is on the other threads and in
the ALC census (`AssemblyLoadContext._state` through the GC handles), never in the GC thread that
trips over the result.
Measured base rate on Plugins CI: **0.74 %** over 1,197 runs (2026-08-29 → 09-03,
denominator = non-cancelled runs) and **1.17 %** over 343 runs (2026-09-06 → 09-08, denominator =
runs whose portal-host shard reached a verdict) — **different denominators, so not a trend**; both
are `Portal hosts (shard 1)` only, `main` included, and the affected suites are
`MeshWeaver.FutuRe.Test` **and `MeshWeaver.GitSync.Test`**. Full table:
DebuggingNativeCrashes.md.
🚨 **A newer runtime is not the fix (2026-09-11).** dotnet/runtime#131708 — the `release/10.0`
backport of the GC hole behind dotnet/runtime#131267, whose frame is sightings #10/#11 — ships in
`10.0.12`, and the family crashed on `10.0.12` twice (#15, #16, build-id `79945f51…` read out of the
crashed process). With every run's runtime read from its own log: `10.0.11` **8/444 = 1.80 %**,
`10.0.12` **2/275 = 0.73 %** — rate ratio 0.40, exact 95 % CI 0.04–2.02, so even a *reduction* is not
established; that takes ≈2,030 runs per arm (≈2026-09-24). And the deployed portals do not run
`10.0.12` at all: both images are framework-dependent on the Aug-12 `memex-portal-ai-base`.
🚨 **One more non-ours case, specific to test hosts:** `exit=139` in a process where **ClrMD** runs
anywhere means suspect the DAC first. `DataTarget.Dispose` dlcloses `libmscordaccore.so` without
deleting a process-global `pthread_key` whose destructor points into DAC code — any thread exiting
later segfaults *in whatever test is running*. Pinned by
`test/MeshWeaver.Hosting.Monolith.Test/ClrMdDacPin.cs`. The crashing test name is meaningless here.
## Family A — an ALC unloaded out from under live code
Every dynamic NodeType recompile mints a collectible `AssemblyLoadContext`. Unload one while a
thread can still enter its code and you get a genuine native fault. **The canonical crash of this
codebase**, CI run `32713409169`: a dedicated thread faulted on its **first managed call**, taking
the prestub into `UnsafeJitFunction` and dying in `LCGMethodResolver::GetCodeInfo`.
🚨 **That last frame is now known to have been misread, and the correction matters more than the
story.** Measured 2026-09-06 against the `10.0.11` symbols (DebuggingNativeCrashes.md, sighting #10):
`LCGMethodResolver::GetCodeInfo+0x1f7` allocates nothing and touches no `LoaderAllocator` — it is
`memcpy(code, dataArray->GetDataPtr(), codeSize)`, reading the **MethodTable word of a managed
`byte[]`** that holds a dynamic method's IL. A fault there says *that array's header is garbage*, not
*an ALC was unloaded*. So do **not** reach for `AlcLeaseRegistry` on this frame; read `si_code` /
`si_addr` and the object at the base register first. The defect the `✅` below fixes is real and the
shape is worth knowing — but this frame is not the evidence for it.
**The cause was a timer race in a `Task`-returning override** — async *and* an unsubscribed disposal
signal, in one defect:
```csharp
// ❌ parks the grain turn, then races a timer to decide whether the work happened
var done = hub.DisposalCompleted.FirstOrDefaultAsync().ToTask(ct);
if (await Task.WhenAny(done, Task.Delay(TimeSpan.FromSeconds(5))) == done)
loadContext.Unload(); // …and if the timer won? unload anyway? THAT is the bug
// ✅ subscribe; the turn returns immediately and the work belongs to the signal
hub.DisposalCompleted
.Take(1) // unsubscribes on first emission — no rooted subscription
.Catch<Unit, Exception>(ex => { logger.LogError(ex, "disposal faulted — KEEPING its context"); return Observable.Empty<Unit>(); })
.Subscribe(_ => UnloadContextIfSafe(reason, grainId));
return Task.CompletedTask;
```
The ✅ is live at `src/MeshWeaver.Hosting.Orleans/MessageHubGrain.cs:784` (the subscription at
`:844`). With a timer you must answer "what if the wait expired?", and the tempting answer is the
crash. With a subscription there is no branch: **the callback runs when the signal says so, or it
never runs — and "never runs" means the context is simply retained.**
> **Prefer retaining memory over acting on a guess.** A retained context costs memory until process
> exit; an unload with a live user costs the process.
Three things enforce this, all current:
- **`AlcLeaseRegistry`** (`src/MeshWeaver.Mesh.Contract/Threading/AlcLeaseRegistry.cs`) — per-ALC
in-flight accounting plus a reactive quiescence signal. `UnloadWhenQuiesced` reports `false` and
**leaves the context loaded** rather than unloading on timeout. Used by
`ScriptCompilationService` when a recompile evicts its predecessor on a *live* silo.
- **`MeshTeardownSignal`** — the mesh-wide equivalent at silo stop. It is mesh-wide and says nothing
about one context being retired, which is the gap `AlcLeaseRegistry` closes.
- **`Take(1)`** — a subscription that never completes roots whatever the callback closes over. See
Family B.
## Family B — a dispose nobody waited for *(this is most of them)*
Also called teardown phase inversion, but that names the symptom. The cause is always the same: a
phase treated `Dispose()` as if it had finished the work, when it had only started it. Find the
completion signal that exists — `Disposed`, `DisposalCompleted`, `StoppedAsync`, the drain — and ask
who was supposed to be waiting on it.
**#613, root-caused 2026-08-08.** Node ALCs unloaded in `DisposeImpl()` — *before*
`SignalDisposalCompleted()` — while in-flight pooled render work was only joined *after*
`DisposalCompleted`. In that window a Layout-pool leaf ran a cast over a NodeType-compiled `T` whose
`LoaderAllocator` had just been freed → `AccessViolationException` → SIGABRT. **The evidence was a
symbolized managed stack in the job log, not the dump** — when a crash prints `Fatal error.` with a
managed stack, read the job log first; the 185 MB dump downloads at ~30 KB/s and is usually
unnecessary.
The ordering traps that produce this shape:
- **"Registered first ⇒ stops last" is FALSE on an ASP.NET host.** `GenericWebHostService` is
registered by `WebApplication.CreateBuilder`, strictly before anything the mesh builder can
register — so **Kestrel stops LAST**, and the mesh was being drained while the portal still
served. Nine separately-filed issues, one root. The fix is
`IHostedLifecycleService.StoppedAsync`, invoked only after every `StopAsync` has returned:
`src/MeshWeaver.Hosting/MeshTeardownHostedService.cs` (its `StopAsync` is deliberately
`Task.CompletedTask`). Pinned by `MeshTeardownRunsAfterEveryOtherHostedServiceTest`.
- **The Orleans silo is itself a hosted service**, so it stops before the mesh drains:
`silo stops → grains deactivate → every Unload() → only THEN the mesh drains`.
`IoPoolSiloTeardown` subscribes at `ServiceLifecycleStage.First` ⇒ **stops LAST**, and composes
`registry.Disposed` into **one** `ReactiveCompletion.ObserveCompletion(reportLateFault, ct)` at
the `ILifecycleObserver` boundary — the only sanctioned place a Task appears, and never Rx's
`.ToTask()` (forbidden repo-wide, 2026-08-30): that bridge resumes its caller INLINE on the
signalling thread — here, the very lifecycle thread the unload is waiting on.
- **A blocking `Dispose()` is not the fix.** Making `IoPool.Dispose()` join first *hangs* — `using
var pool = …` in an async method runs Dispose on a ThreadPool thread and blocks it for the whole
`DrainTimeout`. An 18-core dev box hides it completely. The shape that works:
**`Dispose()` cancels and returns; the WAIT lives on `Disposed`** (an `AsyncSubject`, so a late
subscriber still gets the report) — `src/MeshWeaver.Mesh.Contract/Threading/IoPool.cs:76`.
- **`CancellationTokenSource.Cancel()` is SYNCHRONOUS** — it runs every registered callback inline
on the caller's thread, and here those callbacks are entire downstream teardowns. Called from the
teardown thread it runs arbitrary application cleanup unbounded; one leg that never returns parks
mesh teardown silently forever (`exit=124` with `DISPOSE_INVOKED` and no `DISPOSE_DONE`, #2394).
- **A leaf that ignores the pool token holds its permit** — `Drain()` then burns its full 30 s
`DrainTimeout` and reports a leak. Signature: a test failing **in teardown** with
`teardown DIRTY — N pooled I/O leaf(s) still running` and a dispose of **~32 000 ms**. The named
test is the victim, not the defect.
### The subscription mistakes that turn into crashes
- **A discarded `IDisposable` is a leak, not a style slip.** `Observable.Timer(…).Subscribe(…)` whose
return value is dropped leaves a pending entry on the process-wide `TimerQueue` — a strong GC root
holding the hub alive past disposal. **Seven sites of this one defect class**, found months apart;
every one was discovered by a sampling probe firing at random on an unrelated PR.
- **`Subscribe(onNext)` with no error arm is a host kill**, not a missing log. Rx's default `onError`
is `Stubs.Throw`, which rethrows on whatever thread carried the fault — a timer thread, unhandled,
process dead (#2468).
- **An Rx subscribe-time throw goes to the SUBSCRIBE CALLER, not `OnError`** — so every `.Catch` in
the chain is blind to it, and if the caller is a scheduler it is unhandled. Wrap the leg in
`Observable.Defer`: Defer's *factory* exceptions are forwarded to `OnError`.
- **Hosted services must be stopped before the mesh is disposed**, in reverse start order — a service
with an in-flight `Observe` at test end leaves a pending hub callback (PR #432).
## Procedure
1. **Classify the exit.** Read the `[CI] <name> exit=<n>` marker and the `HOST_CRASHED` trx entry.
`124`/`137` ⇒ a hang, go to [/debug](../debug/SKILL.md). `2` ⇒ grep `FATAL ERROR`.
2. **Rule out a managed exception.** `grep "Unwind: exception type"` (pod logs) — if it names a type,
you are done with the dump and you have a different bug.
3. **Read the trace log, and check it is complete.** `grep FAULT-BUDGET` first, then the `[FAULT]`
records and the `TEST_START`/`TEST_END` window around the death.
4. **Ask which PHASE** — read the bottom of the stack, not the top. A `Build`/`CreateHub` bottom is
construction; a dispose cascade is teardown. The FutuRe crash was framed as a teardown race for
weeks while the dump showed construction. 🚨 A method's *name* is not a phase:
`SubscribeToOwnDeletion` is a `.WithInitialization(...)` hook.
5. **Only now, the dump** — and only for the `si_code`/`si_addr` discriminator of Step 1.
🚨 **macOS traps** (from a run that cost real time):
- **`dotnet-dump analyze` REFUSES Linux dumps on macOS.** There is no flag. But the file is a
plain ELF64 core, so parse it yourself — the faulting frame is ~10 minutes of `struct.unpack`.
A container is needed only for the **managed** (DAC) commands.
- **`createdump` prints the crashing thread id in HEX** — `Crashing thread 1684` is `0x1684` =
5764. Looking up 1684 finds nothing and invites the wrong conclusion.
- **`NT_PRSTATUS` is the signal handler, not the crash** (createdump runs ~20 s later, from
inside the handler). Recover the faulting registers from the `ucontext` on the alternate signal
stack. For the same reason **post-hoc memory reads are stale**; registers at fault time are
authoritative.
- **`libcoreclr.so` as shipped is stripped** — resolve against the build-id `.debug` from the
symbol server, using the runtime named in `collected-logs/symbols-<Project>/_runtimes.txt`. A
mismatched patch yields no symbols at all and looks like the technique failing.
6. **Prefer a deterministic repro over more forensics.** A stack says *where* the process died, not
*why*. The cheapest proofs on record are ordering tests, not dumps: record
`UnloadNodeContexts` vs `DrainAll()` during teardown; park a stub leaf that exits only on its own
token and assert `DrainAll()` returns 0. A negative control — revert only the fix lines, watch the
test fail, restore, watch it pass — is what turns "plausible" into "verified".
🚨 **Do not re-run to see if it reproduces.** A crash is a real race; a re-run passing on the
identical commit means you won the timer race, not that the code is clean.
## Checklist — before you call a crash fixed
- [ ] The exit code was classified, and a managed exception was ruled **out** (not assumed away).
- [ ] The fingerprint was read: `si_code`/`si_addr` say ours (non-canonical #GP) vs CoreCLR GC
(`si_addr = 0x0` in `gc_heap`) vs the ClrMD DAC. No unload fix is being aimed at a GC fault.
- [ ] No `await` / `Task.WhenAny` / `Task.Delay` / `.ToTask()` decides whether a disposal happened.
The wait for disposal is a **`.Subscribe(...)`** on a completion observable — never a poll, a
timeout race, a `TaskCompletionSource`, or a fixed delay.
- [ ] Every unload is gated on a **positive** quiescence signal (`AlcLeaseRegistry.UnloadWhenQuiesced`
/ `DisposalCompleted`), with **no "expired, unload anyway" branch**. Retention is the
acceptable outcome.
- [ ] `.Take(1)` on the disposal subscription; its `IDisposable` is not discarded; the fault arm is
**fluent** (`.Catch`), not a `try`/`catch` around `Subscribe`.
- [ ] Every `Subscribe` on the teardown path has an **error arm** (no `Stubs.Throw` host kill).
- [ ] Nothing blocks in a `Dispose()`; no `Cancel()` is issued from a thread teardown needs to keep
moving.
- [ ] Ordering holds: hosted services stopped → `StoppedAsync` → drain → unload. Not registration
order.
- [ ] A deterministic repro or ordering test pins the cause — and no bound was raised to make it
pass ([AGENTS.md](../../../AGENTS.md) → "No band-aids").
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |