Systemorph/MeshWeaver / .claude/skills/testing
testing
VerifiedWrite and run MeshWeaver tests, and triage a red CI run. Use when adding a test, running one locally, or reading a CI failure. The house standards are non-negotiable: no mocking of IMessageHub / IMeshService / core interfaces (use the real mesh test bases), never Task.Delay to wait for propagation (wait on the condition), never assert an exact change-event count on a change feed, and never re-run a test to see whether it was a flake. Also covers the xUnit v3 configuration, why methodTimeout does not bound fixture or teardown wedges, and how to read a shard artifact (trx TextMessages, the trace log that survives a killed host, the CI exit markers).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 12.6 KB |
SKILL.md
View on GitHub---
name: testing
description: 'Write and run MeshWeaver tests, and triage a red CI run. Use when adding a test, running one locally, or reading a CI failure. The house standards are non-negotiable: no mocking of IMessageHub / IMeshService / core interfaces (use the real mesh test bases), never Task.Delay to wait for propagation (wait on the condition), never assert an exact change-event count on a change feed, and never re-run a test to see whether it was a flake. Also covers the xUnit v3 configuration, why methodTimeout does not bound fixture or teardown wedges, and how to read a shard artifact (trx TextMessages, the trace log that survives a killed host, the CI exit markers).'
user-invocable: true
allowed-tools:
- Read
- Bash
- Grep
- Edit
---
# /testing — real meshes, real conditions, one run
> Canonical references:
> [WritingTests.md](../../../src/MeshWeaver.Documentation/Data/Architecture/WritingTests.md) ·
> [TestStateIsolation.md](../../../src/MeshWeaver.Documentation/Data/Architecture/TestStateIsolation.md) ·
> [CqrsAndContentAccess.md](../../../src/MeshWeaver.Documentation/Data/Architecture/CqrsAndContentAccess.md) ·
> [SatelliteEntityPatterns.md](../../../src/MeshWeaver.Documentation/Data/Architecture/SatelliteEntityPatterns.md).
## The standards
**No mocking.** Use the real mesh test bases — never mock `IMessageHub`, `IMeshService`, or core
interfaces (#1810).
**Always `run_in_background: true`** for test runs (they take minutes).
**Never `--verbosity minimal`** when tests may fail — it hides stack traces.
**Never `Task.Delay` to wait for propagation.** A fixed sleep races CI load: too short → flakes, too
long → wastes minutes across the suite. Wait on the actual condition via
`stream.Where(...).FirstAsync().Timeout(...)`. When the source is request/response (not an
observable), wrap the re-query in
`Observable.Interval(50.Milliseconds()).StartWith(0L).SelectMany(...).Where(predicate).FirstAsync().Timeout(...)`.
Hand-rolled `while + Task.Delay(50)` poll loops are forbidden. Sanctioned `Task.Delay` uses: forcing
distinct timestamps for sort assertions, and "wait to confirm nothing happened" negative tests where
there's no positive signal to filter for. See WritingTests.md → "Polling loops around QueryAsync".
A negative that HAS a later positive fact to end on (a sentinel through the same pipeline, the owner's
answer to a follow-up request, a response that follows the work) uses
`.Should().Within(t).NotEmitBefore(settled)` instead of `NotEmit(window)` — a passing `NotEmit` spends
its whole window, 36 s on CI at `TestTimeouts.Quick` (#6817). WritingTests.md → "A negative assertion
that spends a fixed window when a settled signal exists".
**Never assert "exactly N change events"** on a stream backed by pg_notify or any change feed that
can race the initial-snapshot path. Filter on the emission shape (e.g.
`.Where(c => c.ChangeType == QueryChangeType.Initial)`), not the count.
🚨 **A reachability / GC assertion joins `DisposalCompleted` BEFORE it collects.** `Dispose()` only
STARTS a hub's teardown — it posts `ShutdownRequest(Quiescing)` and returns — so on return the hub is
still rooted by its own in-flight shutdown. `WeakReference` + `Dispose()` + `GC.Collect()` therefore
measures the teardown's speed, not the reference graph, and it fails GREEN: #3321's version passed
inside a 485-test project and failed alone on the same binary. Take the weak reference and drop the
strong one inside a `[MethodImpl(NoInlining)]` helper, return the completion observable rather than
the object, await it, then collect. **Suite-green / alone-red with nothing rebuilt is its own signal
— and the PASSING run is the control**: it names the ambient wait the test was silently relying on.
See WritingTests.md → "A reachability assertion that collects before teardown has finished".
**🚨 NEVER re-run a test (single or suite) unless code under test has changed.** Re-running to "see
if it was a flake" hides the bug — flakes are real races. Either fix the race or pin the failure
with a smaller repro; do not retry. The only exceptions: (a) the test harness itself crashed
(MSBuild MSB4166, infrastructure error — a re-run is the same input), (b) the previous run was
killed by the user before completion.
🚨 **`.ToTask()` is FORBIDDEN in tests too** (maintainer, 2026-08-30 — the earlier test exemption is retracted: a Task completed inside an Rx pipeline resumes inline on the signalling thread, still inside the trampoline, so the bridge changes what the test measures). 🚨 **Awaiting the observable DIRECTLY is not the alternative** — Rx's awaiter is an `AsyncSubject<T>` that completes its continuation from inside `OnCompleted` and resumes inline just the same (`InlineResumptionMechanismTest`); this line used to say it was, and `DirectObservableAwaitSites.allow` is the 228-site inventory that advice produced. Assert on the stream through `MeshWeaver.Reactive.Assertions`, or wait through `.Await(ct)` / `.ObserveCompletion(reportLateFault, ct)` — see
[/async](../async/SKILL.md).
**No static collections, in `test/` as much as `src/`** — a `Clear()` added "for test isolation" is
the proof of the bug, not the fix.
## Configuration and its blind spot
xUnit v3 config (`test/xunit.runner.json`): `parallelizeAssembly: false`,
`parallelizeTestCollections: false`, `maxParallelThreads: 1`, `methodTimeout: 316000`.
🚨 **xunit.v3 ignores `methodTimeout`.** The bound is applied by the execution host
(`MeshWeaver.Testing.Xunit.BoundedTestFramework`, declared for every test assembly by
`test/Directory.Build.props`): a test that outlives it fails by name and leaves a dump of the stuck
thread (`TEST_TIMEOUT` in the phase trace). WritingTests.md → "The per-test bound".
🚨 **The bound only covers time spent INSIDE a test method.** A wedge in fixture construction,
class init, or teardown is outside it and runs unbounded — on 2026-08-04 an orphaned local run sat
at a pegged core for 25+ minutes and, together with a leaking e2e container, drove the colima VM to
128 MB free / load average 195, OOM-killing unrelated containers.
So **always give a local test run its own wall-clock cap** — but **not with `timeout`**: neither
`timeout` nor `gtimeout` exists on this macOS host, so `timeout 20m dotnet test …` runs **nothing**
and reports `command not found`. Background the run and hold the deadline yourself against `date -u`
— the five-step shape is in [/worktree](../worktree/SKILL.md) → "A verification step that cannot
fail".
CI has the equivalent at two levels — its runners are Linux, where `timeout` does exist: `timeout
8m` per project inside the shard loop, and `timeout-minutes: 20` on the shard job itself for a wedge
BETWEEN projects (which the per-project cap cannot see). If a run hits either, it is stuck — do not
raise the bound, find what is not completing (AGENTS.md → "No band-aids").
## Running tests
```bash
dotnet build test/MeshWeaver.Hosting.Monolith.Test/MeshWeaver.Hosting.Monolith.Test.csproj
dotnet test test/MeshWeaver.Hosting.Monolith.Test --no-build
dotnet test test/MeshWeaver.Graph.Test --filter "FullyQualifiedName~AccessAssignment" --no-build
```
**Build first, every time**: `dotnet test --no-restore`/`--no-build` against a project this worktree
has never built exits **0 with no output and no `.trx`** — an unmissable-looking pass that ran
nothing. One project per `dotnet build` invocation (several args is an `MSB1008` no-op).
Workflow: run once in background → read failures → fix → run once more.
### In-mesh suites: a node's `Tests` area — run it ONCE, as an activity
🚨 **Rendering a `Tests` area RUNS the suite.** Every `get @<node>/area/Tests` (or `render_area`)
opens a fresh subscription, so polling the area re-runs every live case, files their side effects
again, and can still show a verdict an earlier run cached. The area streams now (every case pending
→ the running case with its elapsed time and output → the verdict), and every case is bounded, so a
hung case fails as `timed out: no verdict within Ns` instead of leaving the page on *Rendering …*.
From a terminal or an agent: `memex tests @<node>` (exit 0 passed, 1 failed, 4 no verdict in time),
or the MCP `run_tests` tool — both drive `MeshOperations.RunTests`, ONE subscription written to a
`TestRun` activity you poll with `get`, which starts nothing. Only a verdict frame counts: a
progress frame (`tests-running`) shows ✔/✖, never ✅/❌ or *N/M passed*. Full reference:
[WritingTests.md](../../../src/MeshWeaver.Documentation/Data/Architecture/WritingTests.md), "A `Tests`
area streams its progress".
## Test triage — when CI fails
**DO NOT run entire test projects.** Iterate one test at a time:
1. Read failed test names from CI logs (`gh run view <id> --log`).
2. `dotnet build <project>.csproj` — in a fresh worktree, skipping this makes step 3 a silent no-op
(exit 0, no output, no `.trx`).
3. `dotnet test <project> --filter "FullyQualifiedName~<TestName>" --no-build`.
4. **No skipping** — CI-only failures catch real timing/state bugs.
🚨 **Read the run's evidence before reasoning about it.** The shard artifact carries three things and
they answer different questions:
- the per-project `.trx` — output lives in `<Output><TextMessages>`, **not** `<StdOut>`; the native
xUnit v3 writer does not use `StdOut`, so finding it empty says nothing;
- `collected-logs/_meshweaver-test-trace.log` — the ONLY log that survives a host killed at the
wall-clock cap, carrying `TEST_START`/`TEST_END` window markers and `[FAULT]` records with stacks,
joinable by `pid=` and timestamp;
- the classified `[CI] <name> exit=<n>` markers.
A host that CRASHED is written into the trx as a `<project>.HOST_CRASHED` failure, so no summary can
report a pass over a dead process. Full map: WritingTests.md → "Reading a CI Failure".
A hub-handler test that hangs, or a message that disappears, is a message-flow problem, not a
timeout problem — see [/debug](../debug/SKILL.md) and
[DebuggingMessageFlow.md](../../../src/MeshWeaver.Documentation/Data/Architecture/DebuggingMessageFlow.md)
first. Never rerun a hung test "to see".
## Test bases and fixtures
- **`MonolithMeshTestBase`** (recommended) — full integration with persistence, messaging, DI; use
`AwaitResponseAsync(request, ...)` for request/response in tests.
- **`HubTestBase`** — message routing / layout tests; assert on the stream, e.g. `await hub.Observe<TResponse>(request).Should().Within(TestTimeouts.Convergence).Emit("the response arrives")`, or wait through the one bridge: `await hub.Observe<TResponse>(request).FirstAsync().Timeout(TestTimeouts.Convergence).Await(ct)`. Never `.ToTask()`, and never a bare `await hub.Observe<TResponse>(request).FirstAsync()` — Rx's own awaiter resumes inline on the signalling thread exactly as `.ToTask()` does.
- **`MeshWeaver.Hosting.Orleans.TestBase`** — the core Orleans cluster machinery (test cluster,
disposal drain, shutdown-race suppression). `OrleansMeshTestBase` is the ONE base a suite
derives from; which cluster it gets is `Bootstrap => MeshBootstrap.Orleans(…)` plus
`SiloConfiguratorType`. The AI-flavoured Orleans rig (the derived fixture and the swappable
chat-client factory) ships with the AI engine in
**MeshWeaver.Plugins**, not here (#2276).
**Use `hub.Observe(...)`, not `RegisterCallback`/`AwaitResponse`** — those overloads are
`[Obsolete]` and deadlock. Tests use `MonolithMeshTestBase.AwaitResponseAsync(...)`.
### DevLogin and access control
`MonolithMeshTestBase` auto-logs in `rbuergi@systemorph.com` as Admin. Available helpers:
`TestUsers.Admin`, `TestUsers.SampleUsers()`, `builder.AddSampleUsers()`.
For per-user access-control tests, use
`accessService.SetCircuitContext(new AccessContext { ObjectId = "...", Name = "..." })` before
creating test data; set `null` after.
### Node types available in a test mesh
From `AddGraph()`: `Markdown`, `Code`, `Group`, `User`, `VUser`, `Role`, `Notification`, `Approval`,
`AccessAssignment`, `GroupMembership`, `PartitionAccessPolicy`, `ActivityLog`, `UserActivity`,
`Comment`, `Redirect`.
`Agent`, `Skill`, `Thread` and `ThreadMessage` are declared by the **AI engine module**, which lives
in MeshWeaver.Plugins (#2276) — a core-only test mesh does not have them.
Custom types: `builder.AddMeshNodes(new MeshNode("MyType") { Name = "My Type" })` in
`ConfigureMesh`.
## Checklist
- [ ] Real test base, no mock of a core interface.
- [ ] Every wait is on a condition (`Where(...).FirstAsync().Timeout(...)`), never a sleep.
- [ ] No exact change-event count asserted on a change feed.
- [ ] The project was built before `--no-build`, and the run produced a fresh `.trx`.
- [ ] Any reachability / GC assertion awaits the owner's `DisposalCompleted` before collecting.
- [ ] A failure was investigated, not re-run.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |