clice-io/kotatsu / .claude/skills/test-style
test-style
VerifiedHow kotatsu's tests are organized and written — levels (unit, system, integration), the module-first layout, dependency and trust rules, naming, zest checks, determinism. Read BEFORE writing, moving or reviewing any test.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 18.1 KB |
SKILL.md
View on GitHub---
name: test-style
description: How kotatsu's tests are organized and written — levels (unit, system, integration), the module-first layout, dependency and trust rules, naming, zest checks, determinism. Read BEFORE writing, moving or reviewing any test.
---
# kotatsu Test Style
## Levels
A test's level is decided by what it touches, not by the module it tests.
| Level | May touch | Lives in |
| ----------- | ------------------------------------------------------------------------------------------------------ | ----------------------------------------- |
| unit | the process's memory; an event loop, its timers and in-memory transports count as memory | `tests/<module>/unit/` → `unit_tests` |
| system | the operating system: files, processes, sockets, pipes, signals, threads and thread pools, environment | `tests/<module>/system/` → `system_tests` |
| integration | several programs talking over real protocols, black box | `tests/<module>/integration/` → ctest |
- The moment a test opens a file, spawns a process, binds a socket, installs a signal handler or starts a thread, it is a system test. zest's own I/O does not count: printing, and reading or writing snapshots through `zest::snapshot`, are fine in a unit test.
- Compile-time facts that must hold are `ZSTATIC_EXPECT` in a unit case.
- Nothing reaches the network beyond loopback. Tests that need the internet are not part of any suite.
- A target that must build without another, as ipc's codecs without the event loop, is checked by a program linked against it alone, without zest, which needs the loop; so is a check that replaces what a whole process shares, as fs's of remove_all() under failing allocations replaces the global operator new: `tests/<module>/<name>/main.cpp`, run by ctest and Bazel as `<module>_<name>` (label `unit`, or `system` for one that touches the operating system); it prints each failed check and exits with 1.
- zest's own runner check (`tests/zest/integration/`) is integration-level, but it is the one integration test in C++ and CMake rather than TypeScript: it runs from CMake as a bootstrap stage (see Trust), so it needs nothing beyond the build.
## Layout
Tests are grouped by module, then by level. `tests/<module>/` mirrors `include/kotatsu/<module>/`; the modules are `support`, `fs`, `sys`, `meta`, `codec` with one directory per backend (`codec/json`, `codec/toml`, `codec/fbs`, `codec/bincode`, `codec/blob`, `codec/dyn`, `codec/debug`), `cli`, `async`, `ipc` with `ipc/lsp`, `http` and `zest`. `tests/examples/` holds the tests of `examples/`, by the module an example shows (`tests/examples/async/`, `tests/examples/ipc/`): unit tests of what an example builds on, and the example programs run end to end. `tests/harness/` holds what every module's integration tests share.
```
tests/<module>/
unit/<path mirroring the headers>/...
system/... same shape
integration/*.test.ts integration tests, TypeScript; nothing else but drivers/
integration/drivers/*.cpp the programs they spawn, one per file, or one per
directory of sources
<name>/main.cpp a check that is a program of its own (Levels),
in kotatsu_check_programs so that it builds into the
build root beside the runtime DLLs
harness/*.h, harness/*.ts helpers for this module's tests and the modules above,
integration tests' helpers included
CMakeLists.txt kotatsu_add_module_tests(LIBS <the module's libraries>)
kotatsu_add_integration_tests(LIBS ... PROGRAMS ...
DEBUG_ONLY ...) if it has integration/; PROGRAMS are
targets built elsewhere
tests/BUILD.bazel the same for Bazel: each module's harness library,
its <module>_<level>_tests binaries, its drivers and
its <module>_integration test
tests/harness/*.ts what every module's integration tests share
tests/fixtures/ types shared by several modules' tests
tests/snapshots/<suite>/ snapshot files
```
- The tests of header `<module>/a/b.h` are `unit/a/b_tests.cpp`. A large header's tests split by aspect, into `unit/a/b_<aspect>_tests.cpp` or into a directory `unit/a/b/<aspect>_tests.cpp`.
- codec's protocol (the core headers in `codec/`, `codec::repr`, attributes, formats) is tested once, by a kit every backend runs: `codec/harness/visit/kit.h` holds the primitives and `Caps`, and each area (`values.h`, `attrs.h`, `repr.h`, `variants.h`, `probing.h`) registers its cases for a backend. A backend adapts itself in `codec/<backend>/harness/backend.h` and runs each area in `codec/<backend>/unit/visit/<area>_tests.cpp`, one `ZEST_CASE_GROUP` per file; blob, which lays out the struct itself rather than speaking the protocol, has no `visit/` tests. Fixtures only the codec's tests use live in `codec/harness/fixtures/`.
- Build options are checked once, where `tests/CMakeLists.txt` (or the parent module's `CMakeLists.txt`) adds the module's directory. `tests/BUILD.bazel` lists the modules too, building every test binary per module and level, with its snapshots and the files beside its sources as data.
## Dependencies
What each module's headers include, and so what its tests may use:
- `support`; `fs` on support; `sys` on fs; `meta` on support; `cli` on codec's core and meta, privately sys.
- `codec`: its core (the headers in `codec/` itself) on meta; the backends `blob`, `debug`, `dyn`, `toml` and `fbs` on the core; `bincode` and `json` on the core and `dyn`.
- `async` on support, `fs` and `sys`; `ipc`'s codecs (`codec.h`, `codec/`, `framing.h`, `protocol.h`) and its blocking client (`blocking_client.h`) on codec (`json`, `dyn`, `bincode`), the rest of `ipc` on those and async; `ipc/lsp` on ipc; `http` on async and codec `json`.
- `zest` on all of the above that it checks, prints, parses or runs with. Every test includes zest; that is the one exception.
Rules:
- A module's tests include only what their module depends on, plus the harnesses of those modules: `#include "support/harness/throws.h"`, rooted at `tests/`. Shared fixtures in `tests/fixtures/` follow the same rule for the lowest module that uses them. The build does not enforce this; review does.
- Tests use the public API: `include/kotatsu/`, never a header from `src/`, nor anything from `examples/` outside `tests/examples/`.
- Behaviour defined once in the library is tested once. Backends of one protocol share one suite through the module's harness; a backend's own files test only what is specific to that backend.
## Integration tests
- An integration test is a TypeScript file on node's test runner (`node:test`, `node:assert/strict`), run by node directly (type stripping, no build step) and type-checked by `pixi run typecheck`. Its npm packages are the root `package.json`'s devDependencies (`vscode-jsonrpc`, `vscode-languageserver-protocol`, `vscode-uri`, `vscode-languageserver-textdocument`, `fast-check`), installed by `pixi run npm-ci`.
- A driver is a C++ program under test, `integration/drivers/<name>.cpp`, or the sources in `integration/drivers/<name>/` for one too heavy for a single translation unit, linked against the module's libraries by `kotatsu_add_integration_tests`; an example it runs is named in `PROGRAMS`. ctest passes each one's path in `KOTATSU_<NAME>`; a test whose driver is unset or missing fails, it never skips. A driver too heavy to build outside a plain Debug build, and the tests that run it, are named in `DEBUG_ONLY`; MinGW's GCC leaves them out too, its assembler being too slow for them.
- All of a module's `integration/*.test.ts` are one ctest test, `<module>_integration` (label `integration`); it needs nothing from zest and runs beside the other stages.
- `Driver` (`tests/harness/driver.ts`) spawns a driver and hands its stdio to a channel in its protocol: JSON lines (`tests/harness/jsonl.ts`) for a driver that is no JSON-RPC peer; in ipc's harness, a vscode-jsonrpc connection (`connection.ts`), or a raw channel (`raw.ts`, and `session.ts`, which pairs responses with requests and keeps the strays; `jsonrpc_driver.ts` has what jsonrpc_driver's tests share). `Driver.spawn` takes the test's context: a driver the test did not finish with is killed when the test ends, and how it ended printed with its stderr.
- A driver logs one `[<level>] <message>` line each to stderr, ipc's through `test::stderr_logger()` (`ipc/harness/stderr_logger.h`).
- Every test ends with `expectExit(code)`, or `expectSignal(signal)` for a driver a signal ends, which checks how the driver ended (`expectExit` takes a list of codes where a host may end it with either) and fails on a sanitizer report in the driver's stderr, and on a warn or error line the case did not declare with `expectLog(pattern, count)`: a message the driver drops with a warning fails the case that sent it. An allowance is a number of lines, one unless it says more, and never switches the check off; only input that may make a driver log anything takes `Infinity`. The count is checked at exit over the whole run, each line taking the first allowance it matches with one left, so declare an allowance just before the step that logs it, for the lines that step logs.
- Cases are named like zest cases: `snake_case`, shaped `<subject>_<behaviour>`, `_fails` for an error. A file's comment says what its cases have in common.
- No sleeps: wait for the message or the exit that says it happened. The runner's timeout (`--test-timeout` in `kotatsu_add_integration_tests`) bounds each test, driver included.
- A randomized test runs on fast-check through `fuzz()` (`tests/harness/fuzz.ts`): a fixed seed, reported with the shrunk counterexample; `KOTATSU_FUZZ_SEED` and `KOTATSU_FUZZ_RUNS` override the seed and the 100 runs for long runs by hand, which then have no time limit. Otherwise the run still going at 60% of the test's timeout (`FUZZ_TIMEOUT`) is cut short: a failure found by then is reported, shrunk as far as the time allowed, and a leg too slow for every run passes on those it made. A run that stalls fails through `within()` before the test times out, and a run that fails kills its driver and reports its stderr (`Driver.reported`). Strings are drawn from `anyChar`, half ASCII and half any code point, so that quotes, backslashes, control characters and text past ASCII all come up.
- LSP's types are drawn from the pinned metaModel (`ipc/lsp/harness/protocol_values.ts`), which also says whether protocol.h reads a value. Where kotatsu reads LSP otherwise, `ipc/lsp/harness/known_deviations.ts` has an entry marked "bug" or "design" that says how; the fix of a bug deletes its entry, and the tests then check it.
- Behaviour the library owes but does not have yet is a case written for the correct behaviour, skipped with the reason, in words rather than a plan's numbering (`{ skip: "an error response drops its data" }`); the fix removes the skip.
## Trust
zest decides pass or fail with meta's comparisons, prints operands with the debug codec, matches test filters with support's glob patterns, parses its options with cli, drives its worker processes with kotatsu::async and writes their logs with kotatsu::fs. ctest therefore runs in stages, each needing the one before:
1. `zest_bootstrap_unit`, `zest_bootstrap_system`: the tests of what zest relies on, in this process, without the worker pool. Their filters are in `tests/CMakeLists.txt`.
2. `zest_runner`: the worker pool end to end, failed `ZASSERT`s and crash tests, LoopFixture's watchdog, and `run_cli`'s command line.
3. `unit_tests`, `system_tests`: everything.
A test in a bootstrap suite must not judge itself with what it tests: meta's comparison tests use unary checks (`ZEXPECT(eq(a, b))`) or parenthesized plain bools (`ZEXPECT((a.x == 1))`), never a split comparison. A new suite covering something zest relies on joins a bootstrap filter; a renamed one updates it.
## Naming
- Files: `<stem>[_<aspect>]_tests.cpp`, `.cpp` only.
- Suites: the file's path under `tests/`, without the level directory and the `_tests` suffix, its components joined by `_`, and a word dropped when it repeats the word before it. For example, `tests/codec/toml/unit/toml_variant_tests.cpp` is `codec_toml_variant`, `tests/meta/unit/schema/schema_attrs_tests.cpp` is `meta_schema_attrs`, and `tests/async/unit/runtime/when/cancel_tests.cpp` is `async_runtime_when_cancel`.
- One suite per file. Suite names are unique across both binaries, since the suite name is also its snapshot directory: when one header has both unit and system tests, the system file names its aspect (`io/loop_threads_tests.cpp` beside the unit `io/loop_tests.cpp`).
- Cases: `snake_case`, shaped `<subject>_<behaviour>`. No suite prefix, no numbering. A case about an error ends in `_fails`, a thrown exception included. Spell it `roundtrip`, not `round_trip`.
- Fixture types are PascalCase.
## Namespaces
- A test file puts its fixtures and suite in an anonymous namespace inside the namespace it tests: `namespace kotatsu::ipc { namespace { ... } }`, so that the names of two files never collide. zest registers a suite in any namespace. Inside `kotatsu::async`, `fs::` names async's file system, `kotatsu::async::fs`; the synchronous one is written `kotatsu::fs::`.
- Harness headers use `namespace kotatsu::test`.
- Specializations of library templates (traits, `codec::repr`) are declared in the library's namespace, as C++ requires; the types they name stay in the test's anonymous namespace.
- Test types never live in a library namespace outside an anonymous one. No `using namespace` at namespace scope except `std::literals`.
## Checks
- One expression per check. `ZEXPECT(a == b)` reports both operands; `ZEXPECT(x)` and `ZEXPECT(!x)` report `x`. Split `a && b` into two checks.
- `ZASSERT` before anything that relies on the check: dereferencing an optional, expected or pointer, indexing, `.value()`. A failed `ZASSERT` ends the test's process without unwinding, so it serves alike in a coroutine, a helper that returns a value, a callback or another thread.
- What a failed `ZASSERT` would leave behind outside the process (a temporary directory, a child process) is cleaned up by a `zest::FatalHook`, declared after what it cleans up: a member of the owning type, as `test::TempDir` has, or a local beside a resource the test cannot change. A suite's constructor and destructor are its setup and teardown; a failed `ZASSERT` skips the destructor.
- An expected error is checked for what it is, not only that it happened: `ZASSERT(!result); ZEXPECT(result.error().kind == ...)`.
- Predicates: `contains`, `starts_with`, `ends_with` for text and ranges, `type_eq<A, B>()` for types, `throws(fn)` / `throws<E>(fn)` and `!throws(fn)` for exceptions.
- `ZEST_CONTEXT` in helpers and loops, naming the input a failing check was about.
- Large expected text (pretty output, schemas, diagnostics) is a snapshot: `ZEXPECT(zest::snapshot(text))`, or `zest::snapshot(text, "name")` for more than one in a case; a value that is not text is snapshotted as the debug codec renders it. Updating snapshots is a deliberate act.
- A case that must crash its process (an abort, a failed `assert`, `std::terminate`) is declared `crashes = true`; it passes only by crashing, and crashes by defined means, never by undefined behaviour. A crash that depends on the build mode is a constant attribute, e.g. `crashes = !KOTATSU_ENABLE_EXCEPTIONS`.
- `ZEXPECT((a == b))` compares with the type's own operator; use it only for a type meta cannot compare.
- Every case checks something. Checks inside a callback or coroutine are backed by a check, after the run, that they ran.
## Determinism
- Cases are independent. Each case runs on a fresh instance of its suite, so a case never clears or resets suite members, at its start or its end. Cases run in no set order, spread over worker processes that run at once, so a case never relies on another having run, or on its leftovers. A worker does run several cases one after another, so its process-wide state (globals and singletons, the environment, the current directory) is not fresh: a case that changes it restores it with an RAII guard such as `test::ScopedEnv`.
- Unit tests never order events by sleeping; they use events, latches or the loop's own ordering. A timer is fine as the subject of a test.
- System tests order events the same way wherever they can: wait for the event, a relay or a semaphore that says the other side is ready. A wait is left only to show that something does not happen, with a comment saying so.
- System tests bind port 0 and read the port back. A case that needs a port twice (a second bind to share it, or to find it in use) binds port 0 first and reuses the port it read back.
- System tests create temporary directories under the system temp directory with an owner that removes them on every path, and never use fixed file names. Windows named pipes live outside the file system; they take a random name instead (e.g. the temporary directory's name).
- `serial = true` and `skip = true` carry a one-line comment giving the reason. A case that can only tell at run time that it cannot run (no IPv6 loopback, no pseudo-terminal) calls `zest::skip()` and returns; a comment on the check, or on the helper it asks, gives the reason.
- A randomized test uses a fixed seed and prints it when it fails.
## Running
`pixi run test [preset]` runs every stage. To run one binary by hand, e.g. with a filter, run it from the repo root with the snapshot directory:
```bash
./build/debug/unit_tests --snapshot-dir=tests/snapshots --test-filter='codec_json_*'
```
`pixi run integration-test [preset]` runs only the integration tests. To run some by hand, do what ctest does (`ctest -N -V -L integration` prints its command), from the repo root with the driver's path set (`lsp_stub_server.exe` on Windows), and filter by name:
```bash
KOTATSU_LSP_STUB_SERVER=build/debug/lsp_stub_server pixi run node --import ./tests/check_npm_packages.ts --test --test-timeout=20000 --test-name-pattern='^hover_' tests/ipc/lsp/integration/requests.test.ts
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |