nWave-ai/nWave / nWave/skills/nw-property-based-testing
nw-property-based-testing
VerifiedLoad when a design law holds over a broad input, state or sequence domain and needs a generated property test.
Property tests for broad laws
A property states "for all valid inputs this observation holds". It searches for counterexamples; it is not proof and not a substitute for a real-boundary test. Use the project's native generator with shrinking and replay.
1. Decide: property or example
- A law over a broad domain (unbounded inputs, state transitions, sequences, serialization, business rules) gets a property. A table of examples never stands in for it.
- A closed finite domain gets native examples or an exhaustive loop over every member; no property.
- Write one property per distinct law. Combine laws only when one generated observation falsifies each.
- Pick the pattern: invariant (holds for every input), roundtrip (
decode(encode(x)) == x), oracle (compare with a simple reference), metamorphic (a known input change gives a known output change), idempotence (f(f(x)) == f(x)), commutativity (order of operands or operations does not matter). When none is obvious, start from a known example, embed its answer in a generated input, and predict where it appears in the output. - Write no property for a layer that declares no law.
2. Write the generator
- Declare domain and size bounds in the test code. Never narrow a boundary to suit the storage engine or the generator.
- Make every generated component influence the input, the call or the oracle; delete any that does not.
- For a branching law, generate the case tag first from a closed sum, then build inputs that satisfy that case by construction. Never reach rare branches by filtering, assume/discard or a larger example count.
- Build structured data from the valid-input type or constructor, so invalid states are never generated unless the law is about rejecting them.
- Keep it bounded: case count and size so the property fits the default-suite budget (about 10 ms per case). Failures print a replayable seed.
@given(st.lists(st.integers(1, 10**6), min_size=1), st.data())
def test_transfers_keep_total_balance(amounts, data): # law id in docstring
acct = Ledger.open(sum(amounts))
for a in amounts: acct.transfer(a % acct.max_move() or 1)
assert acct.total() == sum(amounts)
3. Write the observation and oracle
- Compute the expected result independently (reference model, inverse operation, algebraic relation). Never copy the production algorithm into the oracle.
- Assert the promised observation, not "returns a collection", "has the type" or "never raises", unless that is the law.
- Name the property in domain words ("transfers keep the total balance"); put the design law id in the docstring.
4. Stateful and sequence laws
- Model the observable contract with the simplest pure structure (dict, set, list); never mirror the implementation.
- Each command has a precondition on the model, a transition on the model, an execution on the real system and a postcondition comparing real result to model.
- Put every precondition in the framework's precondition mechanism, not only in the generator, so shrunk sequences stay valid. A missing precondition makes shrinking leave long or ghost failures.
- Treat values returned by the real system as opaque in model code: store and pass, never inspect.
- Weight commands so every critical command runs; a sequence that never reaches the law's commands proves nothing.
- Test concurrency (linearizability) only when concurrent correctness is the stated risk.
- A partial model covering the promised operations well beats a complete model covering all of them weakly.
5. Where it runs
- A broad state or sequence law gets at least one property at integration level: through the real port with real adapter I/O, or at the nearest law-bearing seam with a stated map from its observation to the real-port observation. A model without that map is a proxy.
- Adapter and recovery properties inject deterministic faults; keep the representative real journey and persistent-storage tests beside them.
- When no honest seam exists or the tool lacks the capability, report an EVIDENCE_GAP with its cause; never reduce the law to examples.
6. Prove it can fail
- A property for newly promised behaviour must fail on the pre-change code for the promised observation, not for a setup reason. If it passes there, fix or delete it. A property that pins existing behaviour needs only to pass.
- On a deliberate defect, confirm shrinking yields a small failing case and the seed replays it. If the tool cannot shrink or replay, say so in the test file.
7. Self-check before recording the step
- One law per property, named in domain words, id in docstring.
- Generator declares domain and size bounds; invalid states not generated by accident.
- Every generated component is used; rare branches built by construction.
- Oracle is independent of the production algorithm.
- Finite domains use examples or an exhaustive loop; broad laws never use a table alone.
- Stateful: model pure, preconditions in the framework, critical commands exercised.
- New-behaviour property failed on pre-change code for the promised observation.
- Broad state law has an integration-level property with its real-port map.
- The property fits its time budget.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |