Skip to content

sb-dev/openquant / .agents/skills/property-based-testing

property-based-testing

Verified

Writes, reviews, and debugs property-based tests — Hypothesis, fast-check, proptest, jqwik, rapid, and Echidna or Medusa for Solidity invariants. Use whenever tests should cover a whole input domain instead of a hand-picked list of examples: encode/decode and serialize/deserialize pairs, parsers, canonicalizers and normalizers, validators, numeric and Decimal types, comparators and sort order, data structures, and smart-contract state invariants. Also use when adding cases to an existing @given, fast-check, or proptest suite, when judging whether existing property tests assert anything real, and when a generator has shrunk a counterexample and you need to tell a wrong property from a genuine bug. Not for coverage-guided binary fuzzing (libFuzzer, AFL), mutation-testing campaigns, static analysis, benchmarking, or end-to-end UI tests.

FileKindSize
README.mdmarkdown15.0 KB
SKILL.mdskill md4.2 KB
agents/openai.yamlconfig254 B
assets/trail-of-bits-mark.svgother3.0 KB
references/generating.mdmarkdown2.6 KB
references/interpreting-failures.mdmarkdown3.3 KB
references/libraries.mdmarkdown2.9 KB
references/refactoring.mdmarkdown4.7 KB
references/reviewing.mdmarkdown2.8 KB

references/generating.md

View on GitHub
# Generating Property-Based Tests

Writing the `@given` decorator is the easy part. These are the decisions that make
the difference between a suite that finds bugs and one that just runs.

## Put constraints in the strategy, not in `assume()`

This is the single highest-value habit. `assume()` discards inputs after generation,
so a filter that rejects most candidates wastes the budget and eventually trips
Hypothesis's exhausted-filter guard — which surfaces as a warning nobody reads.

```python
# Slow, and mostly discards
@given(st.integers())
def test_positive(x):
    assume(x > 0)
    ...

# Generates only what you want
@given(st.integers(min_value=1))
def test_positive(x):
    ...
```

Reserve `assume()` for conditions you genuinely cannot express as a generator — a
relationship between two already-generated values, usually.

Build compound inputs with `st.builds`, and derive dependent fields with
`st.composite` or `.flatmap` rather than generating independently and filtering:

```python
@st.composite
def sized_list_and_index(draw):
    xs = draw(st.lists(st.integers(), min_size=1))
    i = draw(st.integers(min_value=0, max_value=len(xs) - 1))
    return xs, i
```

## Pin the edge cases you already know about

Random generation finds boundaries eventually; `@example` finds them on every run and
documents that you thought about them.

```python
@given(st.lists(st.integers()))
@example([])
@example([1])
@example([1, 1, 1])
def test_sort(xs): ...
```

Empty, single-element, all-duplicates, zero, negative, and the maximum representable
value are the ones that recur.

## Settings

Defaults (100 examples, 200ms deadline) are wrong at both ends of the workflow:

```python
@settings(max_examples=10)                      # local iteration
@settings(max_examples=200)                     # CI
@settings(max_examples=1000, deadline=None)     # nightly
```

Set `deadline=None` for anything doing real work — the default deadline turns a slow
machine into a failing test, and that flake gets the whole suite deleted.

## Determinism as a property

`f(x) == f(x)` is a tautology for a pure function and a real test for anything else.
Serializers over dicts or sets, anything involving hashing, iteration order, or time —
those can and do return different output for the same input. Assert it where a broken
implementation could falsify it, and not otherwise.

## Testing the error path

`st.binary()` against a decoder is worth writing: the contract is usually "raises
`DecodeError` or succeeds, never `IndexError`, never hangs". Catch only the documented
exception and let everything else fail the test.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 319kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 319kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 319kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 196kUpdated Claude CodeCodexVerifiedInstall