Skip to content

sb-dev/openquant / .agents/skills/property-based-testing

property-based-testing

Verified

Writes, reviews, and debugs property-based tests — Hypothesis, fast-check, proptest, jqwik, rapid, and Echidna or Medusa for Solidity invariants. Use whenever tests should cover a whole input domain instead of a hand-picked list of examples: encode/decode and serialize/deserialize pairs, parsers, canonicalizers and normalizers, validators, numeric and Decimal types, comparators and sort order, data structures, and smart-contract state invariants. Also use when adding cases to an existing @given, fast-check, or proptest suite, when judging whether existing property tests assert anything real, and when a generator has shrunk a counterexample and you need to tell a wrong property from a genuine bug. Not for coverage-guided binary fuzzing (libFuzzer, AFL), mutation-testing campaigns, static analysis, benchmarking, or end-to-end UI tests.

FileKindSize
README.mdmarkdown15.0 KB
SKILL.mdskill md4.2 KB
agents/openai.yamlconfig254 B
assets/trail-of-bits-mark.svgother3.0 KB
references/generating.mdmarkdown2.6 KB
references/interpreting-failures.mdmarkdown3.3 KB
references/libraries.mdmarkdown2.9 KB
references/refactoring.mdmarkdown4.7 KB
references/reviewing.mdmarkdown2.8 KB

references/refactoring.md

View on GitHub
# Refactoring to Expose a Property

"This code has no algebraic shape" is often a fact about how the code is *arranged*
rather than about what it does. A function that mixes a pure calculation with a database
write has a property; it just does not have a seam to assert it through. These are the
rearrangements that expose one, strongest first.

Suggest the refactor, name the property it unlocks, and let the author decide. A change
to production code to make a test possible is their call, not yours — the same rule that
governs adding a PBT dependency at all.

## 1. Extract the pure core

The highest-value one by a wide margin, and the reason most "untestable" code is
testable. I/O at the edges, calculation in the middle, assert against the middle.

```python
# Before: the arithmetic is real but unreachable without a database
def process_order(order_id: str) -> None:
    order = db.fetch(order_id)
    total = apply_discount(order, calculate_discount(order))
    db.save(order_id, total)

# After: the pure core takes arguments and returns a value
def order_total(order: Order, rules: DiscountRules) -> Decimal:
    return apply_discount(order, calculate_discount(order, rules))

def process_order(order_id: str) -> None:
    order = db.fetch(order_id)
    db.save(order_id, order_total(order, get_discount_rules()))
```

`order_total` now supports invariants (never negative, never above the undiscounted
total), monotonicity in the discount rate, and an oracle against a reference
calculation. `process_order` keeps example tests with a mocked `db`, which is the right
tool for a two-line wrapper.

The same move applies to anything whose observable is a side effect: build the message,
the request, the query object — then send it. Construction is testable; delivery is
mocked.

## 2. Add the missing inverse

A one-way operation has no roundtrip by definition. Sometimes the inverse is worth
having in production anyway, and sometimes it is worth having *only* for the test — say
which.

```python
def encode_message(msg: dict) -> bytes: ...
def decode_message(data: bytes) -> dict: ...   # unlocks decode(encode(x)) == x
```

Unlocks roundtrip, the strongest property in the catalog. Worth asking for even when the
production code never decodes: a serializer nobody can read back is usually a latent
bug, not a design.

## 3. Structured representation plus a renderer

String building by concatenation has nothing to assert beyond "contains a substring".
Split the value from its rendering and the inverse becomes available.

```python
# Before
def build_query(table: str, filters: dict) -> str:
    q = f"SELECT * FROM {table}"
    ...

# After
@dataclass
class Query:
    table: str
    filters: dict

def render(q: Query) -> str: ...
def parse(sql: str) -> Query: ...   # now render/parse is a roundtrip
```

This is pattern 2 wearing different clothes, and it is where escaping bugs live: a
roundtrip over generated filter values finds quoting errors that no hand-written example
will.

## 4. Return a value instead of mutating

An in-place mutation gives you nothing to compare against, because the input is gone by
the time you want to assert on it.

```python
def sort_tasks(tasks: list[Task]) -> None: ...      # before/after comparison impossible
def sorted_tasks(tasks: list[Task]) -> list[Task]:  # unlocks is_sorted, permutation,
    ...                                             # idempotence, length preservation
```

If the mutating signature has to stay, a wrapper that copies and returns is enough for
the test to have something to hold.

## 5. Inject the dependency

A function reading a global, a module constant, or `os.environ` can only be tested at
whatever those happen to be, so the edges of its input domain are unreachable.

```python
def validate(data: str) -> bool:          return len(data) <= CONFIG.max_length
def validate(data: str, max_len: int):    return len(data) <= max_len
```

Parameterising the bound is what lets a generator drive `max_len` to 0, to 1, and to the
maximum representable value — the boundaries where validators actually break.

## When not to suggest this

- **The property you would unlock is "no crash".** Restructuring production code to
  enable the weakest property in the catalog is a bad trade. Say the code is a poor PBT
  candidate and stop.
- **The module needs wholesale restructuring.** Say that once, plainly. Twenty
  individually-reasonable suggestions on one file is noise, and it reads as a rewrite
  request rather than a testing recommendation.
- **The refactor breaks a public API.** Flag it as breaking and offer the
  backwards-compatible version, even when the clean version is obviously nicer.
- **Existing tests cover the code.** Run them after any refactor and say you did.
  "Enabled a property test and broke two example tests" is not progress.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 319kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 319kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 319kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 196kUpdated Claude CodeCodexVerifiedInstall