Martingale42/nautilus-dev / skills/nt-adapters
nt-adapters
VerifiedUse when working with exchange or data provider adapters, HTTP/WebSocket clients, instrument providers, or venue integration in NautilusTrader.
references/guides/ffi.md
View on GitHub# FFI Memory Contract
NautilusTrader exposes several **C-compatible** types so that compiled Rust code can be
consumed from C-extensions generated by Cython or by other native languages. The most
important of these is `CVec` – a *thin* wrapper around a Rust `Vec<T>` that is passed across
the FFI boundary **by value**.
The rules below are *strict*; violating them results in undefined behaviour (usually a double-free or a memory leak).
## Fail-fast panics at the FFI boundary
Rust panics must never unwind across `extern "C"` functions. Unwinding into C or Python is
undefined behaviour and can corrupt the foreign stack or leave partially-dropped resources
behind. To enforce the fail-fast architecture we wrap every exported symbol in
`crate::ffi::abort_on_panic`, which executes the body and calls `process::abort()` if a panic
occurs. The panic message is still logged before the abort, so debugging output is preserved
while avoiding undefined behaviour.
When adding new FFI functions, call `abort_on_panic(|| { … })` around the implementation (or
use a helper that does so) to maintain this guarantee.
## CVec lifecycle
| Step | Owner | Action |
|-------|-------------------------------|--------|
| **1** | Rust | Build a `Vec<T>` and convert it with `into()` – this *leaks* the vector and transfers ownership of the raw allocation to foreign code. |
| **2** | Foreign (Python / Cython / C) | Use the data while the `CVec` value is in scope. **Do not modify the fields `ptr`, `len`, `cap`.** |
| **3** | Foreign | Exactly once, call the *type-specific* drop helper exported by Rust (for example `vec_drop_book_levels`, `vec_drop_book_orders`, `vec_time_event_handlers_drop`). The helper reconstructs the original `Vec<T>` with `Vec::from_raw_parts` and lets it drop, freeing the memory. |
:::warning
If step **3** is forgotten the allocation is leaked for the remainder of the process; if it
is performed **twice** the program will double-free and likely crash.
:::
## Capsules created on the Python side
Several Cython helpers allocate temporary C buffers with `PyMem_Malloc`, wrap them into a
`CVec`, and return the address inside a `PyCapsule`. **Every such capsule is created with a
destructor** (`capsule_destructor` or `capsule_destructor_deltas`) that frees both the buffer
and the `CVec`. Callers must therefore *not* free the memory manually – doing so would double
free.
## Capsules created on the Rust side *(PyO3 bindings)*
When Rust code pushes a heap-allocated value into Python it **must** use
`PyCapsule::new_with_destructor` so that Python knows how to free the allocation
once the capsule becomes unreachable. The closure/destructor is responsible
for reconstructing the original `Box<T>` or `Vec<T>` and letting it drop.
```rust
use pyo3::types::PyCapsule;
Python::attach(|py| {
// Allocate the value on the heap
let my_data = Box::new(MyStruct::new());
let ptr = Box::into_raw(my_data);
// Move it into the capsule and register a destructor that frees the memory
let capsule = PyCapsule::new_with_destructor(
py,
ptr,
None,
|ptr, _| {
// Reconstruct the Box and let it drop, freeing the allocation
let _ = unsafe { Box::from_raw(ptr) };
},
)
.expect("capsule creation failed");
// ... pass `capsule` back to Python ...
});
```
Do **not** use `PyCapsule::new(…, None)`; that variant registers *no* destructor
and will leak memory unless the recipient manually extracts and frees the
pointer (something we never rely on). The codebase has been updated to follow
this rule everywhere – adding new FFI modules must follow the same pattern.
## Why there is no generic `cvec_drop` anymore
Earlier versions of the codebase shipped a generic `cvec_drop` function that always treated the
buffer as `Vec<u8>`. Using it with any other element type causes a size-mismatch during
deallocation and corrupts the allocator's bookkeeping. Because the helper was not referenced
anywhere inside the project it has been removed to avoid accidental misuse.
Instead, use the **type-specific** drop helper for your element type (e.g., `vec_drop_book_levels`,
`vec_drop_book_orders`). If no helper exists for your type, add one following the pattern in
`crates/core/src/ffi/cvec.rs`.
## Box-backed `*_API` wrappers (owned Rust objects)
When the Rust core needs to hand a *complex* value (for example an
`OrderBook`, `SyntheticInstrument`, or `TimeEventAccumulator`) to foreign
code it allocates the value on the heap with `Box::new` and returns a
small `repr(C)` wrapper whose only field is that `Box`.
```rust
#[repr(C)]
pub struct OrderBook_API(Box<OrderBook>);
#[unsafe(no_mangle)]
pub extern "C" fn orderbook_new(id: InstrumentId, book_type: BookType) -> OrderBook_API {
OrderBook_API(Box::new(OrderBook::new(id, book_type)))
}
#[unsafe(no_mangle)]
pub extern "C" fn orderbook_drop(book: OrderBook_API) {
drop(book); // frees the heap allocation
}
```
Memory-safety requirements are therefore:
1. Every constructor (`*_new`) **must** have a matching `*_drop` exported
next to it.
2. Validate parameters before heap allocation to fail fast and avoid allocating invalid objects.
3. The *Python/Cython* binding must guarantee that `*_drop` is invoked
exactly once. Two approaches exist:
• **Preferred for new code**: Wrap the pointer in a `PyCapsule` created with
`PyCapsule::new_with_destructor`, passing a destructor that calls
the drop helper.
• **Legacy pattern** (v1 Cython modules only): Call the helper explicitly in
`__del__`/`__dealloc__` on the Python side:
```python
cdef class OrderBook:
cdef OrderBook_API _mem
def __cinit__(self, ...):
self._mem = orderbook_new(...)
def __del__(self):
if self._mem._0 != NULL:
orderbook_drop(self._mem)
```
Whichever style is used, remember: **forgetting the drop call leaks the
entire structure**, while calling it twice will double-free and crash.
New FFI code must use `PyCapsule` with destructors and follow this template before it can be merged.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |