benschac/snap_post / .agents/skills/react-native-vision-camera
react-native-vision-camera
UnverifiedBest-practices guide for React Native VisionCamera v5 setup, migration, capture, controls, outputs, and basic frame processing. Use the separate react-native-vision-camera-realtime skill for production low-latency GPU, ML, CV, Skia or WebGPU pipelines and frame-coupled overlays.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 8.6 KB |
| references/advanced-features.md | markdown | 12.2 KB |
| references/capture-and-controls.md | markdown | 11.4 KB |
| references/frame-processors.md | markdown | 7.4 KB |
| references/migration-templates.md | markdown | 16.3 KB |
| references/migration-v4-to-v5.md | markdown | 22.5 KB |
| references/outputs-and-constraints.md | markdown | 10.1 KB |
| references/quickstart-v5.md | markdown | 6.8 KB |
references/frame-processors.md
View on GitHub# Frame Processors — v5 (Nitro)
V5 frame processors are worklet callbacks attached to a `CameraFrameOutput`. The worklets engine is **`react-native-worklets` (Software Mansion)**, not `react-native-worklets-core`. Native plugins are **Nitro `HybridObject`s** — the v4 `FrameProcessorPlugin` subclass + `VISION_EXPORT_SWIFT_FRAME_PROCESSOR` macro is gone.
This reference covers API fundamentals. For production low-latency GPU, ML, CV, zero-copy interop, or frame-coupled overlays, use the separate `react-native-vision-camera-realtime` skill.
## Required dependencies
```sh
npm i react-native-vision-camera-worklets react-native-worklets
```
Do NOT install `react-native-worklets-core`. If a user reports "frame processor does nothing", check for this. Remove any `react-native-worklets-core/plugin` entry from `babel.config.js` and add `react-native-worklets/plugin` in its place (Reanimated 4 depends on `react-native-worklets` under the hood, so Expo SDK 54+ templates already include this plugin). See https://docs.swmansion.com/react-native-worklets/docs/.
## The basic shape
```tsx
import { Camera, useFrameOutput, useCameraDevice } from 'react-native-vision-camera'
const device = useCameraDevice('back')
const frameOutput = useFrameOutput({
pixelFormat: 'yuv', // NOTE: default is 'native' (zero-copy). 'yuv' = CPU-accessible YUV; 'rgb' forces conversion. source: useFrameOutput.ts:123
targetResolution: CommonResolutions.VGA_16_9, // start small
onFrame(frame) {
'worklet'
try {
// frame.width, frame.height, frame.pixelFormat, frame.orientation, frame.timestamp
// Call Nitro plugins here (see below).
} finally {
frame.dispose()
}
},
})
<Camera device={device} isActive={true} outputs={[frameOutput]} />
```
### Non-negotiable: `frame.dispose()`
The camera maintains a bounded GPU-backed buffer pool. A leaked frame stalls the pipeline and frames are dropped. **Always** dispose, even in error paths:
```ts
onFrame(frame) {
'worklet'
try { /* work */ }
catch (e) { /* log */ }
finally { frame.dispose() }
}
```
The same rule applies to `Depth` frames from `useDepthOutput`.
## Pixel format decision
| Format | When |
|---|---|
| `'native'` | **Default.** Zero-copy GPU path — streams the session's negotiated `nativePixelFormat`. Resolved format may be YUV, RGB, RAW, or `'private'` — verify via `frame.pixelFormat`. |
| `'yuv'` | Best CPU-accessible choice: OpenCV, native camera pipelines, MLKit, Skia. A 4K YUV frame is ~12MB vs ~31MB RGB. |
| `'rgb'` | ML frameworks that hard-require RGB and don't convert internally. Prefer the GPU **Resizer** over paying RGB conversion on every frame. |
<!-- source: useFrameOutput.ts:123 (default 'native'); CameraFrameOutput.nitro.ts:71-95; VideoPixelFormat.ts:52-62 -->
## Async frame work when stale results are acceptable
Use an `AsyncRunner` when non-visual work cannot keep up and the feature accepts results from older frames. For frame-coupled visual feedback, use the real-time skill and optimize synchronous same-frame processing first.
```tsx
import { useAsyncRunner } from 'react-native-vision-camera'
const asyncRunner = useAsyncRunner()
const frameOutput = useFrameOutput({
onFrame(frame) {
'worklet'
const accepted = asyncRunner.runAsync(() => {
'worklet'
try {
const detections = runMlModel(frame)
// update Reanimated SharedValues directly — no runOnJS needed in v5
detectionsShared.value = detections
} finally {
frame.dispose()
}
})
if (!accepted) frame.dispose() // runner full; drop and keep camera flowing
},
})
```
- `runAsync` returns `boolean`. `true` = accepted, dispose **inside** the async callback. `false` = busy, dispose **immediately**.
- There is no `runAtTargetFps` in v5. Throttle by counting inside the worklet (`count.value = (count.value + 1) % 3`) and skipping, or rely on backpressure through the async runner.
- Each `useAsyncRunner()` gets its own dedicated worklet runtime. Multiple stages → multiple runners.
## Reanimated integration
Worklets in v5 can mutate Reanimated `SharedValue`s directly. No `runOnJS` round-trip:
```tsx
import { useSharedValue } from 'react-native-reanimated'
const faces = useSharedValue<Face[]>([])
const frameOutput = useFrameOutput({
onFrame(frame) {
'worklet'
try {
faces.value = detectFaces(frame) // drives Reanimated animations on UI thread
} finally { frame.dispose() }
},
})
```
Use SharedValues for state or animation that may update asynchronously. Do not drive frame-locked bounding boxes, meshes, or masks through SharedValues; draw them from the matching frame with Skia or WebGPU as described by the real-time skill.
## Coordinate conversions for view-based overlays
When an asynchronous view overlay is acceptable, convert native frame coordinates into preview-view space rather than reimplementing orientation math:
```ts
// Inside a worklet:
const cameraPoint = frame.convertFramePointToCameraPoint(framePoint)
// Inside a regular JS context (with a PreviewViewMethods ref):
const viewPoint = previewView.convertCameraPointToViewPoint(cameraPoint)
// ScannedObject convenience:
const viewObj = previewView.convertScannedObjectCoordinatesToViewCoordinates(scannedObject)
```
## Creating native plugins — Nitro only
Native Frame processor plugin requires to be created in nitro modules. Use build-nitro-modules skill or ask user to install that
### C++ cross-platform plugin
Also supported. Access the underlying buffer via `frame.getNativeBuffer()` and work against a single C++ codebase.
### Depth variant
Same pattern: accept a `Depth` in the spec, cast to `NativeDepth` to get `AVDepthData` / Android's depth representation.
## Best-practice checklist
- [ ] `pixelFormat`: keep the default `'native'` (zero-copy) for GPU pipelines; pass `'yuv'` when you need CPU pixel access (MLKit/OpenCV); `'rgb'` only if a consumer hard-requires it. <!-- source: useFrameOutput.ts:123 -->
- [ ] `targetResolution` on the frame output — smaller is faster. VGA or 720p is enough for most ML models.
- [ ] Every `onFrame` wrapped in `try { ... } finally { frame.dispose() }`.
- [ ] When stale results are acceptable, offload over-budget work with `useAsyncRunner` and explicit accepted/rejected disposal.
- [ ] No `runOnJS` for ordinary Reanimated `SharedValue` mutations (v5 supports direct mutation); do not use them for frame-locked overlays.
- [ ] For a CPU-visible ML tensor, use `react-native-vision-camera-resizer` for GPU-accelerated resize and conversion instead of doing those operations on the CPU.
- [ ] For coordinate-space math: use the `convertFramePointToCameraPoint` / `convertCameraPointToViewPoint` pair — don't re-implement orientation math.
- [ ] Native plugin: must be a Nitro `HybridObject`; there is no other supported path in v5.
## Pointers
- Docs — Async Frame Processing: https://visioncamera.margelo.com/docs/async-frame-processing
- API — `Frame`: https://visioncamera.margelo.com/api/react-native-vision-camera/hybrid-objects/Frame
- Worklets (SWM, babel plugin `react-native-worklets/plugin`): https://docs.swmansion.com/react-native-worklets/docs/
- Repo: https://github.com/mrousavy/react-native-vision-camera
- Nitro scaffolding: use the `build-nitro-modules` skill.
- Related: [advanced-features.md](./advanced-features.md) (Depth, Resizer, Barcode, SkiaCamera), [outputs-and-constraints.md](./outputs-and-constraints.md), [capture-and-controls.md](./capture-and-controls.md), [migration-v4-to-v5.md](./migration-v4-to-v5.md)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |