Skip to content

benschac/snap_post / .agents/skills/react-native-vision-camera-realtime

react-native-vision-camera-realtime

Verified

Design and review production-grade low-latency VisionCamera v5 pipelines. Use for real-time GPU, ML, CV, Skia or WebGPU overlays, Nitro frame plugins, zero-copy interop, frame budgets, and latency profiling. Use the general react-native-vision-camera skill for setup, capture, controls, basic frame outputs, or v4 migration.

Real-time VisionCamera pipelines

This is the specialized companion to react-native-vision-camera. Optimize the complete path from Camera buffer to final result, not an isolated stage. Before relying on exact APIs, check installed versions against current VisionCamera docs and the consumer's official docs or source.

Choose by final consumer

Final consumerPreferred path
Frame-coupled rendering, effects, or overlaysKeep processing and drawing on one GPU timeline with <SkiaCamera /> or WebGPU
WGSL compute or GPU inferenceFrame.getNativeBuffer() to a WebGPU video frame to device.importExternalTexture(...)
Native plugin that depends on VisionCameraA long-lived Nitro HybridObject whose hot method accepts a typed Frame
Native library without a VisionCamera dependencyThe untyped NativeBuffer pointer and explicit release contract
State-only ML or scanningBenchmark the platform runtime across ANE or NPU, GPU, and CPU backends; return compact state
CPU-only consumerUse the smallest useful resolution and format with a bounded, reusable CPU buffer path

Load references/interop.md only when implementing or reviewing Nitro, NativeBuffer, WebGPU, Skia, Resizer, or ArrayBuffer interop.

Hot-path invariants

  1. Keep orientation and mirroring as metadata. Set enablePhysicalBufferRotation: false, then pass frame.orientation and frame.isMirrored to the consumer or apply them in the same GPU transform that scales, crops, or renders. Never rotate the Camera buffer physically.
  2. Stay in one execution and memory domain. In a GPU pipeline, import once, keep preprocessing, inference, postprocessing, and rendering on the GPU, and read back only a compact result when required.
  3. Prefer pixelFormat: 'native' for a verified GPU-only path. Check frame.pixelFormat and frame.hasNativeBuffer because the resolved native format may be YUV, RGB, RAW, or private.
  4. Do not use getPixelBuffer(), getPlanes(), plane pixel buffers, mapped GPU buffers, or typed pixel views in the normal GPU path. CPU visibility can force synchronization or download.
  5. Create and warm pipelines, shaders, samplers, model sessions, resizers, large buffers, and native processors once. Reuse them for the component or session lifetime; never allocate them per frame.
  6. Draw frame-coupled overlays from the same Frame with Skia or WebGPU. Do not route per-frame geometry through React state, ordinary views, or Reanimated shared values.
  7. Release every Frame, NativeBuffer, wrapper, texture, and pooled slot exactly once on every path. Release wrappers in reverse ownership order and dispose the Frame last.

Prefer same-frame processing

Keep detection, tracking, decisions, and drawing synchronous with the matching frame when they must align visually. At 60 FPS the hard interval is 16.67 ms; at 30 FPS it is 33.33 ms. Target under roughly 16 ms and 33 ms to leave scheduling margin.

"Synchronous" means same-frame dataflow, not blocking the CPU until the GPU finishes. Encode dependent GPU stages in one command graph when possible. Do not add per-frame queue.onSubmittedWorkDone(), buffer mapping, readback, or another CPU or GPU fence.

Before making work asynchronous, remove copies and readbacks, reduce input resolution or FPS, fuse passes, optimize model tensors, and reuse warmed state. Use async only when the optimized work still cannot fit the frame interval, often around 50 ms or more, and the product accepts stale results. For frame-coupled visuals, prefer simplifying the work over visible lag.

The async delivery patterns are peers:

  • native Nitro work with a retained completion callback
  • native Nitro work that stores completed state behind a synchronous latest-state getter
  • a synchronous native method scheduled with VisionCamera's useAsyncRunner()

Every async design must bound in-flight work. Use one active task or a small fixed pool, reject or replace stale pending input, and never build an unbounded FIFO queue. dropFramesWhileBusy is an overload guard, not the architecture. With useAsyncRunner(), dispose an accepted Frame inside the task and a rejected Frame immediately.

Choose ML compute end to end

If inference feeds a same-frame Skia or WebGPU render, prefer keeping the entire path on the GPU. Crossing to an ANE, NPU, or CPU and returning geometry to the renderer is worthwhile only when end-to-end profiling proves it is faster while preserving the frame budget.

For state-only scanning, benchmark the platform runtime's available compute units. An ANE or NPU can avoid GPU contention and accelerate supported models; a CPU can win for tiny models when accelerator dispatch and transfer cost dominates. Measure input conversion, synchronization, inference, and result delivery, not inference alone. Normal React state or navigation is fine after a scan that has no frame-coupled overlay.

Development and production checks

When all native dependencies support it, use a resizable iPad-shaped Mac Catalyst or iPad-on-Mac build as a rapid iteration harness. A desktop agent can relaunch, resize, and screenshot it while using a built-in Mac camera or external UVC camera via useCameraDevice('external'). Fall back to a phone when the Mac target or required plugin is unavailable.

The Mac loop is for functional iteration, not performance prediction. Validate release builds on every production device class and representative GPUs. Test long enough to expose thermal throttling and pool leaks. Track:

  • camera timestamp to matching result or presentation latency at median, p95, and p99
  • dropped frames and maximum in-flight frames
  • CPU and GPU time, readbacks, maps, and synchronization points
  • allocations per frame, steady-state memory, sustained FPS, temperature, and power

Sample GPU timings asynchronously and sparsely enough that instrumentation does not become a synchronization point.

Authoritative references

writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 319kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 117kUpdated Claude CodeCodexVerifiedInstall