Skip to content

ByteTerrace/Puck / .claude/skills/rendering

rendering

Verified

Maps Puck's SDF renderer and GPU code: mapCore, mesh bake, HLSL kernels, sdf-isa.hlsli, SdfOp, C#/HLSL layouts, world.shaders.reload, field scopes, GPU cost and occupancy. Sculpting: sdf-authoring.

FileKindSize
SKILL.mdskill md139.7 KB
evals/g02-ao-edit-has-no-effect/graders/ambient-is-a-separate-dispatch.mdmarkdown140 B
evals/g02-ao-edit-has-no-effect/graders/isolate.mdmarkdown84 B
evals/g02-ao-edit-has-no-effect/graders/reload-loop.mdmarkdown79 B
evals/g02-ao-edit-has-no-effect/graders/right-file.mdmarkdown96 B
evals/g02-ao-edit-has-no-effect/graders/skill-fired.mdmarkdown36 B
evals/g02-ao-edit-has-no-effect/graders/views-only-reads.mdmarkdown287 B
evals/g02-ao-edit-has-no-effect/prompt.mdmarkdown413 B
evals/g03-framerate-after-placements/graders/budget.mdmarkdown70 B
evals/g03-framerate-after-placements/graders/capacity-growth.mdmarkdown132 B
evals/g03-framerate-after-placements/graders/counted-work-not-wall-clock.mdmarkdown215 B
evals/g03-framerate-after-placements/graders/counters-gpu.mdmarkdown72 B
evals/g03-framerate-after-placements/graders/quality-knobs.mdmarkdown119 B
evals/g03-framerate-after-placements/graders/skill-fired.mdmarkdown36 B
evals/g03-framerate-after-placements/prompt.mdmarkdown371 B
evals/g04-hand-written-isa-define/graders/encoding-probe.mdmarkdown73 B
evals/g04-hand-written-isa-define/graders/generated-from-the-csharp-model.mdmarkdown138 B
evals/g04-hand-written-isa-define/graders/name-follows-the-enum-member.mdmarkdown135 B
evals/g04-hand-written-isa-define/graders/regenerate-verb.mdmarkdown98 B
evals/g04-hand-written-isa-define/graders/skill-fired.mdmarkdown36 B
evals/g04-hand-written-isa-define/prompt.mdmarkdown201 B
evals/g06-extra-map-call-sites/graders/each-site-is-an-inlined-copy.mdmarkdown110 B
evals/g06-extra-map-call-sites/graders/register-pressure.mdmarkdown74 B
evals/g06-extra-map-call-sites/graders/reuse-through-a-loop.mdmarkdown119 B
evals/g06-extra-map-call-sites/graders/rolled-loop.mdmarkdown65 B
evals/g06-extra-map-call-sites/graders/skill-fired.mdmarkdown36 B
evals/g06-extra-map-call-sites/prompt.mdmarkdown234 B
evals/g07-abi-change-with-reload/graders/both-sides-change-together.mdmarkdown103 B
evals/g07-abi-change-with-reload/graders/needs-a-rebuild.mdmarkdown113 B
evals/g07-abi-change-with-reload/graders/reload-compiles-kernel-sources-only.mdmarkdown118 B
evals/g07-abi-change-with-reload/graders/skill-fired.mdmarkdown36 B
evals/g07-abi-change-with-reload/graders/sync-pairs.mdmarkdown252 B
evals/g07-abi-change-with-reload/prompt.mdmarkdown192 B
evals/g08-dynamic-transform-capacity/graders/capacity-does-not-grow.mdmarkdown138 B
evals/g08-dynamic-transform-capacity/graders/fix-the-emission.mdmarkdown124 B
evals/g08-dynamic-transform-capacity/graders/identity-slot-is-a-bug.mdmarkdown117 B
evals/g08-dynamic-transform-capacity/graders/skill-fired.mdmarkdown36 B
evals/g08-dynamic-transform-capacity/prompt.mdmarkdown238 B
evals/g10-hard-op-in-one-kernel-only/graders/cpu-interpreter.mdmarkdown74 B
evals/g10-hard-op-in-one-kernel-only/graders/encoding-probe.mdmarkdown73 B
evals/g10-hard-op-in-one-kernel-only/graders/not-done.mdmarkdown71 B
evals/g10-hard-op-in-one-kernel-only/graders/parts-walk.mdmarkdown66 B
evals/g10-hard-op-in-one-kernel-only/graders/skill-fired.mdmarkdown36 B
evals/g10-hard-op-in-one-kernel-only/prompt.mdmarkdown144 B
evals/g12-hard-carve-reads-phantom-surfaces/graders/extend-the-carve-or-use-union.mdmarkdown141 B
evals/g12-hard-carve-reads-phantom-surfaces/graders/skill-fired.mdmarkdown36 B
evals/g12-hard-carve-reads-phantom-surfaces/graders/subtraction-exact-only-near-its-subject.mdmarkdown115 B
evals/g12-hard-carve-reads-phantom-surfaces/prompt.mdmarkdown217 B
evals/g13-hard-wrap-each-primitive-in-a-scope/graders/clamps-are-bounds-not-cost.mdmarkdown101 B
evals/g13-hard-wrap-each-primitive-in-a-scope/graders/make-the-primitive-lipschitz.mdmarkdown138 B
evals/g13-hard-wrap-each-primitive-in-a-scope/graders/scope-is-a-per-sample-cost.mdmarkdown88 B
evals/g13-hard-wrap-each-primitive-in-a-scope/graders/skill-fired.mdmarkdown36 B
evals/g13-hard-wrap-each-primitive-in-a-scope/graders/skips-early-out-and-rigid-planning.mdmarkdown110 B
evals/g13-hard-wrap-each-primitive-in-a-scope/prompt.mdmarkdown167 B
evals/g14-hard-two-lanes-both-added-an-op/graders/ci-checks-only.mdmarkdown122 B
evals/g14-hard-two-lanes-both-added-an-op/graders/fingerprint-rewritten.mdmarkdown100 B
evals/g14-hard-two-lanes-both-added-an-op/graders/generate-verb-or-build.mdmarkdown116 B
evals/g14-hard-two-lanes-both-added-an-op/graders/skill-fired.mdmarkdown36 B
evals/g14-hard-two-lanes-both-added-an-op/prompt.mdmarkdown207 B
evals/g15-signed-modulo-on-vulkan/graders/driver-answers-wrongly.mdmarkdown172 B
evals/g15-signed-modulo-on-vulkan/graders/lift-and-wrap-as-uint.mdmarkdown121 B
evals/g15-signed-modulo-on-vulkan/graders/skill-fired.mdmarkdown36 B
evals/g15-signed-modulo-on-vulkan/prompt.mdmarkdown353 B
evals/g16-per-pixel-field-view/graders/no-field-walk-in-a-pixel-kernel.mdmarkdown203 B
evals/g16-per-pixel-field-view/graders/sample-a-lattice-through-the-bake.mdmarkdown164 B
evals/g16-per-pixel-field-view/graders/skill-fired.mdmarkdown36 B
evals/g16-per-pixel-field-view/graders/slice-verb.mdmarkdown83 B
evals/g16-per-pixel-field-view/prompt.mdmarkdown311 B
evals/g17-new-bake-source-stale-meshes/graders/identity-follows-the-calls.mdmarkdown436 B
evals/g17-new-bake-source-stale-meshes/graders/names-mesh-identity.mdmarkdown70 B
evals/g17-new-bake-source-stale-meshes/graders/skill-fired.mdmarkdown36 B
evals/g17-new-bake-source-stale-meshes/graders/store-keyed-by-bake-identity.mdmarkdown175 B
evals/g17-new-bake-source-stale-meshes/prompt.mdmarkdown293 B
evals/moving-blend-crease-copies/graders/one-tuple-and-one-skin.mdmarkdown648 B
evals/moving-blend-crease-copies/prompt.mdmarkdown597 B
evals/shade-helper-cost/graders/one-admission-contract.mdmarkdown419 B
evals/shade-helper-cost/prompt.mdmarkdown312 B
references/authored-shading.mdmarkdown31.3 KB
references/gpu-cost.mdmarkdown37.1 KB
references/kernels.mdmarkdown34.6 KB
references/mesh-bake.mdmarkdown123.2 KB
references/mesh-gi.mdmarkdown64.6 KB
references/runtime-contracts.mdmarkdown26.7 KB
references/sync-pairs.mdmarkdown69.2 KB

references/gpu-cost.md

View on GitHub
# GPU cost: registers, waves, memory and sync

Read this before changing a kernel's live state, a groupshared layout, a
dispatch shape, a submission, a fence wait or a GPU allocation, or when a pass
reads slow in `world.gpu-timing`. Back to [../SKILL.md](../SKILL.md). The
skill's [performance and cost](../SKILL.md#performance-and-cost) section owns
the live levers and counters; this page owns what a kernel costs on the
hardware and how to read it. Each rule names the Puck kernel or host code and
the measurement behind it; a general claim cites two public sources
([sources](#sources)).

## Attribute memory by layer

`GpuDeviceMemoryWork.IsDeviceLocal` sorts each allocation's role into one of two
disjoint backend rows: device-local bytes (aperture storage is already inside them)
and host-visible bytes (upload, staging and readback heaps). `host.memory owners 0`
appends both; neither includes driver internals or swapchain images. Device-local
plus host-visible is what the backend's counted allocations hold, and on the Deck's
unified memory both come out of the same physical pool. Those readings overlap
the account's; what adds is `host.memory`'s physical rows per pool (device
blocks, native allocator, native direct, managed, runtime and driver), each with
the part its owners name and its residual. Device memory is named by owner
family (`GpuObjectName.MemoryFamily`: the creator's owner without its `$`
instance, then the part), counted at placement by the ledger
(`GpuDeviceMemoryWork.CountPlaced`, or `CountAllocated` for a dedicated
allocation), so `host.memory`'s `family` lines name who holds a segment and who
churns it (allocated against released), and a segment's `residual` is the room
inside shared blocks. A creating member's `GpuObjectName` is therefore what its
memory is attributed to: a default name lands in the `unnamed` family. Read full
GPU node counters when per-node ownership matters: summary submission counters
omit the node's owned-byte gauge.

Keep driver residency/budget observations separate from resource allocations.
[DXGI reports process usage and an OS budget](https://learn.microsoft.com/en-us/windows/win32/api/dxgi1_4/ns-dxgi1_4-dxgi_query_video_memory_info),
while [Vulkan's memory-budget extension](https://docs.vulkan.org/refpages/latest/refpages/source/VK_EXT_memory_budget.html)
reports estimated process heap usage and a changing budget. Neither is an owner
census or a portable conversion from process-private bytes to physical pages.
Correlate a held normal-play window with retained owners and these driver/host
observations; a replay observer's memory does not qualify the player's budget.

## Measure transfer windows

`world.counters gpu transfers --json` reads `GpuTransferTotals` from the existing
work ledgers. Unlike `summary`, these are monotonic counts at submission/discard
checkpoints, not one completed submission. The currently open record is excluded
until its checkpoint. Host writes count CPU payload bytes; buffer-copy bytes
include upload, readback and device-local carry copies once each. Image copies
have operations but no inferred bytes. None is an allocation or physical bus count.

Use submitted and completed deltas separately: accepted work between endpoints is
not necessarily the work whose completion became known there. Completion prefixes
belong to one concrete ordered queue; another node's fence proves nothing about
this ledger. `GpuWorkCounting.Wrap` binds that queue. The prefix survives bounded
sample reuse and invalidation through the counting fence's exact captured prefix.

A report reads each source once and reuses the first observation for ledger aliases.
The `gpu-transfers` signal parser deduplicates domains; a pinned route refuses
unsupported sources, changing domains or decreasing counts instead of subtracting
retired/new owners as zero. This covers stable ledger populations; churn requires
retaining terminal owner observations before claiming a complete transfer window.
External image producers on another device remain outside the render ledger's
scope. The native World registry excludes those producers rather than inventing
zero transfer counts. JSON labels the scope `registered-render-device-ledgers`:
complete coverage means those reported domains only, not every GPU operation the
host performs. Win32 capture-feed conversion/readback work is outside this scope.
The ordering premise agrees with [Direct3D queue completion timelines](https://learn.microsoft.com/en-us/windows/win32/direct3d12/user-mode-heap-synchronization)
and [Vulkan fence signal scope](https://docs.vulkan.org/spec/latest/chapters/synchronization.html#synchronization-fences-signaling);
submission order alone does not establish cross-queue completion.

## Read a kernel's registers and occupancy

The SPIR-V and DXIL the build writes beside each `*.hlsl` carry no register
count: the driver compiles them to the GPU's own instructions and allocates
registers there ([KHR-PEP], [MS-DXIL]). `spirv-dis <kernel>.spv` and
`dxc -dumpbin <kernel>.dxil` (both ship with the Vulkan SDK) show the
intermediate code only, though an `alloca` in the DXIL is a dynamically indexed
local array, which NVIDIA places in local memory (the local-array rule below):
count them over every kernel after a shader change. Read the driver's numbers
one of these ways, as a GPU leg under `puck lease gpu`, windowed, after
`world.wait bakes` and `[sdf.mesh-gi] done`.

**In process (Vulkan).** A World whose device names its objects (`--capture-tool
renderdoc` or `--debug-layers`) creates every compute pipeline with
`VK_PIPELINE_CREATE_CAPTURE_STATISTICS_BIT_KHR` and writes the driver's
statistics as one `[pipeline-statistics] <pipeline name> (...)` line on stderr
(`VulkanPipelineStatistics`): `Register Count`, `Local Memory Size` (NVIDIA sets
bit 36; the low 32 bits are the bytes) and `Shared Memory Size`, which on NVIDIA
includes private arrays the compiler moved into shared memory (the resolve's
25-float luma window shows as 6400 bytes a 64-thread group). Every
`world.shaders.reload` that builds a new pipeline prints its line, so a register
change reads in seconds: reload the edited tree, read the line, then reload an
older tree (a `git archive` of the base revision's `Sdf` and `Shared` shader
directories) to compare in the same World. A pipeline the runtime already holds
under the same key prints nothing again. Keep such trees outside `src/`: the
shader generator refuses any `*.interface.hlsli` no engine package owns, so a
copy under `src/` fails the next World build, and a copy under
`Puck.SdfVm/Assets/Shaders` would also enter the build's kernel glob. A copy
outside any checkout or deployment finds `Shared/*.hlsli` in the running
World's own `Assets/Shaders` (`ShaderIncludeDirectories`); a compile error
keeps the kernels installed before the reload.

**PIX counters (Direct3D 12).** `gpu.capture` a PIX capture, then
`pixtool open-capture <capture> save-event-list <csv> --counters=<name>` under
`puck lease gpu --kind Step` (each counter its own `--counters`, no spaces in a
pattern: `--counters=Compute*Register*`). Each pass row (`world/sdf.world$views`)
then carries `lts__throughput.avg.pct_of_peak_sustained_elapsed` and
`lts__t_bytes.sum` (L2), `dram__bytes_read.sum` and `dram__bytes_write.sum`,
`l1tex__t_sectors_pipe_lsu_mem_local_op_ld.sum` and `_st.sum` (local memory),
`sm__warps_active_shader_cs.avg.pct_of_peak_sustained_elapsed` (achieved
occupancy), `Compute Register Allocation(%)` and `sm__inst_executed.sum`. A
debug-world capture is about 1.8 GB and its replay can hold 17 GB of memory for
many minutes, so collect one capture's counters at a time.

**Nsight Graphics GPU Trace (Direct3D 12 and Vulkan).** Its Shader Profiler
supports both APIs ([NV-SHPROF]). Launch the World under the dotnet host, never
`Puck.World.exe` ([kernels](kernels.md#captures-faults-and-validation)). Leave
out `--platform`: Qt reads it as its own platform-plugin switch and `ngfx` stops
with "no Qt platform plugin could be initialized". Run under Git Bash with
`MSYS_NO_PATHCONV=1`, or an absolute path in `--args` is rewritten. The traced
World did not attach (the launcher kept "Searching for attachable processes")
on Nsight Graphics 2026.2 with the Direct3D 12 backend:

```text
<Nsight install>/host/windows-desktop-nomad-x64/ngfx.exe
  --activity "GPU Trace Profiler"
  --exe "C:/Program Files/dotnet/dotnet.exe" --dir <worktree>
  --args "src/Puck.World/bin/Release/net11.0/Puck.World.dll --world worlds/debug --devices none --backend directx"
  --start-after-hotkey --limit-to-frames 3 --set-gpu-clocks base
  --architecture Ada --metric-set-name "Throughput Metrics" --real-time-shader-profiler
  --auto-export --output-dir <worktree>/.claude/scratch/<lane>-gputrace
```

Every flag above is listed by `ngfx --help-all`; `--architecture` names the
GPU's generation (`Ada` for an RTX 40-series card, with its metric sets listed
there) and `--auto-export` exports the metrics data after the trace. Which
metric set carries the real-time shader profiler (the UI marks compatible sets
with a flame), whether the export holds the Shader Pipelines table, and whether
the report writes without the UI open vary by Nsight version; open the capture
in `ngfx-ui` when the table is missing. Press F11 once the bake and GI have
settled. In the report:

- **Shader Pipelines table:** `# Reg` is registers per thread and `# Warp` the
  maximum theoretical warp occupancy, its tooltip naming the limiting resource
  ([NV-SHPROF]).
- **Timeline:** the `SM Warp Occupancy` row; the expert system's `Warp Launch
  Stalled by CS Register Count` (or `PS`) names a register-limited pass.
- Lock clocks (`--set-gpu-clocks base`) and record them: a GPU under Remote
  Desktop holds idle clocks that differ by backend
  ([hazards](../../../../docs/development/contributing.md#hardware-and-toolchain-cautions)).

**PIX (Direct3D 12).** Boot with `--capture-tool pix`, `gpu.capture <path>.wpix 1`,
then inspect the installed `pixtool --help save-event-list` and list available
counters with `pixtool open-capture <path>.wpix list-counters`.
`ComputeRegisterAllocation` is a percentage, not registers per thread. Register
launch stalls and warp activity help select an investigation; they do not alone
prove the pass is register-bound. Expanded indirect events are GPU commands, not
CPU recording calls: Puck already issues one counted indirect call per phase.
Large forest captures
can exhaust PIX analysis memory when exporting every hardware counter. Export
selected families instead: `--counters=*Duration* --counters=*Invocations` for timing
and work, then narrow NVIDIA patterns such as `--counters=Compute*Register*`.
Whitespace-free wildcard patterns also avoid the tool's argument parsing
failure on counter names containing spaces. A programmatic capture can contain
valid API events and named objects without a gold screenshot; inspect a presented
World frame separately. PIX analysis needs Windows developer mode
([Direct3D 12 captures](../../../../docs/rendering/directx.md#gpu-captures)).

Keep mesh settlement, lighting convergence and normal-motion timing distinct.
Lockstep can wait synchronously for GI; a run with changing lighting can wait
repeatedly for multi-second relights instead of draining to a lasting idle
state. Return to normal presentation for frame windows, inspect actual GI work,
and sample the moving interval itself. Controlled route measurements pin the clock
with startup `--lockstep` and retain ordinary asynchronous lighting using
`--budgeted-lighting`; `SdfFrame.ConvergesLighting` owns GI draining independently.
That preserves quality settings and budgets, not identical per-tick lighting
freshness or pixels. The observer must establish semantic input equivalence,
and GPU/lighting work stays a measured result. A trace started after walking can miss a
transient slowdown that has already recovered. Record the route's exact ticks
and input beside query, contact, pose and transfer counts. Region staging uses a
compute copy, so `gpu.copies.buffer-bytes` alone misses it: include
`gpu.uploads.host-visible` on the residency upload node (`sdf:world` for the root)
as well as its render node. That upload count includes staging headers, not only
payload. Exact packed-word changes are a separate CPU-work metric; bounded run
coarsening may upload unchanged words too.

**PresentMon (frame pacing, both backends).** Frame pacing is what the display
shows, which `host.frames` does not measure: present-to-present intervals and
their percentiles. Install the console build with `winget install --id
Intel.PresentMon.Console --exact` (no elevation needed); it lands on the PATH
as `presentmon`. Capture a running World with `presentmon --process_id <pid>
--timed 20 --terminate_after_timed --output_file <csv> --no_console_stats
--stop_existing_session --no_track_gpu --no_track_display`. The capture reads
ETW. Without elevation (it needs administrator rights or Performance Log Users
membership), GPU and display tracking fail silently and leave the CSV empty, so
pass the two `--no_track_*` flags. The capture then gives present-to-present
only. That is a weak proxy on Direct3D 12: it is bimodal, because each present
lands at the end of an iteration's work and alternates
with tick and non-tick frames. The time a frame reaches the display needs an
elevated capture.

**RenderDoc (Vulkan).** From a build that carries SPIR-V (a solution build, or
`-p:PuckShaderSpirvEnabled=true`; [kernels](kernels.md#building)), boot with
`--capture-tool renderdoc --backend vulkan`,
`gpu.capture`, open the `.rdc` in `qrenderdoc`, and read the pipeline's
executable statistics: registers and "Local Memory Size" (printed with bit 36
set; the low 32 bits are the bytes). This reading finds a kernel whose
dynamically indexed arrays live in local memory (the local-array rule below).

**Turn registers into occupancy.** An Ada SM holds 64K 32-bit registers and at
most 48 warps of 32 threads ([NV-ADA], [NV-CCAP]). Registers are handed out per
warp in units of 256 (8 a thread, [NV-CCAP]), and the SM's four schedulers each
own a 16,384-register slice and an equal share of the warps, so the resident
warps are a step function of the register count, not 65,536 ÷ (32 × registers):

```text
warps ≤ min(48, 4 × ⌊16,384 ÷ roundUp(32 × registers, 256)⌋)
```

| Registers a thread | Warps | Occupancy |
|---|---|---|
| ≤ 40 | 48 | 100 % |
| 41 to 48 | 40 | 83 % |
| 49 to 56 | 36 | 75 % |
| 57 to 64 | 32 | 67 % |
| 65 to 72 | 28 | 58 % |
| 73 to 80 | 24 | 50 % |
| 81 to 96 | 20 | 42 % |
| 97 to 128 | 16 | 33 % |
| 129 to 168 | 12 | 25 % |
| 169 to 255 | 8 | 17 % |

Cutting registers helps only when the count crosses a row's lower edge: 140 and
168 both leave 12 warps, and 108 and 128 both leave 16. A block size that does
not divide the warps rounds them down further. The table is derived from the
unit and the split, so confirm a kernel against GPU Trace's `# Warp`. A kernel
at 168 registers allows 12 warps (25 %); GPU Trace's `SM Warp Occupancy` row is
the achieved occupancy, which falls at or under that limit.

## Registers set occupancy; occupancy is not speed

A kernel's peak live state sets its registers, and its registers cap how many
waves an SM keeps resident to hide memory latency. More waves help only while
the kernel waits on memory.

- **Measured here.** The pixel passes were register-limited.
  Reducing live state in the views kernel (the paints draw on the albedo before
  the rest of the material goes live, `sdfMaterialPaints`) cut its registers and
  its time on both backends. The resolve summing each tap as it reads it
  (`PuckReconstructionSum`) cut registers and got faster on Direct3D 12 but
  slower on Vulkan, whose kernel then kept fewer loads in flight
  ([rendering plan](../../../../docs/plans/rendering.md#gpu-submission-and-routed-worlds)).
- **Shorten live ranges, not instruction counts.** Load a value where it is
  used, finish one stage's state before the next loads, and keep the one
  `mapCore` call site ([SKILL](../SKILL.md#editing-kernels)): each copy adds its
  whole live state.
- **Judge by time at locked clocks beside the register count,** never the
  count alone.

Sources: occupancy limited by registers, and higher occupancy improving speed
only while it hides latency ([NV-BP], [AMD-OCC]); AMD adds that more waves can
also thrash the caches ([AMD-OCC]).

## Local memory: index arrays by constants

A dynamically indexed local array can live in per-thread local memory, which is
device memory behind the caches, as register spills are ([NV-BP], [AMD-OCC]).
The [SKILL](../SKILL.md#editing-kernels) owns the rule; read "Local Memory
Size" to confirm it. On Vulkan, RenderDoc's executable statistics showed local
memory in the views kernel (the material's paint stops and weathering stages,
the volume ramp) and the sky kernel (its upper-run summaries and the star
spectrum table) until every such loop unrolled to a constant index, then none.
Read local memory on each backend separately. A runtime index into a vector,
a constant-buffer one included (`passGroup.lightMask[i >> 5]`, a float4 of
slot visibilities), makes DXC copy the vector into a local array on every
evaluation: inside the views light walk that was PIX's 21 million local-memory
sectors a frame. Pick the component with `sdfComponent`
([`field/sdf-component.hlsli`](../../../../src/Puck.SdfVm/Assets/Shaders/Sdf/field/sdf-component.hlsli)).

## Keep a wave on one path

A wave runs both sides of a branch its lanes disagree on, with the inactive
lanes masked. Control flow that ends at a barrier must be reached by every lane.

- **Done right.** The views gather runs on every lane of the group, a lane past
  the render extent included, because it ends at the group's barrier
  (`sdfViewsGatherFilter` in [`sdf-views-filter.hlsli`](../../../../src/Puck.SdfVm/Assets/Shaders/Sdf/passes/sdf-views-filter.hlsli)).
  The bake maps adjacent threads to adjacent lattice samples
  (the lattice search in [`sdf-mesh-bake.comp.hlsl`](../../../../src/Puck.SdfVm/Assets/Shaders/Sdf/passes/sdf-mesh-bake.comp.hlsl)),
  so a wave's instance masks and `mapCore` walks agree where the field is
  coherent.
- **One walk in `mapCore`** ([SKILL](../SKILL.md#editing-kernels)): a second
  walk behind a uniform flag cost every bake group the whole program on Turing
  under Direct3D 12.

Sources: SIMT execution of divergent paths and masked lanes ([NV-CUDA-SIMT],
[AMD-RDNA]); a workgroup barrier in control flow that is not uniform across the
group is undefined ([KHR-SPIRV], `OpControlBarrier`; [NV-CUDA-SIMT],
`__syncthreads`).

## Coalesce memory access

Adjacent lanes reading adjacent words make one wide transaction; a stride
splits it.

- **Done right.** The region copy walks its words grid-stride: thread `i`
  copies word `i`, then `i` plus the dispatch's threads
  (the copy loop in [`region-copy.comp.hlsl`](../../../../src/Puck.Shaders/Assets/Shaders/Residency/region-copy.comp.hlsl)).
- **Measured here.** The resolve's history surface and stationary window
  (`SdfWorldPackage.HistorySurfaceWords` and `HistoryWindowWords` an output
  pixel) lie in planes (`sdfHistoryWord` in
  [`sdf-resolve.comp.hlsl`](../../../../src/Puck.SdfVm/Assets/Shaders/Sdf/passes/sdf-resolve.comp.hlsl)).
  With each pixel's words adjacent, every lane of every load and store touched
  its own sector: PIX read the resolve at 92 % of L2 throughput, 2.7 GB of L2
  traffic a 1280×900 frame against 0.5 GB of DRAM. A store moves a whole sector
  to L2 for its four bytes, so strided stores cost most.
- **Unmeasured.** The visibility record is 32 bytes a pixel, field after field
  ([`frame/sdf-visibility.hlsli`](../../../../src/Puck.SdfVm/Assets/Shaders/Sdf/frame/sdf-visibility.hlsli)),
  so one word's load across a wave strides 32 bytes. Its typed load and store
  functions own the layout, so a per-field layout is one module's change; read
  GPU Trace's L1TEX throughput for the surface and views passes before making it.

Sources: [NV-BP] (coalesced access to global memory) and [AMD-RDNA] (writes in
coalesced 256-byte blocks a wave).

## Lay groupshared out by bank

Groupshared memory is banked in 4-byte words; lanes of one access that hit
different words of one bank serialize.

- **Done right.** The views filter tile stores each field in its own array
  (`float4` positions, `uint` shadows, `uint` normals,
  `sdfViewsFilterTile`, `sdfViewsFilterShadows` and `sdfViewsFilterNormals` in [`sdf-views-filter.hlsli`](../../../../src/Puck.SdfVm/Assets/Shaders/Sdf/passes/sdf-views-filter.hlsli)),
  so no array is a struct of fields a lane reads one of.
- **Not conflict-free.** The tile is 11 words wide and an 8×8 group's warp is
  eight lanes by four rows, so a `uint` read at `(lane.y + y) * 11 + lane.x + x`
  puts row 0 (words 1 to 7) and row 3 (words 33 to 39) on the same banks: a
  2-way conflict on `sdfViewsFilterShadows` and `sdfViewsFilterNormals`, 32 reads
  a pixel (the filter's reads). The `float4` array reads a quarter-warp at a time, one tile
  row of eight consecutive entries. Unmeasured.
- **Wrong today.** The sky environment reduction stores `[28][64]` floats and
  then has each of 28 lanes read its row at the same column
  (`sdfSkyReduce` in [`sdf-sky-environment-reduce.comp.hlsl`](../../../../src/Puck.SdfVm/Assets/Shaders/Sdf/passes/sdf-sky-environment-reduce.comp.hlsl)):
  a 64-word stride puts every lane on one bank, a 28-way conflict on each of 64
  reads. Pad the row to 65 words. It runs one group only when the environment
  refreshes, so the cost is small.

Sources: [NV-BP] (32 banks of successive 4-byte words; padding) and
[AMD-LDS] (32 or 64 banks by architecture; padding or XOR swizzles); they
differ on bank count, and a padded stride of 65 avoids conflicts with either.

## Reduce across a wave before an atomic

A wave intrinsic sums lanes in registers; one lane then does the atomic.

- **Done right.** `puckCountWork` and `puckCountShapes` sum with
  `WaveActiveSum` and write one atomic a wave
  (`puckCountWork` in [`ShaderWorkCounters.AppendHlsl`](../../../../src/Puck.Shaders.Model/Interface/ShaderWorkCounters.cs)).
- Wave size is the device's (32 on NVIDIA, 32 or 64 on AMD), so never size data
  by it. A float reduction's order differs by wave size; counts are integers,
  and the sky reduction sums in a fixed order so its coefficients repeat on one
  device.

Sources: [HLSL-WAVE] and [KHR-SUBGROUP] (lane count is the implementation's).

## Pack in half, compute in float unless measured

Puck packs halves for storage (the visibility record's curvature and AO, the
history surface, the probes' depth) and computes in `float`; `-enable-16bit-types`
is on ([`ShaderCompiler.StepsOf`](../../../../src/Puck.Shaders.Model/Compiler/ShaderCompiler.Identity.cs))
and no kernel declares a 16-bit arithmetic type.

- Packing is lossy at the edges: a coverage just under one rounds to 0.9995 in
  half and invents background on an opaque surface, so the resolve bounds a
  constant component before packing ([SKILL](../SKILL.md#which-dispatch-owns-the-code)).
- 16-bit arithmetic in a register-limited pass (views, resolve) is a candidate
  to measure, not a rule.

Sources: AMD documents double-rate packed FP16 and fewer VGPRs on its GPUs
([AMD-RDNA]); the HLSL 16-bit types are opt-in and replace min-precision
([HLSL-16BIT]). Neither measures Puck's kernels, so read GPU Trace's registers
and time before and after any 16-bit arithmetic change.

## Size dispatches inside the limits

A dispatch's group count is capped per dimension, and a group's thread count
should fill whole waves.

- **Measured here.** The region copy once dispatched more groups than the
  device allows in total to stage the mesh region, and GPU-based validation
  refused it. It now caps rows at 64 and has each thread copy several words (`GpuRegion.CopyGroups`,
  [`GpuRegion`](../../../../src/Puck.Abstractions/Gpu/Residency/GpuRegion.cs)).
- The SDF compute kernels' groups are 64 threads (8×8 for the pixel passes,
  `SDF_VISIBILITY_GROUP_EDGE`), a whole number of waves on both vendors.
- The bake dispatches batches of at most `SdfMeshPathBake.BatchTaps` evaluations
  as background GPU work (`SdfGpuWork`): a cold debug-world bake is about a
  thousand jobs of a few milliseconds each ([cold bake](../../../../docs/plans/open-items.md#bake-proportional-to-change)).

Sources: [MS-DISPATCH] (65,535 groups a dimension) and [VK-LIMITS]
(`maxComputeWorkGroupCount`, per device); [AMD-RDNA] (group sizes a multiple of
64).

## Keep the CPU and GPU from waiting on each other

A frame path waits for the one frame slot it reuses, never for the device to
go idle; it reads results a few frames late and binds what it allocated in
large blocks.

- **Wait for a slot, not the device.** The Direct3D 12 compositor waits on the
  fence of the back buffer it is about to reuse
  ([`DirectXSurfaceCompositor.WaitForFrameSlot`](../../../../src/Puck.DirectX.Presentation/DirectXSurfaceCompositor.cs));
  `WaitIdle` stays for a resize, a reload's install, a capture's readback and
  device loss.
- **Read back late.** The kernel counters arrive a few frames after their frame
  ([SKILL](../SKILL.md#performance-and-cost)), and the background GPU work
  polls each slot's fence at the top of each frame and after each upload and
  never waits on one ([mesh bake](mesh-bake.md#the-library-and-the-store)).
- **Bind read-only where a pass only reads,** or Direct3D 12 keeps the buffer in
  `UNORDERED_ACCESS` and transitions before every reader
  ([kernels](kernels.md#buffer-hazards)).
- **Allocate in blocks.** Both backends bind resources at offsets of 64 MiB
  blocks (`VulkanMemoryArena`, `DirectXHeapArena`, one `GpuMemoryPool` policy), which cut the native
  allocations of a debug-world boot by an order of magnitude on both backends.
  While every Vulkan resource was its own allocation, a crossing's submissions
  could block for tens of milliseconds inside `vkQueueSubmit`; the rendering
  plan attributes that to residency paid on first reference.
- **Write descriptors only when they change.** Skipping writes a set already
  holds cut the debug world's per-frame descriptor writes on Vulkan by an order
  of magnitude.
- **Record only what changed.** Session and window views whose residencies
  move with the tick record on tick frames only, so `RenderFrame` is several
  times cheaper between ticks.
- **Keep rated work off the step's frame.** A `world.rates` floor updates early
  on a frame that runs no simulation step when the next one runs a step
  (`PresentationRate.Holds`, `FrameContext.StepsNext`), so rated bodies, views,
  screens and the far cascade land between ticks; a body no camera, shadow
  cascade or reader of its parts may show holds (`SdfUpdateRateTiers.HeldHertz`).
  Holding what nothing shows is look-neutral and applies everywhere. A lower
  rate for what is visible is the author's to allow (the owner's ruling):
  distant scenery may animate at a reduced tier only where its creation opts
  in (`behavior.reducedRateAtDistance`, which a package's template can expose),
  and the engine default is full rate for anything visible. An opted-in
  placement's tiers are sized for its measured part speed
  (`WorldCreationInstances.MeasurePartSpeed`, one `SdfUpdateRateTiers.BodyHertz`
  for bodies and placements), so slow wind reduces nearer than a body would.
  The items that take a tier together spread over its period: every other
  body, view and screen takes half the period as a lead (`PresentationPhases`),
  and `world.rates`' `burst` reads the most rated views and screens one frame
  updated.
  New rated work decides through the same floor rather than a schedule of its
  own, takes its phase from a `PresentationPhases` of its own, reads the frame's flags before it runs (the stamp pool takes them in
  `PresentAt`), and a new reader of a body's parts goes through the pool's part
  accessors, which record it (`NotePoseRead`).

Sources: minimal fences and no fence where a barrier serves ([NV-SYNC]); heaps
sub-allocated and residency managed per heap ([AMD-RDNA], [MS-RESIDENCY]).
They disagree on two points. NVIDIA prefers committed resources for
uncompressed resources on Direct3D 12 ([NV-MEM]) where AMD sub-allocates
everything; Puck keeps render targets and depth committed and places the rest.
NVIDIA aims for 5 to 10 `ExecuteCommandLists` a frame ([NV-CMD]) where AMD
batches into one submission ([AMD-RDNA]); `world.counters gpu.queue` counts
Puck's per frame (below).

## Count queue work and record jobs

`world.counters gpu.queue` (`GpuQueueWork`) counts every native execute, list,
signal and present the process makes, on both backends and every device. Each
execute is counted under who caused it (`graph`, `bake`, `probe`, `upload`,
`transfer`, `readback`, `present`, `drain`) and why it reached the queue then
(`unbatched`, `batch-end`, `door`, `fence-wait`, `external-wait`, `drain`,
`empty-fence`, `job-end`); lists count under the category they were deferred with. Divide
two readings' deltas by the `frames` delta (presents). `unattributed` nonzero
past boot means a submitter that names no category: give it a
`GpuQueueWork.Attribute` scope rather than reading it as a category of work. A
Vulkan fence or timeline signal rides its `vkQueueSubmit`, so Vulkan's
`signals` are carried completions, not separate calls.

- **One frame should be about two executes plus its background jobs:** the graph
  batch's `batch-end` and the presenter's own. Background `bake` and `probe` jobs
  pumped between frames submit unbatched. Jobs pumped inside the frame's batch
  (`RenderGraphRuntime`'s package preparation) take one hand-over of their own,
  counted as `job-end`: the first frame submission deferred behind a run of jobs
  hands the run to the queue (`GpuSubmissionBatch.Admit`), so the jobs start
  while the frame records and their fences answer from that hand-over's signal
  rather than the frame's last. `jobs` counts the jobs' fences batches handed
  over and `jobs.late` those still answered behind frame work, which reads zero.
  `external-wait` is the cost to watch: on Direct3D 12 an outstanding external
  wait must split the batch, since `ID3D12CommandQueue::Wait` holds everything
  after it on the queue, so every sampling node would split the frame. The
  submitters drop a wait whose fence already reached its value
  (`GpuExternalWait.IsReached`), and `waits` / `waits-reached` count both.
  An export's completion signal reads the queue door, which would split the
  batch, so the runtime completes exports after `FlushBatch`
  (`RenderGraphRuntime.SettleHandOvers`): `transfer.door` stays zero, and the
  reader's wait ends when the whole batch has run rather than after the exporting
  node's lists. Each export is published only when its own copy's submission
  reached the queue (`IGpuSubmissionFence.ReachedQueue`), never on the flush's
  result: the batch may go over in several hand-overs, and any one can fail.
- **`world.counters render.record`** (`RenderRecordWork`) gives the host time of
  render nodes' record jobs per phase: preparation, recording and commit, the
  serial record sum, and `longest-microseconds`, each frame's longest single
  recording. Divided by `frames`, the longest is the critical path a frame would
  keep with its jobs recorded side by side; its gap to the serial sum is all that
  concurrent recording could save. The counter is process-wide: the instance behind
  the critical path is on `world.budget`'s live-frame rows and in `pipeline.inspect`,
  which carry each node's `record-jobs`, `last-` and `mean-` `prepare`, `record` and
  `commit` microseconds and its `longest-record-us`
  (`ShaderPipelineRenderNode.AppendRecordTimes`). `packages-microseconds` is the
  frame's package preparation before any job (the SDF residency's upload and
  pump). Most of it is usually the upload's host wait on the previous upload's
  fence: the CPU waiting for the GPU, which a thread-time trace shows under
  `GpuWorkCountingFence.Wait`. Read it as GPU-bound pacing, not host work.

## Puck's open cases

- **Register-limited pixel passes.** On Vulkan (RTX 4070) the plain views,
  binned views and glass kernels and every specialization of them run at 168
  registers (12 of the SM's 48 warps), so a specialized set adds no registers
  over the kernels that run no program; the interpreter's views, binned views
  and glass run at 255 (8 warps), the probe lookup at 128 and the views filter
  at 40, and the specialized and plain glass keep 96 and 16 bytes of local
  memory. The interpreter costs tens of times the specialized kernels where a
  large program covers the view ([author shaders](../../../../docs/plans/author-shaders.md#the-decision-matrix)).
  The interpreter's light phase, inlined in the light walk with its sixteen
  named `uint4` registers and the whole stamp, sets its count, and less live
  probe state does not lower it; keeping the interpreter out of the pixels
  that run no program, and specialization, does
  ([frame bar](../../../../docs/plans/open-items.md#other-frame-time-findings)).
  Resolve reads 168 and composite 96 on Vulkan; PIX on Direct3D 12 once read
  the interpreted views at 98 % register allocation with 16 % of the warps
  active, and no Direct3D 12 occupancy of the specialized or binned kernels has
  been read. `world.indirect off` still removes a third of views on Vulkan:
  that share is the gather's work.
- **The main mesh pass shades several fragments a pixel,** with no
  front-to-back order and no depth prepass ([rendering plan](../../../../docs/plans/rendering.md#gpu-submission-and-routed-worlds)).
- **Growing the program region waits for the device to go idle**
  ([hot reload](../../../../docs/plans/open-items.md#edit-to-running-hot-reload-and-boot)).
- **Tick-bound views record on tick frames,** beside the tick itself
  ([simulation tick](../../../../docs/plans/open-items.md#engine-job-system)).
- **The sky reduction and the views filter tile have bank conflicts.** The
  filter tile's conflict is 2-way ([frame time](../../../../docs/plans/open-items.md#engine-job-system)).
- **Bake register pressure.** `mapCore` switches between named parent fields,
  and `sdfBakeTap` selects each stencil direction. Read the driver's register,
  local and shared memory counts on the resulting pipeline
  ([field programs](../../../../docs/plans/field-programs.md#build-order)).
- **Probe arrays leave registers.** The gather's arrays (`SdfMeshProbes`'
  sixteen weights, a stage's eight in `SdfMeshProbeCage`) live in memory rather
  than registers: shared memory on NVIDIA's Vulkan driver (`Shared Memory Size`
  against the 2.9 KB the filter tile declares), local memory on Direct3D 12.
  A gather names each stage by its cage word (`sdfMeshProbeCageWord`), never
  sixteen addresses, which spilled sixteen times the local-memory traffic; PIX
  still reads 2.6 million load and 4.2 million store sectors of local memory
  in views a frame at the teal room's floor (1280×900). Guarding the probe
  loops' `continue`s removed the Vulkan spills but raised views to 168
  registers and made it slower.

## Sources

- [NV-SHPROF] NVIDIA, [Nsight Graphics: Shader Profiler](https://docs.nvidia.com/nsight-graphics/UserGuide/shader-profiler.html).
- [NV-ADA] NVIDIA, [Ada tuning guide](https://docs.nvidia.com/cuda/ada-tuning-guide/index.html) (48 warps and 64K registers an SM, 255 registers a thread).
- [NV-CCAP] NVIDIA, [CUDA programming guide: compute capabilities](https://docs.nvidia.com/cuda/cuda-programming-guide/05-appendices/compute-capabilities.html).
- [NV-BP] NVIDIA, [CUDA C++ best practices guide](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html) (occupancy, coalescing, shared memory banks).
- [NV-CUDA-SIMT] NVIDIA, [CUDA C++ programming guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html) (SIMT architecture).
- [NV-SYNC] NVIDIA, [Advanced API performance: synchronization](https://developer.nvidia.com/blog/advanced-api-performance-synchronization/).
- [NV-MEM] NVIDIA, [Advanced API performance: memory and resources](https://developer.nvidia.com/blog/advanced-api-performance-memory-and-resources).
- [NV-CMD] NVIDIA, [Advanced API performance: command buffers](https://developer.nvidia.com/blog/advanced-api-performance-command-buffers/).
- [AMD-OCC] AMD GPUOpen, [Occupancy explained](https://gpuopen.com/learn/occupancy-explained/).
- [AMD-RDNA] AMD GPUOpen, [RDNA performance guide](https://gpuopen.com/learn/rdna-performance-guide/).
- [AMD-LDS] AMD ROCm, [Understanding AMD GPU LDS and bank conflicts](https://rocm.docs.amd.com/projects/composable_kernel/en/latest/conceptual/ck_tile/hardware/lds_bank_conflicts.html).
- [HLSL-WAVE] Microsoft, [Wave intrinsics](https://github.com/microsoft/DirectXShaderCompiler/wiki/Wave-Intrinsics); [HLSL-16BIT] [16-bit scalar types](https://github.com/microsoft/DirectXShaderCompiler/wiki/16-Bit-Scalar-Types).
- [KHR-SUBGROUP] Khronos, [Vulkan subgroup tutorial](https://www.khronos.org/blog/vulkan-subgroup-tutorial).
- [KHR-SPIRV] Khronos, [SPIR-V specification](https://registry.khronos.org/spir-v/specs/unified1/SPIRV.html) (`OpControlBarrier`).
- [KHR-PEP] Khronos, [`vkGetPipelineExecutableStatisticsKHR`](https://docs.vulkan.org/refpages/latest/refpages/source/vkGetPipelineExecutableStatisticsKHR.html).
- [MS-DXIL] Microsoft, [DirectX intermediate language](https://github.com/microsoft/DirectXShaderCompiler/blob/main/docs/DXIL.rst).
- [MS-DISPATCH] Microsoft Learn, [ID3D12GraphicsCommandList::Dispatch](https://learn.microsoft.com/en-us/windows/win32/api/d3d12/nf-d3d12-id3d12graphicscommandlist-dispatch).
- [VK-LIMITS] Khronos, [VkPhysicalDeviceLimits](https://docs.vulkan.org/refpages/latest/refpages/source/VkPhysicalDeviceLimits.html).
- [MS-RESIDENCY] Microsoft Learn, [Residency](https://learn.microsoft.com/en-us/windows/win32/direct3d12/residency).
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall