Thirty frames of vision, one hundred and twenty frames of interaction_

Saccade has two performance targets that look incompatible at first: a complete full-scope neural refresh at 30 Hz, and interaction plus overlay presentation at 120 Hz. If one desktop action depends on the newest model result, how can the interface respond four times while vision finishes once?

The model and interaction loop run on different clocks: 30 Hz for scene publication and 120 Hz for selection and input. Immutable scenes connect them, so input can use the latest complete scene while the next one is still being built. Each published target carries the identity and generations checked by the action preflight before native input.

Start with the deadlines

A 30 Hz path has 33.33 milliseconds between scene deadlines. A 120 Hz path has 8.33 milliseconds between interaction ticks. Putting capture, model execution, hint reduction, and display submission in one loop would give every stage the smaller deadline, even though only the final two need it.

A model pass that takes longer than usual would delay key handling and presentation. A compositor delay would postpone the next capture. An accessibility query in an uncooperative application could block all three. The average might look acceptable while the user experiences the sum of those delays.

Separate clocks keep those delays from accumulating. The neural loop may spend most of its 33.33 milliseconds on capture and inference. The interaction loop reads the latest complete scene without calling the model. The display loop reads a static overlay packet plus a small active-state update.

A desktop frame is already large

For uncompressed BGRA8, frame payload is simply width * height * 4. A 3840 by 2160 frame contains 33,177,600 bytes, or about 31.64 MiB. At 30 frames per second, that represents about 949 MiB/s of pixel payload. A 7680 by 4320 frame contains about 126.56 MiB, and its 30 Hz payload rate is about 3.71 GiB/s.

Those figures are raw payload, not measured bus traffic. Native imports may avoid a copy, row pitch adds padding, and tiling or compression varies by platform. They still show why an accidental full-frame copy can cost more than many compact target operations.

Ten seconds of raw 4K BGRA8 at 30 Hz is about 9.27 GiB. Saccade is not a recorder, so retaining that history would be wasted memory and a privacy mistake. The runtime needs the newest useful frame, not every frame that ever arrived.

A FIFO makes old frames older

Suppose capture produces frame 100, then 101, then 102 while inference is still processing frame 99. A FIFO queue preserves arrival order. Saccade instead needs the newest frame that inference has not started.

If production stays ahead of consumption, queue depth and latency grow together. A depth of ten at 30 Hz represents roughly one third of a second of old desktop state before model time is added. Bounding the queue prevents unbounded memory, but it does not make the oldest retained frame more relevant.

Saccade uses newest-frame replacement. One active frame belongs to the provider. One pending slot per display belongs to the coordinator. A new capture replaces that pending lease if work has not started. The older lease retires immediately because running it could no longer improve the next scene.

The mailbox has exactly one pending handle, one producer, and one consumer. Replacement and consumption are single exchanges. The counters live on separate cache lines. The complete mailbox fits in one header.

Snapshot one pending frame per display

Keeping one newest frame globally would introduce another failure. A fast-changing display could replace every pending frame from a quieter display. The resulting scene would be recent, but incomplete across the selected desktop scope.

The coordinator therefore owns one newest pending frame per display. At the next neural deadline it includes frame 102 in a snapshot that preserves one candidate from each selected display. That batch runs sequentially on the owned accelerator lane and publishes one immutable desktop scene. A frame 103 that arrives during the batch goes into the next per-display snapshot and cannot alter the scene being built from frame 102.

One accelerator lane simplifies ownership, but serializes the per-display batch. The coordinator supplies full-scope fairness above it. A display cannot starve its neighbor, and two provider calls never race over one scratch tensor.

Fixed slots make memory independent of time

Capacity sets steady-state memory.

Capture keeps a fixed number of native frame slots. Preprocessing reuses its output textures or tensors. Model and postprocessing storage is allocated to configured limits. Each overlay surface has three presentation slots.

The configured limits are 16 displays, three capture slots per stream, one replaceable pending frame per display, and three overlay presentation slots per surface. These are capacities, not observed desktop requirements. Display capacity and capture slot capacity are explicit in source.

None of those counts grows because the application ran for another hour. Resolution changes source surfaces and some presentation buffers. Candidate capacity changes model-output and postprocess storage. Each display adds capture and overlay state. Capture duration does not change steady-state storage.

steady_state_bytes =
    capture_slots(display_count, source_resolution)
  + preprocess_lanes(model_shape)
  + model_context(runtime_context)
  + postprocess(candidate_capacity, target_capacity)
  + overlay_slots(display_count, surface_resolution)

Framework and driver memory remain separate. Core ML, Windows ML, Metal, D3D12, the compositor, and the capture framework own allocations the application cannot describe by inspecting its C++ buffers. A complete memory report has to show both owned bytes and operating-system residency. Saying the hot path performs no Saccade-owned allocation is narrower than saying the process allocates nothing.

Keep the wide data on one device

The model input and intermediate feature maps are wider than the final target scene. Saccade keeps capture preparation, inference, target decoding, sorting, and suppression on the GPU where the platform allows it. The CPU-visible boundary contains compact target metadata, not feature maps or every dense candidate row.

On Windows, a production capture surface arrives through the platform's D3D11 capture API. A bounded transfer owner copies it once on the GPU into a shared D3D12 resource and signals a shared fence. The D3D12 inference worker waits on that fence from its own queue. The 120 Hz owner and DirectML worker each keep one high-priority Multimedia Class Scheduler Service (MMCSS) registration for their lifetime, outside both hot loops. The interaction owner and DirectML worker apply that policy independently. The capture owner never reaches into the inference queue.

On macOS, preprocessing writes either a persistent IOSurface-backed image view or a persistent tensor buffer, depending on the admitted model. The inverse letterbox mapping stays with the lane. Core ML executes the graph. Saccade keeps the frame ticket and bounded target output.

The expensive boundary is a full-frame host readback. Windows keeps the unavoidable capture handoff as one GPU-local D3D11-to-D3D12 copy, then leaves preprocessing, DirectML, ordering, suppression, and the compact packet on the D3D12 lane. macOS borrows IOSurface-backed textures into one persistent Metal atlas and hands an IOSurface-backed image to Core ML. Only the bounded target packet crosses to CPU scene code. The macOS ownership path and the Windows path show the transfer boundary.

The interaction clock reads immutable state

The interaction owner never waits for the active neural ticket. Once the scene built from frame 102 is published, a hint activation keeps reading that immutable generation while frame 103 is captured and processed. Each tick reduces physical key state, advances the hint prefix, updates selection, checks deadlines, and publishes active overlay state without entering the model.

If I select a target from the frame 102 scene while frame 103 is still running, the action preflight compares frame 102's window, transform, focus, permissions, and input generations with the current desktop. A mismatch rejects the action. A match allows native input without waiting for frame 103. The action-time checks show the complete comparison.

The display loop does less work. Static targets expand only when the scene or transform changes. A display tick updates the active target, inverse surface size, and presentation time, then issues one bounded draw. After the reveal and active animation finish, the macOS implementation pauses CAMetalDisplayLink. A scene or active-state change wakes it again. The pause decision is in the callback.

The two timing targets

The 30 Hz neural target requires the complete selected scope to influence a result every 33.33 milliseconds on supported hardware. Region priority may decide which work starts first or how overload recovers, but it cannot permanently turn full-scope vision into a collection of favored crops.

The 120 Hz interaction target requires key reduction and pointer feedback to remain within an 8.33 millisecond period while neural work is active. It does not imply that a new model result exists at every presentation tick. It also does not justify spinning every display when the scene and active state are unchanged.

The useful measurements follow the three clocks. Vision reports capture age and full-scope latency. Interaction reports tick latency while inference is busy. Presentation reports missed compositor deadlines and slot pressure. One frames-per-second number would mix them together.