Thirty frames of vision, one hundred and twenty frames of interaction_

Saccade has two performance targets that look incompatible at first: a complete full-scope neural refresh at 30 Hz, and interaction plus overlay presentation at 120 Hz. If one desktop action depends on the newest model result, how can the interface respond four times while vision finishes once?

The neural loop and the interaction loop run on different clocks: 30 Hz for scene publication and 120 Hz for selection and input. Immutable scenes connect the two loops, so the interaction loop can use the latest complete scene while the neural loop builds the next one. Each published target carries its identity and the generations that the action preflight checks before native input.

Start with the deadlines

A 30 Hz loop has 33.33 milliseconds between scene deadlines. A 120 Hz loop has 8.33 milliseconds between interaction ticks. If capture, model execution, hint reduction, and display submission all ran in one loop, every stage would get the smaller deadline. Only the final two need it.

A model pass that takes longer than usual would delay key handling and presentation. A compositor delay would postpone the next capture. An accessibility query in an uncooperative application could block all three: key handling, presentation, and capture. The average might look acceptable while the user experiences the sum of those delays.

Separate clocks keep those delays from accumulating. The neural loop may spend most of its 33.33 milliseconds on capture and inference. The interaction loop reads the latest complete scene without calling the model. The display loop reads a static overlay packet plus a small active-state update.

A desktop frame is already large

For uncompressed BGRA8, frame payload is width * height * 4 bytes. A 3840 by 2160 frame contains 33,177,600 bytes, or about 31.64 MiB. At 30 frames per second, that is about 949 MiB/s of pixel payload. A 7680 by 4320 frame contains about 126.56 MiB, and its 30 Hz payload rate is about 3.71 GiB/s.

Those figures are raw payload, not measured bus traffic. The real traffic can differ because native imports may avoid a copy, row pitch adds padding, and tiling or compression varies by platform. The figures still show why an accidental full-frame copy can cost more than many compact target operations.

Ten seconds of raw 4K BGRA8 at 30 Hz is about 9.27 GiB. Saccade is not a recorder, so retaining that history would be wasted memory and a privacy mistake. The runtime needs the newest useful frame, not every frame that ever arrived.

A FIFO makes old frames older

Suppose capture produces frame 100, then 101, then 102 while inference is still processing frame 99. A FIFO queue preserves arrival order, so inference would get frame 100 next. Saccade needs frame 102, the newest frame that inference has not started.

If capture stays ahead of inference, queue depth and latency grow together. A depth of ten at 30 Hz is roughly one third of a second of old desktop state, before model time is added. A bounded queue prevents unbounded memory, but it does not make the oldest retained frame more relevant.

Saccade uses newest-frame replacement. The provider owns one active frame, and the coordinator owns one pending slot per display. If work on the pending frame has not started, a new capture replaces it. The older frame's lease retires immediately, because running that frame could no longer improve the next scene.

The mailbox has exactly one pending handle, one producer, and one consumer. Replacement and consumption are each a single atomic exchange. The counters live on separate cache lines. The complete mailbox fits in one header.

Snapshot one pending frame per display

One newest frame for the whole desktop would introduce another failure. A fast-changing display could replace every pending frame from a quieter display. The resulting scene would be recent, but incomplete across the selected desktop scope.

So the coordinator owns one newest pending frame per display. At the next neural deadline, the coordinator includes frame 102 in a snapshot that keeps one candidate from each selected display. That batch runs sequentially on the accelerator lane, which one scheduler thread owns, and publishes one immutable scene. If frame 103 arrives during the batch, it goes into the next per-display snapshot and cannot change the scene built from frame 102.

One accelerator lane simplifies ownership, but serializes the per-display batch. The coordinator supplies full-scope fairness above the lane. A display cannot starve its neighbor, and two provider calls never race over one scratch tensor.

Fixed slots make memory independent of time

Capacity sets steady-state memory.

Capture keeps a fixed number of native frame slots. Preprocessing reuses its output textures or tensors. Model and postprocessing storage is allocated to configured limits. Each overlay surface has three presentation slots.

The configured limits are 16 displays, three capture slots per stream, one replaceable pending frame per display, and three overlay presentation slots per surface. These are capacities, not observed desktop requirements. Display capacity and capture slot capacity are explicit in source.

None of those counts grows when the application runs for another hour. Resolution changes source surfaces and some presentation buffers. Candidate capacity changes model-output and postprocess storage. Each display adds capture and overlay state. Capture duration does not change steady-state storage.

steady_state_bytes =
    capture_slots(display_count, source_resolution)
  + preprocess_lanes(model_shape)
  + model_context(runtime_context)
  + postprocess(candidate_capacity, target_capacity)
  + overlay_slots(display_count, surface_resolution)

Framework and driver memory are separate from the Saccade-owned bytes. Core ML, Windows ML, Metal, D3D12, the compositor, and the capture framework own allocations that the application cannot describe by inspecting its C++ buffers. A complete memory report has to show both the bytes Saccade owns and the operating-system residency. Saying the hot path performs no Saccade-owned allocation is narrower than saying the process allocates nothing.

Keep the wide data on one device

The model input and intermediate feature maps are wider than the final scene. Where the platform allows it, Saccade keeps capture preparation, inference, target decoding, sorting, and suppression on the GPU. What the CPU sees is compact target metadata, not feature maps or every dense candidate row.

On Windows, a production capture surface arrives through the platform's D3D11 capture API. A bounded transfer owner copies the surface once on the GPU into a shared D3D12 resource and signals a shared fence. The D3D12 inference worker waits on that fence from its own queue. The capture owner never touches the inference queue. The interaction owner and the DirectML worker each keep one high-priority Multimedia Class Scheduler Service (MMCSS) registration for their lifetime. They register outside both hot loops. The interaction owner and DirectML worker apply that policy independently.

On macOS, preprocessing writes either a persistent IOSurface-backed image view or a persistent tensor buffer, depending on the admitted model. The inverse letterbox mapping stays with the preprocessing lane. Core ML executes the graph. Saccade keeps the frame ticket and bounded target output.

The expensive boundary is a full-frame host readback. Windows keeps the unavoidable capture handoff as one GPU-local D3D11-to-D3D12 copy. After that copy, preprocessing, DirectML, ordering, suppression, and the compact packet stay on the D3D12 lane. macOS borrows IOSurface-backed textures into one persistent Metal atlas and hands an IOSurface-backed image to Core ML. Only the bounded target packet crosses to CPU scene code. The macOS ownership path and the Windows path show the transfer boundary.

The interaction loop reads immutable state

The interaction loop never waits for the active neural ticket. Once the neural loop publishes the scene built from frame 102, a hint activation keeps reading that immutable scene while frame 103 is captured and processed. Each interaction tick reduces physical key state, advances the hint prefix, updates selection, checks deadlines, and publishes active overlay state without entering the model.

Suppose I select a target from the frame 102 scene while frame 103 is still running. The action preflight compares frame 102's window, transform, focus, permissions, and input generations with the current desktop. A mismatch rejects the action. A match allows native input without waiting for frame 103. The action preflight checks show the complete comparison.

The display loop does not redo static work on every tick. Static targets expand only when the scene or transform changes. A display tick updates the active target, inverse surface size, and presentation time, then issues one bounded draw. After the reveal and active animation finish, the macOS implementation pauses CAMetalDisplayLink. A scene or active-state change wakes it again. The pause decision is in the callback.

The two timing targets

On supported hardware, the 30 Hz neural target requires the complete selected scope to influence a result every 33.33 milliseconds. Region priority may decide which work starts first or how Saccade recovers from overload. It cannot permanently turn full-scope vision into a collection of favored crops.

While neural work is active, the 120 Hz interaction target requires key reduction and pointer feedback to remain within an 8.33 millisecond period. The target does not imply that a new model result exists at every presentation tick. It also does not justify spinning every display when the scene and active state are unchanged.

The useful measurements follow the three loops. The neural loop reports capture age and full-scope latency. The interaction loop reports tick latency while inference is busy. The display loop reports missed compositor deadlines and slot pressure. One frames-per-second number would mix them together.