Thirty frames of vision, one hundred and twenty frames of interaction_

Saccade has two performance targets that look incompatible at first: a complete full-scope neural refresh at 30 Hz, and interaction plus overlay presentation at 120 Hz. If one desktop action depends on the newest model result, how can the interface respond four times while vision finishes once?

The answer is not to make the model four times faster by declaration. It is to stop treating seeing and responding as one serial loop. Saccade gives each a separate clock, publishes immutable scenes between them, and makes freshness part of action validation. This article derives that arrangement from the deadlines and from the memory cost of a desktop frame.

The numbers below are format arithmetic and design targets, not a performance report for a particular machine. The goal is to explain what must remain bounded before hardware measurements can mean anything.

Start with the deadlines

A 30 Hz path has 33.33 milliseconds between scene deadlines. A 120 Hz path has 8.33 milliseconds between interaction ticks. Putting capture, model execution, hint reduction, and display submission in one loop would give every stage the smaller deadline, even though only the final two need it.

That serial loop also couples variance. A model pass that takes longer than usual would delay key handling and presentation. A compositor delay would postpone the next capture. An accessibility query in an uncooperative application could block all three. The average might look acceptable while the user experiences the sum of unrelated tails.

Separate clocks turn those tails into explicit handoffs. The neural owner may spend most of its 33.33 millisecond period on capture, preprocessing, inference, postprocessing, and scene publication. The interaction owner reads the latest complete scene and never calls the model runtime. The display owner reads the latest static overlay packet and a tiny active-state update. Each owner has a bounded object to consume and a bounded point where new state becomes visible.

A desktop frame is already large

For uncompressed BGRA8, frame payload is simply width * height * 4. A 3840 by 2160 frame contains 33,177,600 bytes, or about 31.64 MiB. At 30 frames per second, that represents about 949 MiB/s of pixel payload. A 7680 by 4320 frame contains about 126.56 MiB, and its 30 Hz payload rate is about 3.71 GiB/s.

Those rates are not a claim about bus traffic. Native surface import may avoid a copy, tiling and compression are platform-specific, row pitch adds padding, and the model sees a smaller fitted tensor. The arithmetic establishes the scale of the object whose ownership we are designing. One careless extra full-frame copy can be more expensive than many compact target operations.

Time multiplies the problem. Ten seconds of raw 4K BGRA8 at 30 Hz is about 9.27 GiB. Saccade is not a recorder, so retaining that history would be wasted memory and a privacy mistake. The runtime needs the newest useful frame, not every frame that ever arrived.

A queue preserves the wrong property

Suppose capture produces frame 100, then 101, then 102 while inference is still processing frame 99. A conventional queue preserves arrival order. Once frame 99 finishes, the worker starts 100 even though 102 is already a better description of the live desktop.

If production stays ahead of consumption, queue depth and latency grow together. A depth of ten at 30 Hz represents roughly one third of a second of old desktop state before model time is added. Bounding the queue prevents unbounded memory, but it does not make the oldest retained frame more relevant.

Saccade uses newest-frame replacement. One active frame belongs to the provider. One pending slot per display belongs to the coordinator. A new capture replaces the pending lease for that display if it has not started. The replaced lease retires immediately through its owner; it does not wait behind work that can no longer improve the next scene.

Multiple displays need a snapshot, not a race

Keeping one newest frame globally would introduce another failure. A fast-changing display could replace every pending frame from a quieter display. The resulting scene would be recent, but incomplete across the selected desktop scope.

The coordinator therefore owns one newest pending frame per display. At a neural deadline it snapshots up to the fixed display capacity, preserving one candidate from each selected display. That batch runs sequentially on the owned accelerator lane and commits as one desktop scene. Frames arriving during the batch populate the next per-display snapshot.

This is a deliberate compromise. The provider still executes one frame at a time, which keeps its workspace, queue ownership, and cancellation state simple. The coordinator supplies full-scope fairness above it. A display cannot starve its neighbor, and two provider calls never race over one scratch tensor.

Fixed slots make memory independent of time

Once history is gone, memory can be expressed as capacities rather than duration. Capture owns a fixed number of native frame slots. Preprocessing owns persistent output textures or tensors. The model context owns its packed weights and workspace. Postprocessing owns fixed candidate, key, mask, and packet buffers. The overlay owns three in-flight presentation slots per surface.

None of those counts grows because the application ran for another hour. Resolution changes the size of source surfaces and some presentation buffers. Candidate capacity changes model-output and postprocess storage. Display count multiplies per-display capture and overlay state. Time should not be a term in the steady-state memory formula.

steady_state_bytes =
    capture_slots(display_count, source_resolution)
  + preprocess_lanes(model_shape)
  + model_context(weights, workspace)
  + postprocess(candidate_capacity, target_capacity)
  + overlay_slots(display_count, surface_resolution)

Framework and driver memory remain separate. Core ML, Windows ML, Metal, D3D12, the compositor, and the capture framework own allocations the application cannot describe by inspecting its C++ buffers. A complete memory report has to show both owned bytes and operating-system residency. Saying the hot path performs no Saccade-owned allocation is narrower than saying the process allocates nothing.

Keep the wide data on one device

The model input and intermediate feature maps are wider than the final target scene. Saccade keeps capture preparation, inference, target decoding, sorting, and suppression on the GPU where the platform allows it. The CPU-visible boundary contains compact target metadata, not feature maps or every dense candidate row.

On Windows, a production capture surface arrives through the platform's D3D11 capture API. A bounded transfer owner copies it once on the GPU into a shared D3D12 resource and signals a shared fence. The D3D12 inference worker waits on that fence from its own queue. The capture owner never reaches into the inference queue.

On macOS, preprocessing can write a persistent IOSurface-backed image view or a persistent tensor buffer depending on the admitted model. The inverse letterbox mapping travels with the lane. Core ML owns opaque graph execution; Saccade owns the frame ticket and the bounded target output.

In both paths, device ownership is more important than the name of an individual kernel. A fast resize followed by a CPU readback can lose to a slower fused dispatch that keeps the tensor beside inference. End-to-end movement decides whether a local optimization is real.

The interaction clock reads immutable state

The interaction owner never waits for the active neural ticket. At each tick it reduces physical key state, advances the hint prefix, updates selection, checks deadlines, and publishes active overlay state. When a new scene is complete, it can start a new activation from that generation. While inference is running, the previous immutable scene remains readable.

This does not mean old targets are always safe to act on. It means freshness is checked at the right boundary. The action plan carries scene, transform, topology, session, focus, and permission context. Immediately before native input, the executor compares that context with the current desktop. A stale scene can remain useful when nothing relevant changed and becomes harmless when something did.

The display path is narrower still. Static target expansion occurs when the scene or transform changes. A display tick changes the active target, inverse surface size, and presentation time, then issues one bounded draw. A static display can sleep after revealing a new scene. Model cadence does not force a transparent overlay to redraw unchanged geometry.

What the targets actually require

The 30 Hz neural target requires the complete selected scope to influence a result every 33.33 milliseconds on supported hardware. Region priority may decide which work starts first or how overload recovers, but it cannot permanently turn full-scope vision into a collection of favored crops.

The 120 Hz interaction target requires key reduction and pointer feedback to remain within an 8.33 millisecond period while neural work is active. It does not imply that a new model result exists at every presentation tick. It also does not justify spinning every display when the scene and active state are unchanged.

Those distinctions shape the measurements that eventually matter. Neural cadence needs capture age, batch latency, full-scope latency, replacements, missed deadlines, and per-display skew. Interaction needs tick latency while the neural worker is busy. Presentation needs slot pressure, compositor deadlines, and visible-frame behavior. A single frames-per-second counter would hide the ownership boundaries the architecture was built to expose.

Reconstructing the two rates

The two clocks are not a concession to a slow model. They follow from the information each path needs. Vision consumes wide, expensive surfaces and produces a compact scene. Interaction consumes that compact scene plus current physical state. Presentation consumes even less: immutable geometry and a small active-state update.

Newest-frame replacement keeps neural latency from becoming queue history. Per-display snapshots keep full-scope work fair. Fixed slots make memory scale with resolution, display count, and configured capacities rather than process lifetime. Epochs let the interaction path reuse a scene without pretending it is timeless.

So 30 and 120 are not two versions of the same loop. They are two different contracts joined by one immutable object. Once that object exists, making the user wait for the next model pass would throw away the separation we worked to create.