Pressing a hint key looks like the last step in a keyboard-driven pointer tool. The label says A, I press A, and the pointer clicks the control underneath it. But the label came from an older view of the desktop. The application may have moved, a dialog may have appeared, the display scale may have changed, or my hand may have touched the mouse after the label was drawn.
Saccade treats that gap as the main problem. It is not enough to find a rectangle that looked clickable. The system has to preserve the identity and coordinate history of that rectangle from native capture, through a local vision model, into a frozen hint map, and finally through one last validation immediately before native input.
This article follows one control through that path. I will use a Save button on the foreground window, assigned the hint A. Model training, permission setup, and the details of each operating-system API stay outside the trace. The proof is whether we can explain exactly which state reaches the click and exactly which changes make the click disappear.
frame 804
same device
local model
compact targets
safe point
native input
The capture is a lease, not an image copy
On macOS, ScreenCaptureKit delivers a native surface. On Windows, Windows Graphics Capture delivers one. Saccade wraps the surface in a bounded frame lease carrying a frame identifier, a capture timestamp, a source transform, and a display-topology generation. The lease says who owns the native object and when it may be retired. It does not turn the desktop into a long-lived CPU bitmap.
That matters before we discuss inference. A 4K BGRA surface contains 33,177,600 bytes. Copying it through general-purpose memory simply to hand it to another device would make bandwidth and lifetime harder at the same time. The native paths instead import or copy within the platform graphics stack, then keep the resulting dependency with the frame. The inference worker cannot consume the surface until that dependency is satisfied, and capture cannot reclaim it until the worker retires the lease.
Frame 804 also carries a topology epoch. Suppose the Save button is on a display whose scale changes while the model is running. The pixels still describe a real button, but the old source-to-desktop transform no longer describes its destination. Keeping the topology and transform beside the frame lets the later scene owner reject the result instead of stretching an old coordinate into a new desktop.
The model sees a bounded tensor
The model does not receive the full capture object as an abstract image. An owned Metal or D3D12 preprocessing kernel maps the selected scope into the model's fixed input shape. It applies the aspect-preserving fit, letterbox placement, channel mapping, and normalization in one device dispatch. The placement parameters are retained because model coordinates are meaningless without the inverse mapping.
The output is intentionally plain: a fixed number of rows containing normalized x, y, width, height, confidence, and role. Saccade does not expose the model framework through its runtime ABI. Core ML and Windows ML can compile and schedule the graph in platform-specific ways, while the next stage sees the same target-row contract.
A fixed output is not the same as accepting every row. Empty rows carry zero confidence. A global threshold rejects weak candidates, and an optional size band can use a lower threshold for small controls. The accepted rows still need stable ordering and duplicate suppression before any of them becomes a desktop target.
Reduction belongs beside inference
It would be easy to read every candidate back and sort it on the CPU. That would also insert a device-to-host transfer at the widest point in the output. Saccade instead decodes candidates into a 16-byte internal record, builds an ordering key, performs stable radix passes, computes containment and intersection masks, and emits a bounded target packet on the GPU.
The ordering rule is deterministic: confidence descends first, then vertical position, horizontal position, and original candidate index. The scalar implementation uses the same rule as the Metal and D3D12 kernels. This gives platform code a byte-level oracle without putting a scalar sort in the production path.
The Save candidate now has source-local geometry. The inverse letterbox transform maps it back to the captured region, then the frame's source transform maps it into signed desktop coordinates. A candidate that falls entirely in letterbox padding is discarded. A candidate whose interior vanishes when its edges are eroded by one source pixel may remain visible, but it loses action capabilities. The model can propose a location; deterministic geometry decides whether that location is safe enough to use.
Accessibility can add identity, not time travel
Visual detection and accessibility answer different questions. Pixels say that a bounded region looks like a button. The native accessibility tree may say that the region is an enabled button named Save with an invoke action. Fused mode correlates those records, but only when their frame, window, transform, and topology context agree.
A fresh semantic target can add a role, text, capability bits, or a stronger safe point. A disabled or secure semantic target can remove action capabilities. What it cannot do is make stale visual geometry current. If an accessibility query belongs to another foreground window or another transform generation, fusion leaves it out.
This is why the scene is immutable after publication. The scene owner takes the newest compatible visual packet, the newest compatible semantic packet, and the current window catalog, then creates one bounded generation. Consumers can hold a handle to that generation without racing a writer that is filling the next one.
The label freezes one interpretation
When I activate Saccade, the current scene becomes the source of a hint map. Targets are ordered by the configured policy, then assigned prefix-free labels from the active alphabet. Prefix-free means no complete label is the prefix of another complete label, so accepting A can never steal the beginning of AF.
The visible labels freeze for the activation. A later model result may improve the next activation, but it cannot silently move A from Save to Cancel while I am typing. If the scope or source mode changes, Saccade ends the activation and creates a new frozen map. Stability is scoped honestly rather than simulated by mutating one live table.
The overlay packet is separate from the semantic scene. It contains surface-local fixed-point geometry, styles, and glyph indices, with no pointers. A compute kernel expands each target into four thin outline strips and one label quad. The display callback changes the active target independently, so typing the first symbol does not rebuild every static target.
Resolving A builds a plan, not an event
Pressing A reduces the frozen prefix state to target 17. The target record contributes its stable identifier, capabilities, safe point, window, and scene epochs. The active action contributes the requested button, modifiers, repeat count, final pointer policy, and deadline. The action planner combines them into a bounded input plan.
Nothing has reached CGEvent or SendInput yet. The planner checks whether target 17 still supports the requested action, whether its safe point is inside valid geometry, and whether the request shape is internally consistent. The execution preflight then compares current focus, permissions, physical-input state, transform, topology, and session generations with the values the plan captured.
This second validation is not duplicated model work. It is the point where a visual intention becomes authority to affect another application. The checks are deliberately cheap and deterministic because they sit between a human key press and native input.
Two clocks make the final check possible
The neural path targets a complete full-scope refresh at 30 Hz. Key reduction, pointer feedback, and overlay presentation target 120 Hz. These are independent clocks because waiting for a model pass before acknowledging a key would turn inference latency into interaction latency.
While frame 805 is running, the interaction owner can resolve A against the immutable scene from frame 804. That is safe only because the plan carries frame 804's context and execution checks the parts of the desktop that matter now. If nothing relevant changed, the older scene remains useful. If focus or topology changed, the plan expires. Freshness is a property of an action, not a rule that every subsystem must block until the newest frame exists.
The same separation handles overload. Capture can replace an unstarted pending frame with a newer one instead of building a queue of stale desktops. The active inference ticket retires normally, but a late result whose epochs no longer match cannot become the actionable scene. The interaction clock keeps running throughout.
Reconstructing the click
We can now walk backward from the native event. The input provider emitted one click because a preflight accepted a bounded plan. The plan existed because hint A resolved to target 17 in one frozen activation. Target 17 existed because compatible visual and accessibility records became one immutable scene. Its safe point survived because GPU reduction, letterbox inversion, source transforms, and deterministic geometry all retained the information needed to place it.
The original capture was never enough by itself. It needed ownership, time, transforms, and topology. The model output was never enough by itself. It needed ordering, suppression, capabilities, and a safe interior. The label was never permission to click. It was a compact reference into a generation that still had to match the live desktop.
That is the useful boundary for Saccade: computer vision decides where an action probably belongs, and a small deterministic path decides whether that proposal is still allowed to become input.