Pressing a hint key looks like the last step in a keyboard-driven pointer tool. The label says A, I press A, and the pointer clicks the control underneath it. But the label came from an older view of the desktop. The application may have moved, a dialog may have appeared, the display scale may have changed, or my hand may have touched the mouse after the label was drawn.
Saccade treats that gap as the main problem. Finding a clickable-looking rectangle is only the start. Its identity and coordinates have to survive capture, local inference, the frozen hint map, and one final check before native input. The local target protocol carries geometry, identity, capabilities, a safe point, source, and the generations needed for that check.
Take the Save button in the foreground window, assigned the hint A. Its identity and coordinates must still be valid when the click is finally posted.
The macOS and Windows applications connect native capture, local model execution, scene publication, overlay rendering, and input. In the request trace, target 17 is the Save button at every stage.
frame 804
same device
local model
compact targets
safe point
native input
The capture is a lease, not an image copy
ScreenCaptureKit on macOS and Windows Graphics Capture both deliver native surfaces. Saccade wraps each surface in a bounded frame lease with its identifier, readiness fence, transform, dimensions, format, storage kind, and current owner. The lease records who may use the native object and when it can be retired. The desktop never becomes a long-lived CPU bitmap. The public frame lease layout lists those fields.
That matters before we discuss inference. A 4K BGRA surface contains 33,177,600 bytes. Copying it through general-purpose memory simply to hand it to another device would make bandwidth and lifetime harder at the same time. The native paths instead import or copy within the platform graphics stack, then keep the resulting dependency with the frame. The inference worker cannot consume the surface until that dependency is satisfied, and capture cannot reclaim it until the worker retires the lease.
Frame 804 also carries a topology epoch. Suppose the Save button is on a display whose scale changes while the model is running. The pixels still describe a real button, but the old source-to-desktop transform no longer describes its destination. Keeping the topology and transform beside the frame lets the later scene owner reject the result before an old coordinate reaches a new desktop.
The model sees a bounded tensor
The model does not receive the full capture object as an abstract image. An owned Metal or D3D12 preprocessing kernel maps the selected scope into the model's fixed input shape. It applies the aspect-preserving fit, letterbox placement, channel mapping, and normalization in one device dispatch. The placement parameters are retained because model coordinates are meaningless without the inverse mapping.
The output is a fixed number of rows containing normalized x, y, width, height, confidence, and role. Core ML and Windows ML can compile and schedule the graph differently, while the next stage always receives the same target rows.
A fixed output is not the same as accepting every row. Empty rows carry zero confidence. A global threshold rejects weak candidates, and an optional size band can use a lower threshold for small controls. The accepted rows still need stable ordering and duplicate suppression before any of them becomes a desktop target.
Reduction belongs beside inference
It would be easy to read every candidate back and sort it on the CPU. That would also insert a device-to-host transfer at the widest point in the output. Saccade instead decodes candidates into a 16-byte internal record, builds an ordering key, performs stable radix passes, computes containment and intersection masks, and emits a bounded target packet on the GPU.
The ordering rule is deterministic: confidence descends first, then vertical position, horizontal position, and original candidate index. The tests compare the GPU packet with the scalar implementation byte for byte at candidate counts 0, 1, 256, 257, and 777 on both Metal and D3D12. The scalar sort is not used in the production GPU path.
The Save candidate now has source-local geometry. The inverse letterbox transform maps it back to the captured region, then the frame's transform maps it into signed desktop coordinates. A candidate entirely inside the padding is discarded. If eroding its edges by one source pixel removes its interior, it may remain visible but cannot be clicked. The model proposes a location. Deterministic geometry decides whether Saccade can use it safely.
Accessibility adds identity to the captured scene
Visual detection and accessibility answer different questions. Pixels say that a bounded region looks like a button. The native accessibility tree may say that the region is an enabled button named Save with an invoke action. Fused mode correlates those records, but only when their frame, window, transform, and topology context agree.
A fresh semantic target can add a role, text, capability bits, or a stronger safe point. A disabled or secure semantic target can remove action capabilities. The compact hybrid merge keeps visual geometry and adds semantic fields only when the frame, window, transform, and topology generations agree. If an accessibility query belongs to another foreground window or another transform generation, fusion leaves it out.
This is why the scene is immutable after publication. The scene owner takes the newest compatible visual packet, the newest compatible semantic packet, and the current window catalog, then creates one bounded generation. Consumers can hold a handle to that generation without racing a writer that is filling the next one.
| mismatch | detected at | result before input |
|---|---|---|
| deadline elapsed | execution preflight | SACCADE_ERROR_TIMEOUT |
| input unavailable, secure surface, or secure point | execution preflight | SACCADE_ERROR_PERMISSION |
| frame, model, scene, session, transform, topology, source, focus, window, permission, or initial buttons changed | execution preflight | SACCADE_ERROR_STALE_HANDLE |
| target missing, disabled, occluded, secure, or no longer actionable | target lookup | SACCADE_ERROR_NOT_FOUND |
The label freezes one interpretation
When I activate Saccade, the current scene becomes the source of a hint map. Targets are ordered by the configured policy, then assigned prefix-free labels from the active alphabet. Prefix-free means no complete label is the prefix of another complete label, so accepting A can never steal the beginning of AF.
Labels stay fixed until the activation ends.
A later model result may improve the next activation, but it cannot silently move A from Save to Cancel while I am typing. A scope or source-mode change ends the current activation and publishes a new table.
The overlay packet is separate from the semantic scene. It contains surface-local fixed-point geometry, styles, and glyph indices, with no pointers. A compute kernel expands each target into four thin outline strips and one label quad. The display callback changes the active target independently, so typing the first symbol does not rebuild every static target.
Resolving A builds a plan, not an event
Pressing A reduces the frozen prefix state to target 17. The target record contributes its stable identifier, capabilities, safe point, window, and scene epochs. The active action contributes the requested button, modifiers, repeat count, final pointer policy, and deadline. The action planner combines them into a bounded input plan.
Nothing has reached CGEvent or SendInput yet. The planner checks whether target 17 still supports the requested action, whether its safe point is inside valid geometry, and whether the request shape is internally consistent. The execution preflight then compares current focus, permissions, physical-input state, transform, topology, and session generations with the values the plan captured.
The executor does not run the model again. It checks the captured plan against the current state. These are bounded integer and bit checks against the values captured with the selected target.
This is the last check before Saccade calls CGEventPost on macOS or SendInput on Windows. The macOS call submits one prepared event at the HID event tap, while the Windows call submits a bounded array of mouse or keyboard records. Both use the same preflight.
Resolve v0 compares a planner coordinate with the current candidate table. It can select a candidate, preserve the original coordinate, or abstain. Optional local visual text carries enough frame identity for the host to verify the same unchanged screenshot, application, and scope. Any mismatch abstains. A selected target then enters the public action planner and execution preflight.
Two clocks make the final check possible
The neural path targets a complete full-scope refresh at 30 Hz. Key reduction, pointer feedback, and overlay presentation target 120 Hz. These are independent clocks because waiting for a model pass before acknowledging a key would turn inference latency into interaction latency.
While frame 805 is running, the interaction owner can resolve A against the immutable scene from frame 804. An older scene remains usable only while the action's scene, transform, topology, focus, permission, and capability state still match. Otherwise the executor rejects it.
Capture can replace an unstarted pending frame with a newer one, leaving no queue of stale desktops. The active inference ticket retires normally, but a late result whose epochs no longer match cannot become the actionable scene. The interaction clock keeps running throughout.
The final click
By the time the native event is posted, A still means the Save button selected at activation. Its visual and accessibility records came from the same scene. Its coordinates survived GPU reduction and the trip back from model space to the desktop. The final preflight confirms that the desktop still agrees.
If any relevant state changed, Saccade cancels the click and returns to selection. Otherwise it posts the event without running vision again.