Teaching a smaller model to see actionable controls_

A desktop detector has an awkward job. It must find a tiny checkbox beside a paragraph, a borderless icon in a toolbar, a text field whose outline is nearly invisible, and a large button with obvious contrast. It has to do that across native applications, browsers, games, remote desktops, multiple display scales, and themes it did not see during training. Then it has to finish quickly enough that its answer still describes the desktop.

A large general-purpose vision model can find obvious controls, but its latency, memory use, and variable output do not fit a 33.33 ms local scene deadline. I started from an existing compact UI detector instead. Even that model found only 7.98% of web controls whose native short side was below 12 pixels. Large controls remained stable while tiny checkboxes, toolbar icons, and borderless buttons disappeared in the downsampled feature maps.

Saccade uses larger offline teachers and human review. A compact student learns from that material and is the only model that ships in the application. The public runtime contract exposes the student output, not the training material.

The small checkbox beside a paragraph is easy for a large teacher to see at high resolution and easy for a compact student to lose after the image has been reduced.

How teacher outputs become training data

A teacher model is useful precisely because it can spend resources the final application cannot. It may run at a larger input size, tile the desktop, use a broader backbone, or inspect an image with more than one prompt. None of those choices commits the runtime to the same architecture. They create additional observations about where controls might be.

Those observations are not ground truth. A confident teacher can mistake a decorative icon for a button. Two teachers can place different boxes around the same dropdown. A model trained heavily on browser screenshots can be poorly calibrated on a native settings panel. Treating every high score as a label would make the student inherit each teacher's confidence errors.

The training pipeline converts each teacher's output into the same record: box geometry, objectness, a small role vocabulary, and the source that produced it. Scores are calibrated for each source and visual domain before comparison. Human annotations take priority when they exist.

Suppose two teachers place nearly the same box around that checkbox. Their agreement can strengthen its localization. If they agree on the box but disagree about its role, the region can still teach objectness while the role remains unset. Materially different boxes turn the checkbox region into an ignore area, so the student is not penalized for refusing an answer the teachers could not settle.

The detector proposes a control and a confidence score. The executor still checks the current scene, transform, topology, focus, permissions, physical buttons, and target capability before it can click. Those checks are deterministic runtime code, not another learned confidence threshold. The complete action preflight shows them in order.

A stride-four path for small controls

Ordinary object detection benchmarks reward correct boxes across photographs whose objects are often large relative to the image. Desktop controls have a different size distribution. In the starting checkpoint, web recall below 12 pixels was 7.98% while recall at 12 to 24 pixels was 63.88%. The failure was concentrated at the scale where a stride-eight map had already collapsed most of the control into one feature cell. That observation led to the P2 path.

The checkbox is exactly the kind of control that disappears there. The student keeps a high-resolution P2 path with a stride of four, while lower-resolution stages still supply the context that distinguishes a checkbox beside a setting from a similar square inside a chart. A lightweight top-down fusion combines that context with the fine geometry, then a dense class-agnostic head predicts bounded candidate rows.

Class-agnostic does not mean semantics are irrelevant. It means the hot detector's first question is whether a region behaves visually like an actionable control. A compact role signal can help ranking and fusion, but the model does not need a sprawling taxonomy of every widget framework. Platform accessibility providers are better sources for exact semantic roles when those roles exist.

The student only needs the six values Saccade consumes for each candidate: normalized x, y, width, height, confidence, and compact role. The graph emits at most 1,024 rows and runtime calibration publishes at most 160 targets. The decoder accepts the same FP16 or FP32 row layout on both platforms, while GPU postprocessing orders, suppresses, and compacts the rows before the CPU sees them. The six-value decoder is public source.

Training with several losses

The student trains against hard human targets and soft teacher signals at the same time. For the checkbox, a human box anchors its location and objectness. Soft teacher outputs describe near misses and the uncertainty around its thin edges. Each term has its own weight and admission rule.

Feature matching stays optional because two layers can have the same dimensions while representing different things. Forcing a compact convolutional map to match a different teacher backbone can reduce accuracy even when the tensors line up. I use an adapter only after showing that the two representations correspond. Otherwise I match model outputs.

Training alternates visual domains so the largest source does not dominate every batch. Hard negatives matter disproportionately: disabled controls, decorative glyphs, boxed-looking text, game HUD elements, and repeated icons whose behavior changes with context. Scaling, theme shifts, blur, compression, and display resampling are useful augmentations. A transform that destroys a thin border makes the label unreliable, so I leave it out.

I evaluate candidate checkpoints separately by application family, theme, scale, and control size. Recall is measured at the safe point, not only at a generous intersection threshold. False positives matter because every one becomes overlay noise and an extra candidate for an agent. I also track latency, peak framework memory, output stability, and calibration drift because the student has to fit the application, not only the training set.

Testing the exported model

An accurate training checkpoint is not automatically a deployable checkpoint. Conversion can rewrite operators. Half precision can alter ordering around a confidence threshold. Integer quantization can preserve average accuracy while erasing the smallest low-contrast controls. Platform compilers can choose different kernels for the same graph shape.

I therefore test the exact Core ML or Windows ML artifact on its shipping backend. I compare its decoded rows with the training graph, then repeat the control-size, application-family, and calibration slices that exposed the original tiny-target failures. If the exported model loses the checkbox class of controls, a better average does not rescue it. Platform calibration travels with the artifact, and I keep it only when accuracy, capacity, latency, and memory still fit the runtime.

Saccade handles image preparation and candidate reduction around the graph. Metal and D3D12 kernels crop, resample, normalize, decode, rank, and suppress without round-tripping full tensors through the CPU. Core ML and Windows ML execute the graph. Platform compilers can improve while the surrounding work stays visible and measurable.

The shipping artifact declares its input dimensions, precision, target limit, output size, provider compatibility, signature, and payload. Those fields make the runtime boundary inspectable without publishing training material. The public artifact definition lists them.

These numbers describe the Pixel model. Accessibility fusion and completed actions are measured separately.

The exact FP16 export reaches 61.77% Pixel candidate coverage on the frozen web suite and 84.54% on unseen desktop applications. Its safe-center rate is 100% on web and 99.97% on desktop. For controls below 12 pixels, which include small checkboxes like this one, web recall rose from 7.98% in the starting checkpoint to 23.93% after export.