Teaching a smaller model to see actionable controls_

A desktop detector has an awkward job. It must find a tiny checkbox beside a paragraph, a borderless icon in a toolbar, a text field whose outline is nearly invisible, and a large button with obvious contrast. It has to do that across native applications, browsers, games, remote desktops, multiple display scales, and themes it did not see during training. Then it has to finish quickly enough that its answer still describes the desktop.

The tempting starting point is a large general-purpose vision model. It already knows a great deal about images, so why not run it directly? The answer is that a teacher can afford to deliberate over a training image. A desktop runtime cannot. The runtime needs a bounded output, predictable memory, a small graph that compiles well on two operating systems, and enough spatial detail to preserve controls that occupy only a few pixels in the model input.

Saccade therefore separates learning what matters from executing what was learned. Larger models, independent detectors, and human annotations produce private training evidence offline. One compact student learns from that evidence. Only the student and its deployment calibration belong in the application.

The teacher is not the product

A teacher model is useful precisely because it can spend resources the final application cannot. It may run at a larger input size, tile the desktop, use a broader backbone, or inspect an image with more than one prompt. None of those choices commits the runtime to the same architecture. They create additional observations about where controls might be.

Those observations are not ground truth. A confident teacher can mistake a decorative icon for a button. Two teachers can place different boxes around the same dropdown. A model trained heavily on browser screenshots can be poorly calibrated on a native settings panel. Treating every high score as a label would make the student inherit each teacher's confidence errors.

The training pipeline first converts teacher-specific outputs into a source-neutral form: box geometry, objectness, a small role vocabulary, and source identity. Scores are calibrated per source and per visual domain before they are compared. Human annotation remains authoritative whenever it exists.

When independent teachers overlap closely on the same control, their agreement can strengthen localization. When they disagree on role but agree on geometry, the box can still teach objectness while the role remains uncertain. When their boxes conflict materially, the region becomes an ignore area rather than a fabricated answer. Abstention is useful training data because it stops ambiguity from becoming false certainty.

This distinction also protects the action boundary. A detector predicts visual candidates; it does not grant permission to click. Accessibility data, window scope, freshness, occlusion policy, and action capabilities are fused later by deterministic runtime code. Distillation should make the detector better at seeing controls, not teach one neural confidence value to impersonate the complete safety policy.

What the student has to preserve

Ordinary object detection benchmarks reward correct boxes across photographs whose objects are often large relative to the image. Desktop controls have a different size distribution. Downscaling a full display can turn a radio button into a handful of samples. If the backbone's first useful detection map has a stride of eight or sixteen pixels, that control may disappear before the head gets a chance to classify it.

The student keeps a high-resolution P2 path with a stride of four. Lower-resolution stages still supply context: a small square near the word Cancel means something different from the same square in a chart legend. A lightweight top-down fusion combines that context with the fine geometry, then a dense class-agnostic head predicts bounded candidate rows.

Class-agnostic does not mean semantics are irrelevant. It means the hot detector's first question is whether a region behaves visually like an actionable control. A compact role signal can help ranking and fusion, but the model does not need a sprawling taxonomy of every widget framework. Platform accessibility providers are better sources for exact semantic roles when those roles exist.

The output contract matters as much as the backbone. The graph emits a fixed-capacity tensor of normalized boxes, objectness, and compact role values. It does not allocate a variable tree or expose framework objects. GPU postprocessing can decode, order, suppress, and compact those rows before the CPU sees them. That keeps dense intermediate traffic on the device and gives both platform runtimes the same narrow boundary.

Distillation is several losses, not one copy

The student trains against hard human targets and soft teacher signals at the same time. Hard box and objectness losses anchor it to facts. Soft objectness teaches relative uncertainty around near misses. Soft box targets can reveal a teacher's smoother localization around edges that are hard to annotate consistently. Each term has its own weight and admission rule.

Feature matching is optional, and it is easy to misuse. Two layers can have the same width and height while representing different things. Forcing a compact convolutional feature map to numerically match a different teacher backbone can reduce accuracy even when the tensor shapes line up. Representation alignment needs either an explicit adapter with evidence that the spaces correspond, or it should be omitted. Matching output behavior is often the cleaner contract.

Training alternates visual domains rather than allowing the largest source to dominate each batch. Hard negatives matter disproportionately: disabled controls, decorative glyphs, text that looks boxed, game HUD elements, and repeated icons whose actionable state changes with context. Augmentation must preserve the properties being learned. Scaling, theme shifts, blur, compression, and display resampling are useful; transforms that destroy thin borders and then call the result truth are not.

Every candidate checkpoint is evaluated on frozen suites split by application family, theme, scale, and control size. Recall is measured at the safe point, not only at a generous intersection threshold. False positives matter because every one becomes overlay noise and an extra candidate for an agent. Latency, peak framework memory, output stability, and calibration drift are release dimensions alongside recall.

Quantization is another model change

An accurate training checkpoint is not automatically a deployable checkpoint. Conversion can rewrite operators. Half precision can alter ordering around a confidence threshold. Integer quantization can preserve average accuracy while erasing the smallest low-contrast controls. Platform compilers can choose different kernels for the same graph shape.

For that reason, export and quantization sit inside qualification rather than after it. The exact artifact that runs through Core ML or Windows ML must pass the frozen suites. Its decoded rows are compared with the training graph, and any platform-specific calibration is versioned with the weights. A graph is admitted only when its accuracy, capacity, latency, and memory envelope all fit the runtime contract.

Saccade owns the image preparation and candidate reduction around that graph. Metal and D3D12 kernels can crop, resample, normalize, decode, rank, and suppress without round-tripping full tensors through the CPU. Core ML and Windows ML remain graph executors. This division lets platform compilers improve without turning the whole vision path into an opaque framework pipeline.

The private boundary is deliberate. Desktop samples can contain personal information. The dataset, teacher traces, weights, calibration tables, and rejected experiments stay outside the public source tree. The public architecture can still be precise about tensor contracts, ownership, fixed capacities, validation stages, and the conditions under which a model is accepted.

The goal is not to make a small model imitate every thought of a large one. It is to identify the part of the teacher's behavior that the runtime actually needs, preserve the spatial detail that desktop controls demand, and surround the learned proposal with deterministic evidence and action policy. The teacher helps build the instrument. The student is the instrument that has to keep time.