A video stream through Vidarax_

A 30 fps stream produces 108,000 frames in an hour. In a deployment dashboard or a fixed camera feed, long runs of adjacent frames contain only small or irrelevant changes. The filter tries to reject those frames before the model path.

Vidarax turns continuous video into an ordered event stream and keeps source frames for events that need them. An event can say that a deployment dashboard appeared at 12.4 seconds or an error dialog appeared at 18.1 seconds, with the retained frame showing what the engine saw.

Recorded files and live streams

Recorded sources use a two-phase decode path. Vidarax first extracts cheap frame signals, then JPEG-encodes only selected frames. A live session keeps its decoder and ordered analysis state alive across packets.

The API has two entry points. analyze runs deterministic frame analysis without a prompt. reason adds a semantic prompt, optional JSON schema, crops, temporal context, tiered models, and clip or keyframe modes. A live prompt change is tagged with the current pipeline generation and takes effect only after the vision-language model (VLM) worker acknowledges it. Callers can query sequence-numbered events through REST, follow them over cursor-based server-sent events (SSE), or register signed webhooks through the REST API and TypeScript SDK.

Start with one frame

Take an illustrative frame 4,218 from a live application stream. The network and codec layers have already done their work, so the analysis stage receives planar image data plus a frame index and presentation timestamp. From the luma plane it computes a compact set of signals: a perceptual hash, average brightness, flicker, ghosting, and noise-variance scores.

Those signals are small. That matters because the first decision runs for every frame, including the thousands that will be discarded. The filter compares the candidate with the last committed keyframe and checks seven conditions in priority order:

• the first frame establishes the baseline
• a large perceptual-hash distance indicates a scene cut
• a periodic keepalive prevents a quiet scene from disappearing forever
• an exposure shift, flicker, ghosting, or a noise spike marks a suspected visual artifact.

If nothing fires, the frame is skipped. If the frame is a keyframe, it becomes the next comparison point. Artifact decisions do not silently move that baseline, because a broken frame should not redefine normal.

For the worked trace, suppose frame 4,218 differs from the last kept hash in 26 of 64 bits. The scene-cut threshold is 18, so bit one wins and the filter returns KeepKeyframe with reason SceneCut and confidence 26 / 64. The inputs are illustrative, while the decision rule and output fields are the real ones.

The frame decision fits in one byte

The readable reference is a chain of early returns. The implementation evaluates every predicate, packs the results into one byte, and asks for the lowest set bit:

let mask: u8 = (!initialized as u8)
    | ((hash_distance >= scene_cut_threshold) as u8) << 1
    | ((frames_since_keep >= keepalive_frames) as u8) << 2
    | ((luma_shift >= luma_threshold) as u8) << 3
    | ((flicker >= flicker_threshold) as u8) << 4
    | ((ghosting >= ghosting_threshold) as u8) << 5
    | ((noise >= noise_threshold) as u8) << 6
    | 0x80;

let decision = mask.trailing_zeros() as usize;

Bit zero has the highest priority. Bit seven is a permanent sentinel for "no trigger," so the lookup index is always valid. Static tables map that index to the event type and reason code. Replacing a string pointer with an enum reduced the event from 48 bytes to 32 and made the possible outcomes exhaustive.

I kept the early-return implementation as a reference and compare both versions across every trigger and conflicting-trigger case. The latency result later in the article covers the bit-mask implementation. I did not retain a comparison showing that it is faster than the reference.

Reusing decoded-frame buffers

The filter is cheap only after a frame has been decoded, and decoded 1080p planes are not small. Reallocating them at frame rate would move the cost upstream and add allocator jitter to the whole stream.

Vidarax wraps those byte vectors in a uniquely owned value. Dropping it returns the backing allocation to a bounded free list. Pool capacity is fixed. When the pool is empty, Vidarax falls back to heap allocation, whose latency and failure behavior are not bounded by the pool size.

Pool sizes come from counting every place a frame can wait. For decoded output that means one frame under construction, the reader channel, the decoder's pending allowance, and one frame held by the consumer. The JPEG pool is sized the same way across analysis, VLM work, clip accumulation, and storage.

Restarting a live stream

Decode, filtering, semantic novelty, and temporal inference share ordered state. Vidarax gives that state one generation number and one supervisor.

The first unexpected exit faults the generation, closes the WebRTC peer, raises one monotonic stop signal, and gives every sibling a bounded interval to finish. A lone decoder or temporal worker is never restarted underneath older stream state. The dashboard distinguishes healthy, faulted, saturated, and idle processes. It also separates clean shutdowns from join-deadline overruns. Recovery starts a new session generation.

That model also makes admission concrete. Before run_created becomes durable, the process reserves the negotiated generation's bounded RTP queue bytes, decoded and JPEG pools, scratch buffers, ffmpeg allowance, and fixed worker count. If the process budget cannot fit the generation, creation returns 503 without leaving a durable run behind.

H.264 and H.265 run in an ffmpeg child, so a native decoder crash cannot abort the API process. Rust cannot safely kill a wedged OS thread. Vidarax reports a join-deadline overrun and closes the peer, but it does not yet have a watchdog that terminates an unresponsive child.

Inference is a second filter

Selected frames can go to local vLLM, SGLang, or MLX. Gemini is available as an optional remote provider. A deployment can also configure a different second-pass model.

The first model emits a schema-reported score named confidence. It is an uncalibrated model score, not a measured probability. If the score falls below the configured threshold, Vidarax can run the same selected JPEG through the second model. A failed second call leaves the usable first answer intact and records the failure against the provider that failed.

For frame 4,218, the first model returns 0.62 against a routing threshold of 0.70. The second returns 0.91, so its answer wins.

Skipping repeated model work

Live capture can place an embedding-only novelty filter between the deterministic filter and the VLM. Setting a TCP sidecar address enables it. The worker sends the selected JPEG over a persistent binary connection and receives a 768-value SigLIP2 embedding. It compares that embedding with the last frame whose VLM call produced a non-empty description.

A small distance may reuse the existing description and avoid a model call. Capture time limits how long that description can survive. Cumulative embedding drift catches a scene that changes through many small steps. A default one-percent shadow sample still sends some reuse decisions through the VLM without changing state or emitting events. Its change ratio is a calibration signal, not a label.

Ignoring shadow samples and failure refreshes, the idealized model cost is E + (1 - d)V. Here E is one embedding, V is one VLM call, and d is the reuse fraction. Vidarax records embedding latency, reuse, forced refresh, sidecar failure, and shadow outcomes. Actual savings depend on the stream and provider.

Writing the event log

Once inference finishes, a synchronous EventSink separates the ordered media workers from storage. The default self-hosted path appends events to a write-ahead log. The API reads that timeline for GET /v1/runs/{id}/events. Worker threads do not need an async runtime, and the HTTP layer does not enter the frame loop.

The stored record contains metadata and the semantic description. JPEG bytes are never placed in the plain-text log or transformed to base64 for persistence. The sink writes each selected JPEG to a content-addressed binary sidecar before appending its keyframe_stored event. The event carries the relative reference, media type, byte count, SHA-256, and vidarax.image.v1 coordinate provenance.

A crash between blob creation and event append can leave an unreferenced blob. The server does not reclaim it automatically yet. That needs a startup scan or retention-based garbage collector.

At that point, the worked frame has changed representation four times: decoded planes, compact signals, a JPEG inference request, then a durable event. The last form can look like this:

{
  "frame_index": 4218,
  "pts_ms": 140600,
  "kind": "vlm_tiered",
  "coordinate_schema": "vidarax.image.v1",
  "coordinates": {
    "source_extent": {"width": 1920, "height": 1080},
    "requested_region": {"x": 0.0, "y": 0.0, "width": 1.0, "height": 1.0},
    "resolved_region": {"x": 0, "y": 0, "width": 1920, "height": 1080},
    "analysis_extent": {"width": 1920, "height": 1080}
  },
  "confidence": 0.91,
  "description": "Deployment dashboard with one failed service"
}

The worked record uses illustrative numbers and a fictional description. Its field mapping and event kind come from the real schema.

The coordinate record says which source pixels were selected for analysis. A transport resize may change the JPEG dimensions while preserving that region, so vidarax.image.v1 records the source crop. It never substitutes the resized output. Camera intrinsics, lens distortion, depth, handedness, and robot or world transforms must be supplied separately. A VLA system needs them to map a visual result into controller coordinates.

Durable events can be queried, resumed over cursor-based SSE, or delivered through signed webhooks. Stable IDs support deduplication, and slow subscribers catch up from the write-ahead log.

If an application needs to turn motion, novelty, or model confidence into a capture or notification, it can attach a bounded trigger program to the live generation. The program can be compiled and replayed before attachment. The trigger API documents the current inputs and actions.

What the frame filter costs

I measured the warmed per-frame filter over 120,000 frames for latency and 60,000 frames for allocation accounting, with Rust's selected global allocator wrapped by counters.

These numbers cover the deterministic decision after decode, without JPEG encoding, network service, or VLM inference. On July 17, 2026, the warmed filter completed at 42 ns p95 with zero observed allocations on an Apple M5 Max running macOS 26.4, arm64, with the Rust release profile.