A video stream through Vidarax_

Vidarax is a self-hosted video intelligence engine. It accepts a file or a live stream, decides which moments deserve closer inspection, runs vision-language inference on those moments, and emits timestamped events that another program can query.

The useful output is not another video. It is a sequence of records such as: at 12.4 seconds the screen changed to a deployment dashboard; at 18.1 seconds an error dialog appeared; at 26.7 seconds the presenter returned to the editor. Each record carries the stream position, event kind, confidence, and description. The selected keyframe remains evidence. The event is the index over that evidence.

I built Vidarax because treating every decoded frame as an inference request is the wrong unit of work. A 30 fps stream produces 108,000 frames in an hour. Most adjacent frames say nothing new. The engine therefore spends most of its design effort deciding not to ask a model a question.

That is the product promise: downstream software gets an ordered event stream instead of inheriting a media stack. The decoder, selection policy, write-ahead log, and keyframe store run on infrastructure you control; inference goes only to the backend you configure.

One engine, several ways in

The same pipeline accepts local files, owned uploads, bounded remote downloads, opt-in remote HLS, RTSPS cameras (or explicitly allowed RTSP), and live WebRTC through WHIP. Recorded sources use a two-phase decode path: first extract the cheap frame signals, then JPEG-encode only the selected frames. A live session keeps its decoder and ordered analysis state alive across packets.

The API exposes two different questions instead of pretending they are one. analyze runs deterministic frame analysis and takes no prompt. reason adds a semantic prompt, optional JSON schema, crops, temporal context, tiered models, and clip or keyframe modes. Live prompts can change while a WHIP session is running. On the other side, callers read sequence-numbered events, markers, interactions, keyframes, cross-run queries, and description search through REST or the TypeScript SDK.

That breadth only stays understandable if each layer answers a narrower question than the next one:

Start with one frame

Take an illustrative frame 4,218 from a live application stream. The network and codec layers have already done their work, so the analysis stage receives planar image data plus a frame index and presentation timestamp. From the luma plane it computes a compact set of signals: a perceptual hash, average brightness, flicker, ghosting, and noise-variance scores.

Those signals are small. That matters because the first decision runs for every frame, including the thousands that will be discarded. The gate compares the candidate with the last committed keyframe and checks seven conditions in priority order:

• the first frame establishes the baseline;
• a large perceptual-hash distance indicates a scene cut;
• a periodic keepalive prevents a quiet scene from disappearing forever;
• an exposure shift, flicker, ghosting, or a noise spike marks a suspected visual artifact.

If nothing fires, the frame is skipped. If the frame is a keyframe, it becomes the next comparison point. Artifact decisions do not silently move that baseline, because a broken frame should not redefine normal.

For the worked trace, suppose frame 4,218 differs from the last kept hash in 26 of 64 bits. The scene-cut threshold is 18, so bit one wins and the gate returns KeepKeyframe with reason SceneCut and confidence 26 / 64. Those values are illustrative inputs passed through the real decision rule, not a captured Vidarax run.

The hot decision stays boring

The readable form of the gate was a chain of early returns. It was easy to audit, but the compiler produced a branchy path whose shape depended on the input. The current path evaluates each predicate, packs the results into one byte, and asks for the lowest set bit:

let mask: u8 = (!initialized as u8)
    | ((hash_distance >= scene_cut_threshold) as u8) << 1
    | ((frames_since_keep >= keepalive_frames) as u8) << 2
    | ((luma_shift >= luma_threshold) as u8) << 3
    | ((flicker >= flicker_threshold) as u8) << 4
    | ((ghosting >= ghosting_threshold) as u8) << 5
    | ((noise >= noise_threshold) as u8) << 6
    | 0x80;

let decision = mask.trailing_zeros() as usize;

Bit zero has the highest priority. Bit seven is a permanent sentinel for "no trigger," so the lookup index is always valid. Static tables map that index to the event type and reason code. The reason is an enum rather than a string pointer, which reduced the event from 48 bytes to 32 and made the possible outcomes exhaustive.

I kept the original early-return implementation under test. A parity suite drives every trigger and conflicting-trigger case through both versions. That reference matters more than the clever byte. Without it, a faster dispatch could quietly change which event wins.

Ownership is the recycling protocol

The gate is cheap only after a frame has been decoded, and decoded 1080p planes are not small. Reallocating them at frame rate would move the cost upstream and add allocator jitter to the whole stream.

Vidarax wraps those byte vectors in an owning value. When the last consumer drops it, its backing allocation returns to a bounded free list. A consumer does not call a special recycle function and does not know the pool exists. If the free list is empty, the producer allocates a buffer. If it is full, the returning buffer is dropped. Both failure modes are bounded and explicit.

The capacities come from the topology rather than a round number. The decode-output pool counts every place a frame can be in flight: one under construction, the bounded reader channel, the decoder's pending allowance, and one held by the consumer. The JPEG pool uses the same accounting over analysis, VLM, clip accumulation, and sink positions.

Inference is a second filter

A kept frame is JPEG-encoded and sent to an inference provider. The provider boundary supports local open-source serving through vLLM, SGLang, or MLX, with Gemini available as an optional remote provider. A deployment can also configure a different second-pass model.

The first model returns structured text including a confidence. If that confidence falls below the configured threshold, Vidarax can run the same evidence through the second model with a stricter schema. If the second call fails, the engine returns the first result and records the second-pass error against the provider that actually failed. It does not erase a usable first answer, and it does not make the failed spend disappear from metrics.

So frame 4,218 arrives here as a JPEG plus its timestamp and the gate's SceneCut reason. In the worked trace, the first model describes a deployment dashboard but returns 0.62 confidence against a 0.70 threshold. That sends the same evidence to the configured second model. A 0.91 second answer becomes the semantic result; a failed second call would leave the 0.62 first answer intact.

Semantic reuse has to fail open

Live capture can now place an embedding-only novelty gate between the deterministic gate and the VLM. It is opt-in: setting a TCP sidecar address enables it. The worker sends the selected JPEG over a persistent binary connection and receives a 768-value SigLIP2 embedding. That embedding is compared with the last frame whose VLM call produced a non-empty description.

A small distance may reuse the existing semantic state and avoid a model call, but two bounds can override that decision. Capture time limits how long one description can survive, and cumulative embedding drift catches a scene that moves through many individually small changes. A default one-percent shadow sample still sends some reuse decisions through the VLM without changing state or emitting events. The resulting change ratio is a warning signal for calibration, not a label.

The cost condition is simple. If one VLM call costs V, one embedding costs E, and a calibrated gate reuses a fraction d, the new expected cost is E + (1 - d)V. The gate helps only when E < dV. Vidarax records embedding latency, reuse, forced refresh, sidecar failure, and shadow outcomes. Those signals make the decision measurable; they do not justify one universal savings number.

The event is the durable boundary

Once inference finishes, a synchronous EventSink separates the ordered media workers from storage. The default self-hosted path appends events to a write-ahead log. The API reads that timeline for GET /v1/runs/{id}/events. Worker threads do not need an async runtime, and the HTTP layer does not enter the frame loop.

The stored record contains metadata and the semantic description. JPEG bytes are deliberately not placed in the plain-text log. Instead, the sink writes each selected JPEG to a content-addressed blob before appending its keyframe_stored event. The event carries the relative reference, media type, byte count, and SHA-256. Identical JPEGs share a blob, and a caller may fetch one only through a run that references it.

At that point, the worked frame has changed representation four times: decoded planes, compact signals, a JPEG inference request, then a durable event. The last form can look like this:

{
  "frame_index": 4218,
  "pts_ms": 140600,
  "kind": "vlm_tiered",
  "confidence": 0.91,
  "description": "Deployment dashboard with one failed service"
}

The numbers and description in this record complete the illustrative trace. The field mapping and event kind are real; this particular event is not presented as observed output.

This boundary is also where Vidarax becomes useful to other software. A caller can poll events, build a searchable index, update a UI, or trigger an automation without knowing anything about codecs, frame pools, confidence routing, or model servers.

What the measurements actually prove

The per-frame gate has a small standalone probe. It wraps the system allocator with counters, runs 120,000 frames for latency percentiles, then 60,000 frames for allocation accounting. The release script consumes the JSON and fails when the contract moves outside its envelope.

Those latency numbers were reproduced on my machine on July 15, 2026. They describe the deterministic gate, not decoding, JPEG encoding, network service, or VLM inference. The ceiling is intentionally loose enough for normal CI variance and tight enough to catch a categorical regression such as adding a string allocation to the frame loop.

Semantic novelty has a different evidence shape. Its live path, fail-open behavior, drift and time bounds, shadow sampling, and calibration metrics are implemented. The remaining claim is deployment-specific: measure embedding p50 and p95, reuse ratio, shadow-change ratio, provider failures, and end-to-end VLM cost on the actual stream mix. Activation is not calibration.

I do not publish one headline throughput number for the complete engine. End-to-end capacity changes with codec, resolution, hardware, model, provider batching, and the distribution of scenes. The repository exposes the counters and focused probes needed to measure those workloads, but each provider and hardware pair still needs its own result. Collapsing them into one number would make the claim easier to repeat and harder to trust.

So what is Vidarax?

For the caller, Vidarax is a service that turns video into structured, timestamped events. It can answer cheap visual questions without a prompt, semantic questions with a configured model, and retrieval questions after the result is durable. Inside, it is a sequence of narrowing decisions. Decode preserves order. The frame gate rejects repetition cheaply. Optional embeddings suppress semantically stable survivors within explicit safety bounds. Tiered inference spends again only when a first answer is uncertain. The WAL and keyframe blobs make the result queryable without pulling storage concerns back into the media workers.

The practical boundary is simple: Vidarax owns decoding, selectivity, inference routing, and durable evidence. Your application starts where video becomes data.