The work described here is from February through July 2026, on vidarax, a real-time video intelligence engine I have been building in Rust on my own time. The repo is closed source for now, available on request. The latency and allocation numbers quoted as current were re-measured in early July with the project's own probe, on my machine.
vidarax ships with a release script that runs the engine's per-frame code tens of thousands of times and fails the build if it sees a single heap allocation. The budget is written as 0.0 allocations per frame, and it is an average [confirm: correct?], so one allocation anywhere in a 60,000 frame run is enough to turn the release red. Next to that check sits a latency ceiling: p95 on the same path must stay at or under 50 microseconds. Two more checks cap the binaries: 25 MB for the CLI, 45 MB for the API server. None of these numbers is in the README as an achievement; each one is a failure condition in the release script.
The reasoning behind the arrangement is ordinary. A performance number in a README is measured once, by hand, on whatever machine was handy, and nothing ever measures it again. The code keeps moving, a dependency grows or someone adds a log line to a loop, and readers keep believing a number that no longer matches the code. A number wired into the release script is re-measured every time anyone tries to ship. So it is either still true or the build is red.
One naming collision needs defusing first. The engine's core component is a per-frame filter called the gate, and release checks are also conventionally called gates. Both appear below, so: the frame gate is the filter, the gate path is the code it runs on every frame, and the release gates are the checks.
What runs on every frame
vidarax is a video intelligence engine: a stream goes in, live over WebRTC or RTSP or from an uploaded file, and structured semantic events come out the other side, scene cuts, exposure shifts, flicker, loop detection, model-written descriptions of what changed. Vision language models are the expensive part, in latency and in money, so the engine filters first and infers second: a deterministic frame gate looks at every single frame and decides whether anything changed enough to be worth model attention, and only the frames that pass get sampled into inference. Inference cost then tracks the sampled frames, not the frame rate.
That split puts the frame gate in a particular position: it is the only code that runs at full frame rate no matter what, and the engine does not choose the frame rate; the source does. A 30 fps security camera and a 240 fps game capture would both push every frame through the same function. The decision itself is built to be constant work: seven trigger conditions each evaluate to one bit in a byte, packed in priority order with a permanent sentinel bit so the byte is never zero, and counting trailing zeros selects the winning condition, which then indexes static tables for the event type, the reason, the confidence. No branch that grows with content, no lookup that grows with history, and nothing that touches the heap.
So the thing under budget is small and specific: a per-frame decision function measured in nanoseconds, sitting in front of model inference that is slower by many orders of magnitude. Keeping it allocation-free is not a bragging point, it is what keeps the cost model honest, because the moment the gate path allocates or blocks, the cost of watching a stream scales with frame rate again, the thing the filter-first split exists to prevent.
A probe that counts every allocation
"Zero allocations on the hot path" is an easy claim to make. Verifying it is the annoying part: heap profilers exist, but they are heavy, and heavyweight checks get run rarely, usually after someone already suspects a problem. I wanted the check cheap enough to run on every release without anyone thinking about it, which in practice means a small dependency-free binary whose output a shell script can parse.
Rust makes the exact version of this check unusually direct. A program can install its own global allocator, the object that every Box, Vec, and String in the process ultimately calls into. Wrap the system allocator and count the calls. The block below is a sketch of the idea, not the project's source:
struct CountingAlloc;
unsafe impl GlobalAlloc for CountingAlloc {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
ALLOCS.fetch_add(1, Ordering::Relaxed);
System.alloc(layout)
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
System.dealloc(ptr, layout)
}
}
#[global_allocator]
static A: CountingAlloc = CountingAlloc;The probe built around this is about eighty lines and depends on nothing outside the standard library. It drives the gate path with 120,000 frames to collect latency percentiles, then a separate 60,000 frame pass reads the allocation counter before and after. Because the counter sits at the global allocator, an allocation has nowhere to hide: one inside a dependency or behind a trait object lands in the same counter as one in the project's own code. If anything on the path touches the heap, the counter moves. That completeness matters, because a budget of exactly zero is only worth enforcing if the count cannot miss.
Running the probe today: p50 41 ns per frame, p95 42 ns, zero allocations across the 60,000 frames.
The pair of latency numbers deserves a second look before moving on. One nanosecond of spread between the median and the 95th percentile means the path does essentially identical work every time. A branch that occasionally went somewhere expensive would widen the spread, and so would a lock that is sometimes contended. An allocator would smear the distribution wide on its own, because allocation cost varies by orders of magnitude depending on what the heap has been doing lately.
Numbers that can fail the build
The probe by itself is just a nicer benchmark. The contract comes from the release gates that read its output. The enforced budgets fit in one table:
| release check | budget |
|---|---|
| gate path, p95 latency | at or under 50,000 ns |
| allocations per frame on the gate path | 0.0 |
| CLI binary size | at or under 25 MB |
| API server binary size | at or under 45 MB |
The two performance budgets are shaped differently on purpose, and the difference is the interesting part.
The latency ceiling sits three orders of magnitude above the measurement: 42 nanoseconds measured, 50,000 allowed. That looks absurdly slack until you ask what the check is for. A ceiling of 60 ns would be a tighter statement of today's numbers, and it would also fail the first time the release runs on a busy shared machine, and a check that fails for reasons other than the code trains everyone to ignore it, at which point it protects nothing. That is the honest constraint on any latency assertion made in CI: it has to hold on hardware you do not control. The ceiling is there for the class of regression that actually threatens this path: a lock, a syscall, a debug print left behind, something accidentally quadratic in a buffer size. Accidents in that class cost microseconds to milliseconds, so even the slack ceiling catches them. Machine noise stays in the nanosecond range and never gets close.
The allocation budget points the opposite way: exact, 0.0, no tolerance. It can afford to be exact because allocation counts are deterministic where nanoseconds are not; there is no CI machine noisy enough to make a non-allocating path allocate. So the smallest possible regression of this kind, one heap allocation on the gate path, however innocent it benchmarks on the laptop of whoever adds it, turns the release red. Together the two checks cover both the single allocation that would compound over months of commits and the sudden large slowdown, and neither produces false alarms.
The binary size caps look out of place next to nanoseconds, but they enforce the same idea one level up. vidarax is self-hosted; people are expected to download the binaries and run them. Dependencies in Rust arrive easily: a convenient crate brings its whole transitive tree with it, and the addition is nearly invisible in review until the binary has doubled. A cap converts that slow accumulation into an explicit decision: if a dependency is worth its megabytes, the change that adds it also raises the cap, visibly, in the same diff. The release profile keeps packaging slack out of the measurement, with fat link-time optimization, a single codegen unit, abort on panic, and symbols stripped, so what the caps track is mostly code.
So the release gates hold four numbers: 50 microseconds, 0.0 allocations, 25 MB, 45 MB, re-checked every time a release is cut. A budget like 0.0 needs structure behind it, because a zero-allocation steady state is not something you reach by deleting Vec::new calls one at a time.
Capacity planning by addition
A steady state with no allocation implies the buffers live somewhere else, and in vidarax they live in pools: memory allocated up front, handed out, returned, reused. The return trip is Rust-shaped. A pool is a bounded channel used as a free list, all operations non-blocking; what it hands out is an owned buffer whose Drop implementation sends the backing memory home to the pool. No consumer anywhere in the pipeline knows pooling exists, because owning the buffer and eventually dropping it is the entire protocol. If the pool is empty, the caller gets a freshly allocated buffer and the system degrades instead of blocking; if the pool is full when a buffer comes home, the buffer is simply dropped and freed normally. A test pins the round trip to pointer identity: release a buffer, reacquire it, and assert you got the same address back with the same capacity.
Every pool raises the same question: how many slots? The usual answer is a guess with margin, 512 and hope. The trouble with the guess is that nobody can tell when it has gone wrong. It encodes no reasoning, so it cannot be audited, and it survives refactors untouched until the day it is quietly too small under a load pattern nobody tried. vidarax computes pool capacities instead, as sums over the positions where a buffer can legally be in flight.
The decode-output pool is the small worked example. At any instant, at most one frame is being constructed by the reader, at most sixteen wait in the bounded reader queue, at most four sit in a pending FIFO allowance, and at most one is held by the consumer. Add the positions: 1 + 16 + 4 + 1, so the pool holds 22 slots, and every term in the sum names a position in the pipeline.
The JPEG pool gets the same treatment at scale, and its documented worst case is an itemized sum: 66 buffers between decode and analysis, 162 across the model and sink stages, 64 in the clip-frame queue, 64 in the clip accumulator, and 128 in active or blocked clip work, for a total of 484 slots. One term deserves its own sentence: the clip work queue has a capacity of zero, written as a named constant, because clip work should never queue at all; it either starts immediately or pushes backpressure upstream.
Arithmetic like this is only trustworthy if the terms are real, and this is where the language earns its mention. In Rust, who holds a buffer right now is not a convention, it is a compiler-checked fact: a stage cannot keep a second handle to a buffer it already passed downstream. So when a term says the reader queue holds at most sixteen buffers and the consumer at most one, the compiler is enforcing both, and adding the terms is legitimate. When the topology changes, a queue resized, a stage added, the sum changes in the same diff, and a reviewer can re-add it in their head.
So pools answer where the buffers come from, and sums answer how many there need to be. What remains is the traffic between stages, and two places where the traffic itself can wedge: a child process on the far end of a pipe, and an async caller that can vanish mid-request.
Drain before write
One stage of the pipeline talks to a decoder in another process: ffmpeg, spoken to over pipes, encoded video written to its stdin, raw YUV frames read back from its stdout. Two processes joined by pipes have one classic failure, and H.264 aims this pipeline straight at it. The decoder cannot emit a single frame until it has seen the stream's parameter sets and a keyframe, so there is a warm-up window where the writer must keep feeding input while nothing at all comes out. Run the two obvious designs against that fact. Lockstep, write one packet then read one frame, deadlocks immediately in the warm-up window, waiting for a frame that cannot exist yet. Writing freely and reading later works until the output pipe's buffer fills; then ffmpeg blocks writing its stdout, stops reading its stdin, the input pipe fills too, and our writer blocks. Both processes are now asleep, each waiting for the other to make room, and the stream is dead.
The protocol that survives has one dedicated thread and one rule. The thread does nothing but read stdout, assemble complete YUV frames, and push them into a bounded channel with blocking sends. The rule binds the writing side: decode always drains that channel before writing more input. So a full handoff channel can never be the reason the decode call is blocked at stdin, because the writer emptied the channel first; and the reader thread always regains somewhere to put the next frame, so it keeps draining ffmpeg's stdout, and neither pipe can wedge the pair.
Overload has a policy of its own. When frames arrive faster than downstream consumes them, the stage sheds decoded output, keeps the freshest frame, and counts every shed in the exported metrics. It never sheds encoded input, and the reason is mechanical: the decoder is stateful, later frames reference earlier ones, and a dropped input packet corrupts everything until the next keyframe. Dropping a decoded frame costs exactly that frame. The system's job is to say what is happening now, so the policy is freshness over completeness, and the counter shows how often the trade happened.
One limitation rides along with the pipe design, and the design notes state it plainly: raw YUV over a pipe carries no timestamps. The software decode path is frame-exact anyway; the GPU path labels frames best effort, correct to within a frame or two, while the pixels and every signal computed from them stay exact. The notes also record the alternative, wrapping the stream in a container format so timestamps survive the pipe, and why it lost.
Plain threads under the async API
Everything above runs on plain OS threads, and given where server-side Rust defaults have drifted, that choice is worth deriving. vidarax is not anti-async: the HTTP layer runs on tokio and should, because an API server's job is mostly holding many mostly-idle connections, which is exactly the workload async multiplexing was built for. The media path is the opposite shape. Decode, analysis, and the model workers are dedicated threads connected by bounded multi-producer multi-consumer channels. Each thread is either CPU-bound or blocked waiting on its neighbor, there are only a handful of them per stream, and work arrives continuously at frame rate. There is no idle mass to multiplex, so a runtime would add nothing here, and its costs would still apply: cancellation points in the middle of the pipeline, coupling to an executor, async signatures on every interface in between. Threads with bounded channels also come with a shutdown story that needs no framework: close an upstream sender and the hangup propagates downstream, so tearing down a stream is dropping the sender side and joining the threads.
The two worlds still have to touch, and the touching point is one trait: the event sink, the interface through which the pipeline publishes results. It is synchronous, Send and Sync, callable from a plain thread with no runtime anywhere in sight, and its documentation explains why it is thread-safe instead of asking the reader to trust the trait bounds: event sequence numbers come from a lock-free atomic counter, and the subscriber registry is swapped wholesale behind an atomic pointer, so publishing threads never take a lock. The async side wraps the sink and the sync side calls it. Neither needs to know more about the other than the trait says.
One invariant on the threading is written down at the point of temptation: worker count is clamped to one per stream. H.264 and VP8 decode carry state from frame to frame, and the frame gate's loop-detection state is likewise order-dependent, so splitting one ordered stream across two workers corrupts the output without raising any error. Parallelism happens across sessions only, and the comment saying so sits exactly where a future maintainer would otherwise raise the constant.
That is the whole concurrency design: tokio holds the connections, threads move the frames, one synchronous trait sits between them, and one worker owns each stream.
Where the sync-versus-async question was genuinely open, it got a harness instead of a position. The transport used for calling remote model providers exists in a blocking-thread flavor and an async flavor, and a benchmark compares the two and emits a recommendation only when one side wins by more than 15% throughput. Under that margin the difference is treated as run-to-run noise and no recommendation is made.
The window with no await in it
The async side has its own way of corrupting state. An async handler in Rust is a future, and the caller polling it, here the HTTP server acting for a connected client, can drop it at any await point. The handler does not observe this as an error; it simply ceases to exist between two of its own lines, and any invariant that was going to be restored after the next await never is.
Walk a session-creation handler with that hazard in mind. A client opens a WebRTC ingest session. The handler inserts the new session into the live registry, then spawns the watcher task that owns its lifecycle: monitors it, cleans up after it, frees its stream slot when it ends. Now let the client disconnect at the wrong moment. If the future is dropped after the insert and before the spawn, the registry holds a live session that no task owns. Its stream slot stays occupied forever, and since slots per principal are capped, enough of these ghosts and the server refuses new streams while doing no work at all. Nothing crashes and nothing logs an error; the server just runs out of slots over time.
The fix is to make the dangerous window contain no await at all. Between insert completion and watcher spawn the handler performs no asynchronous operation, so cancellation has no point at which to land inside the window: either the future died before the session became visible, or the session is visible with its reclaimer already attached. By construction there is no third state. The code says this in a comment at exactly that spot, because the property is invisible in the code's shape; nothing about a stretch of statements with no await in it announces that the absence is load-bearing, and the comment is what stands between the invariant and a future refactor inserting an await in the middle.
Reinforcements sit on both ends of the lifecycle. On the creation end, session creation runs as a detached transaction, so a client disconnect during creation cannot half-create a run. The test for it is as direct as tests get: spawn the creation, abort the task mid-flight, assert the transaction completed anyway.
On the teardown end, reclaim runs as a single idempotent, cancellation-safe winner with bounded-backoff retry, so the stream slot is freed exactly once no matter how many paths race to free it. A small map of already-reclaimed sessions keeps a late DELETE from tripping over a watcher that already won.
The slot itself is held by an RAII guard whose Drop runs even when a mutex was poisoned by a panicking thread, so a panic is not one more way to leak the slot. Together with the closed window, that covers the lifecycle: a session either never becomes visible, or it ends with its slot freed.
The number the README does not print
Given all this machinery, the README's performance section could print an impressive figure, and it refuses to. It says there is no single headline number worth quoting and points at the probe, the harnesses, and the release gates instead, so the numbers you get are from your hardware and your streams. The refusal came out of a documentation pass where I treated claims like code. I deleted a token-savings figure that had gotten into the docs without a measurement behind it. I cut marketing phrasing down to what is checkable in kind: bounded queues, lock-free counters, constant work per frame. The unflattering rows stayed in the stack table, including that the registered GPU decode path currently falls back to CPU ffmpeg and that the optional database mirror is present but not wired into the production path. The release gate numbers are narrower, and they are re-verified on every release, so those are the ones in print.
What the release gates do not cover
The release gates watch the gate path and the binaries, and nothing else. The decode pool's zero-allocation behavior was measured when that change landed and is pinned by the pointer-identity test, but no release gate re-counts allocations across the decode path on every release the way one does for the gate path [confirm: correct?]. End-to-end throughput and the remote HTTP path have harnesses that print p50 and p95, and nothing fails when those drift; they are measurements, and a measurement is a claim until it is enforced. The 50 microsecond ceiling is slack enough that a tenfold regression on the gate path, 42 nanoseconds becoming 420, would pass both checks as long as it allocated nothing. The release gates catch accidents. Slow erosion would need stored history and a tolerance policy, a different and noisier kind of check, and this repo does not have it.
So the enforced contract, stated in full: the gate path performs zero heap allocations, its p95 stays at or under 50 microseconds, the CLI fits in 25 MB and the API server in 45 MB, re-checked on every release, on whatever machine cuts it. Everything else the engine claims is currently measured rather than enforced. The decode path's allocation count is the obvious next number to move across that line, because the counting allocator already exists and the claim is already in writing; the erosion problem stays open, because it needs history I am not keeping yet.