A media pipeline can stop even when each component is behaving correctly. In Vidarax, one deadlock formed between ffmpeg's input and output pipes. A second failure left published sessions with nobody responsible for cleanup.
The pipe needed an ordering rule: drain output before writing more input. Session creation needed an ownership rule: publish the session only after its cleanup task exists.
A video stream through Vidarax follows the complete media pipeline.
How both ffmpeg pipes become blocked
One stage talks to ffmpeg over two pipes: encoded video goes into stdin and raw YUV frames come back from stdout. H.264 gives the pair an awkward warm-up. The decoder cannot emit a frame until it has seen parameter sets and a keyframe, so the writer must keep feeding input while nothing comes out.
Lockstep fails first. Write one packet, then read one frame, and the reader waits for a frame that cannot exist yet. Writing freely and reading later only moves the deadlock. Once stdout fills, ffmpeg blocks on its write and stops reading stdin. Then stdin fills and our writer blocks too.
Each program is waiting for the other. Before sending more data to stdin, our side has to make room for anything ffmpeg has already written to stdout.
Vidarax gives stdout a dedicated reader thread and drains its handoff channel before every input write. The handoff holds 16 decoded frames and uses blocking sends, so the reader can still stop draining stdout when the channel fills. This ordering narrows the deadlock window but cannot remove it without a bound on the maximum output burst from one input write. The current code has no such bound.
When decoded frames arrive faster than the next stage can consume them, Vidarax keeps the freshest decoded frame and counts every frame it drops. It never drops encoded input because later packets may depend on it.
The window with no await in it
An async Rust handler can be dropped at any await. The handler does not receive an error or get a chance to finish the work after that point.
A WebRTC creation handler once inserted the session into the live registry, then spawned the watcher that monitored it, cleaned it up, and released its stream slot. If the client disconnected between those operations, the future could disappear after the insert but before the spawn. The registry then held a session with no cleanup owner.
That session kept its stream slot forever. Because slots per principal are capped, enough orphaned sessions eventually stopped new streams even though the server was doing no work. Nothing crashed or logged an error.
There is no failed cleanup path to find here, because cleanup never acquired an owner.
The fix is to put no await between registry insertion and watcher spawn. Cancellation can then happen before the session is visible or after its reclaimer exists, but not between those states. A comment beside the spawn records why adding an await there would leak the session.
The creation test starts the detached transaction, aborts its caller mid-flight, and checks that the transaction still completes.
During teardown, one idempotent reclaim operation frees the stream slot. Competing callers use bounded-backoff retry, and a small map of reclaimed sessions handles a late DELETE after the watcher has already finished.
The slot guard releases its permit on normal returns and unwinding panics. Vidarax release builds use panic=abort. In that mode the process exits and destructors do not run, so it cannot continue with a leaked slot.
Where the blocking thread hands work to async code
Tokio serves many mostly idle HTTP connections. The media workers use dedicated threads because they stay CPU-bound or block on ffmpeg and downstream consumers.
Bounded channels apply backpressure and carry shutdown. One supervisor closes peer channels, raises the stop signal, and gives the remaining workers a limited time to finish after an unexpected exit.
A deadline can report a stuck OS thread, but it cannot kill one safely. If a worker does not return, Vidarax detaches it and records the forced shutdown. H.264 and H.265 decoding run in an ffmpeg child process so a native decoder crash cannot take down the service. A watchdog for a child that stays alive but stops making progress is still missing.
The blocking event sink returns errors directly. Its write-ahead log stores an event locally before acknowledging it, while keyframe creation writes the JPEG sidecar before appending metadata that points to it.
Each stream uses one media worker because H.264 and VP8 decoding carry state between frames and loop detection depends on order. Vidarax gets parallelism by processing separate sessions concurrently.