At 100,000 vectors, Astral's memory index returned 12.5% recall. It should have been near 99%. The benchmark exposed the problem, but the release process would still have accepted the build.
The smaller 10,000-vector run showed why the failure was serious: individual queries ranged from 25% to 87.5% recall. Searching a little harder could not repair a graph that routed some queries into the wrong neighborhoods. The construction path had to be rebuilt.
Fixing the graph solved the immediate bug. This article is about the second fix: turning the 99% target from a number I checked by hand into a release check that can fail on its own. It starts with the exact-search oracle, then follows the gate through the first storage-format decision it made.
The oracle
A gate needs something to measure against, and for recall the choice writes itself: exhaustive flat float32 search. Exhaustive means the query is compared against every stored vector; the graph and its candidate lists are not involved, and nothing exits early. Float32 means full precision, no quantization loss on the stored data. A search that never skips a candidate and never rounds one cannot miss, so whatever it returns for a query is, by construction, the correct answer for the index's contents. That is the recall oracle.
Recall is then just overlap. Ask the oracle for a query's true nearest neighbors, ask the index under test for the same count, and score the fraction of the true set the index actually found. If the two lists share nine entries out of ten, that query scored 90%. (Illustrative arithmetic; the point is the definition, not the ten.) Averaged over a query set, that one number is the whole quality story for an approximate index, because everything the approximate index does, from the graph to the search budget, exists to approach the oracle's answer at a tiny fraction of the oracle's cost.
One more piece makes the oracle usable: the measurement contract around it. Oracle and candidate answer the same fixture, the same stored vectors and the same query set, because recall is only meaningful as a comparison between two answers to one question. Change the vectors or the queries between the two runs and the overlap measures the difference in the data instead of the index. The fixtures here are fixed sets at two scales, 10,000 and 100,000 vectors at 384 dimensions, and every number in this post is scored on one of them. A contract like that is easy to violate by accident, so the script holds both runs to the same inputs rather than leaving it to discipline.
The obvious objection is that cost. Exhaustive float32 search over 100,000 vectors at 384 dimensions is exactly the work the index exists to avoid, and the diagnosis week had already put a price on doing things the brute-force way. On the broken graph, pushing the per-query search budget to 64x default recovered 100% recall, but at about 12 milliseconds per query against the 65 to 85 microseconds the default budget was spending, roughly 176x on the measured baseline. No query path can pay that.
But acceptance is not a query path. A release run happens occasionally, on a machine with no user waiting on it, and it can take as long as it takes. So the same arithmetic that rules exhaustive search out as the shipping configuration is what makes it affordable as the oracle: a query pays for search on every request, a release pays for ground truth once per release.
The oracle is nothing more than that, an exhaustive search that runs outside the code path it measures. Everything else in the gate is a threshold applied to what it returns.
One release floor and one optional budget
The acceptance script can enforce two numbers: a recall floor against the oracle and a ceiling on p99 insertion latency. The hard release lane currently invokes the first one at 99% for the named storage lanes. It does not pass the optional p99 ceiling today, so insertion latency is a measured budget, not a release-blocking verdict.
The floor is the target I had been carrying in my head, finally written somewhere that can act on it. The optional ceiling exists because recall improvements tend to cost insertion time. During the diagnosis week, one plausible construction change, widening the build-time search, regressed recall at 100k scale and pushed p99 insertion latency up by over a millisecond per record on top of it. The acceptance runner can reject that combination when the caller supplies the ceiling and the lanes to check. The release runner has not made that policy mandatory yet.
When I use the insert budget, it sits on p99 rather than the average because the tail is where construction trouble surfaces first. An average smooths over a small fraction of pathological inserts, and a small fraction of 100,000 inserts is still a lot of pathological inserts, and each one stalls whatever thread issued it.
The part that matters more than either number is where it lives. The 99% floor is an explicit argument in the required release plan, and the script fails when a named lane misses it. The p99 ceiling is also executable rather than a dashboard note, but only in acceptance runs that opt into it. That distinction is visible in the command line instead of being hidden behind prose.
The same runner can wrap its benchmark lanes with hardware performance counters when requested. Those carry no thresholds, because they inform capacity planning and point at the next optimization, but a counter moving is not by itself a reason to block a release. The required release invocation does not require counters.
So the required gate, in full: an exhaustive float32 oracle for ground truth and a 99% recall floor as a hard failure. The acceptance runner also has an opt-in p99 insertion ceiling and optional counters. Underneath it sits one rule: no compressed or approximate format ships as a default without a measured accuracy floor against ground truth. The gate got its first real decision within days: which storage format the index ships as its default.
The shootout
Compression is not optional here: 100,000 vectors of 384 float32 components is around 150 megabytes of raw vector data before any graph structure (100,000 x 384 x 4 bytes), so the stored components get quantized, and the question is how hard and in what format. The per-vector arithmetic makes the stakes plain: 1,536 bytes raw, 384 bytes as 8-bit codes, 288 packed at 6 bits.
Four candidates went in:
Q8, the default the index already shipped, stores each component as an 8-bit integer, mapped back through a scale factor; it is plain and well understood.
The two 6-bit floats spend the same budget two ways. F6 E2M3 has 1 sign bit, 2 exponent bits, 3 mantissa bits, most of its budget on precision rather than range. F6 E3M2 is laid out the other way, 3 exponent bits and 2 mantissa bits, more dynamic range on paper. I built E3M2 myself that same week, decoded to int16 and accumulated with AVX2, and I will get to why.
F8 E5M2 is an 8-bit float: 1 sign bit, 5 exponent bits, 2 mantissa bits. Huge dynamic range for 8 bits, because it spends 5 of them on exponent, and rough precision, because only 2 are left for mantissa.
The measurement contract kept the comparison honest: the same 10,000 x 384 fixture for all four, recall scored against the same float32 oracle, and no float32 rerank pass in the loop, so the numbers reflect the formats themselves.
| Format | Latency (us/query) | Recall |
|---|---|---|
| Q8 | 124.7 | 99.61% |
| F6 E2M3 | 129.2 | 96.48% |
| F6 E3M2 (new) | 455.1 | 90.62% |
| F8 E5M2 | 854.9 | 90.62% |
What the table says, in plain terms: there are two clusters. Q8 and E2M3 sit within 5 microseconds of each other at the top, 124.7 against 129.2, with Q8 ahead on recall by about three points and the only format above the 99% floor. The other two formats are several times slower, and they stopped at the same recall, 90.62%, identical to the second decimal. One of those two was brand new, built by me days earlier, specifically to score higher than the other one.
The ceiling
The cleanest evidence that 90.62% was a representation ceiling came from the path the 10k compact fixture actually ran. Its 3.84 MB E5M2 table falls below Astral's 16 MiB compact-index cutoff, so the benchmark chose exact flat search and scored every stored vector. The graph budget on those captures was not participating. Exact search still stopped at 90.62%, which removes graph routing from the explanation: the stored scores themselves could not reproduce more of the float32 oracle's top results.
The remaining suspect is the encoding, and the mantissa arithmetic makes the case concrete. E5M2 keeps 2 mantissa bits, which means four representable values between one power of two and the next: between 1.0 and 2.0 the format can say 1.0, 1.25, 1.5, or 1.75, and nothing in between. Every stored component snaps to a grid that coarse, so every distance computed from those components is coarse too. Near the top of a nearest-neighbor ranking, where the true winner and the runners-up are genuinely close, coarse distances make different neighbors tie or swap. The true neighbor is not lost by the search; it is outscored by a neighbor whose rounded distance happens to look better. A bigger candidate pool just reorders the same rounded distances, because the rounding happened when the components were stored.
E3M2 tested the same boundary from another direction. I built it with a different layout, a different bit budget, and an int16 decode path with AVX2 accumulation. Its exact 10k flat result also landed at 90.62%. The property the two ceiling formats share is the 2 mantissa bits. E2M3, one row up in the table with 3 mantissa bits, scores 96.48%. That third bit gives eight representable values between one power of two and the next instead of four, and the nearly six points between E2M3 and the ceiling pair are consistent with the finer grid on these vectors.
So the consolidated picture is narrower and stronger: on this 10k fixture, two different 2-mantissa-bit formats both reach 90.62% under exact flat scoring. No graph miss is mixed into that number. The gate did not need to know the mechanism to decide: both formats measured 90.62 against the 99 floor, and both failed the acceptance run.
What the gate decided
Q8 shipped as the default. It was the only format above the floor on its own and it was also the fastest of the four, so there was no tension to arbitrate, and the gate's contribution was making the non-decision official: 99.61% recall at 124.7 microseconds, measured against the oracle, written down next to the settings that produced it.
E2M3 stays interesting. 96.48% is well below the floor, but it is the best recall per bit in the table, 96.48% out of 6 bits against Q8's 99.61% out of 8, and at 288 bytes per vector against Q8's 384 it is real memory savings at scale. If that footprint ever earns a place, E2M3 re-enters through the same gate and clears the same floor, not through my memory of it having been close once.
E3M2 I reverted, and it was my own week of work. Keeping it had a real argument: the int16 decode path was fresh and genuinely fast for what it did, the format was more general on paper than what it was meant to replace, and reverting meant filing the whole effort as a negative result. But effort is not an input the gate reads, and E3M2 cleared the bar on neither axis: slower than both top formats, stuck on the same ceiling as the format it was built to beat.
The two ceiling formats did not disappear from the codebase, though, because a ceiling on the compressed score is not a ceiling on the whole pipeline. Add a float32 rerank pass and the compressed format's job changes: it routes the search toward a candidate pool, and full-precision scores reorder that pool at the end, so the rounding that created the ceiling stops deciding the final ranking. Under that policy the acceptance matrix kept E3M2 and E5M2, each paired with float32 rerank, as high-recall compact graph lanes. E2M3 with rerank and bare E5M2 stayed unpromoted: promoting a lane to a 99% default puts the floor on that lane as it is configured today, and neither configuration had a measured pass at 99 to point at, only the possibility that more tuning finds one.
So a format that failed the floor on its own can still hold a lane where it only routes and float32 decides the final order. The rerank is not free either: formats with a rerank pass get 3x the query-time search budget of plain quantized storage, because reordering final candidates by float32 needs a wider pool than routing alone does. Get that multiplier wrong and one of two things happens: the rerank runs short of candidates worth reordering, or the query spends budget on candidates the rerank was always going to discard.
The latency column is a snapshot. Later kernel work pulled E3M2 to roughly 205 microseconds and E5M2 to roughly 483, while both stayed at 90.62% recall. The gate still rejected their raw lanes because recall, not speed, was the failed axis.
Tuning against a floor
The less obvious payoff came after the format decisions, in what the gate made me willing to try. The rebuilt index's biggest retune was doubling graph connectivity, which let the per-query search budget drop from 4096 to 1536 while still holding the recall target, about 2.7x less search per query, paid for with more memory and build cost up front. That is an aggressive change to make to a system that had been on fire the week before, and the reason I could make it quickly is the floor: the same 99% threshold, scored against the same oracle, would fail the release if the cheaper search quietly cost recall. I did not need to soak the change and watch it, because a recall regression cannot make it out unnoticed.
There is a reconciliation buried in that retune, and it sharpened the whole argument for me. I had already pulled the connectivity lever once, during the diagnosis week: doubling base-layer connectivity relative to the upper layers, a common convention, on the graph as it stood then. At small scale it looked fine; at 100k it dropped recall to 50% and made latency worse on top. The retune pulled what looks like the same lever, doubling connectivity uniformly, on the rebuilt graph, and it paid for the 2.7x budget cut. The same knob got opposite verdicts because the graph underneath was a different graph. So a remembered verdict about a knob transfers poorly even within one codebase and one month; the gate re-measures the current system instead of trusting what the knob did last time.
The retune produced a high-recall starting point for this fixture: 64 neighbors, 128 build-time search, and 1536 query-time search at 100k vectors and 384 dimensions. Each number was written next to the default only after the release script verified the shipped configuration.
vidarax, a Rust video intelligence engine with a real-time media path, carries the same idea further and gates allocation counts and latency percentiles directly in CI. A video stream through Vidarax maps the full system, and The build fails if you allocate follows that particular gate.
What it costs and what stays open
The oracle's price lands on every acceptance run: exhaustive float32 search over the fixture, per lane, per configuration under test. That was the deal from the start: the release run absorbs the brute-force cost. But the matrix wants to grow, because every new format and every rerank policy adds a lane, and each lane multiplies the oracle work. At some size the acceptance run itself becomes a budget to manage.
Every verdict in this post is a verdict on this fixture: 10,000 or 100,000 vectors, 384 dimensions, one embedding distribution. The 90.62% ceiling is a fact about these vectors meeting 2 mantissa bits, and a different embedding model could move that ceiling in either direction. The gate defends the numbers on the fixture it has, and nothing in it notices if the fixture drifts away from what production actually stores. Keeping the fixture representative is a maintenance job the gate cannot do for itself.
And the thresholds themselves are judgment calls. 99% is where I set the floor because the measurements showed the rebuilt index could afford it. The insert ceiling is still an opt-in budget rather than part of the hard release invocation. Whether either judgment is still right a year later is a separate review that no script triggers.
Query latency never received a hard threshold. A ten-point sweep at 100k ran from 69.1% recall at 1.8 milliseconds to 99.2% at 8.8, and the top point showed that the 99% floor was affordable. The sweep also contained a dominated setting: budget 3840 repeated budget 3584's 98.83% recall at 8.366 milliseconds instead of 7.996. A recall floor cannot reject it because both points pass or fail together.
I do not yet have a latency rule that rejects the dominated plateau without also rejecting useful points near the knee. The hard release gate therefore enforces recall. Query latency and the optional insert-p99 budget remain choices made from measured curves and explicit acceptance runs.