The test that rejected my quantizer_

At 100,000 vectors, Astral's memory index returned 12.5% recall. It should have been near 99%. The benchmark exposed the problem, but the release process would still have accepted the build.

At 10,000 vectors, individual queries ranged from 25% to 87.5% recall. A 64x search budget recovered the missing neighbors, but query time rose to about 12 milliseconds. The graph was unusable at the operating point I needed, so construction became the next place to work.

Fixing the graph solved that bug, but another quantizer could still compile and return bad answers. I added a reproducible recall check. Days later, it rejected E3M2 and kept Q8.

Measuring correctness

The reference is exhaustive flat float32 search. It scores every stored vector with the chosen float32 distance. There is no graph to omit candidates and no storage quantization. Floating-point arithmetic still rounds.

Recall is the overlap between the exhaustive top-k result and the index's top-k result. If two top-ten lists share nine entries, recall is 90%. The script checks mean recall over a fixed query set. I inspect the per-query distribution separately because the mean can hide bad queries.

Reference and candidate answer the same stored vectors and query set. The fixtures here use 10,000 or 100,000 vectors at 384 dimensions, and every result names its scale. Change the data between the two searches and the overlap no longer measures the index.

Exhaustive search is expensive. On the broken graph, a 64x query budget recovered 100% recall at about 12 milliseconds per query. The default budget took 65 to 85 microseconds, roughly 176x less on the measured baseline.

The exhaustive reference is used by the test. Production queries still use the index.

The check compares each indexed result with that reference and applies its thresholds.

The check I added

The required test rejects a named storage format below 99% recall. A caller may also add a p99 insertion ceiling.

The optional ceiling catches changes that buy recall with excessive insertion cost. One wider construction search reduced recall at 100,000 vectors and added more than a millisecond to p99 insertion latency per record.

I use p99 because an average hides slow inserts. Even a small fraction matters across 100,000 records.

The first thing it rejected

Before graph storage, 100,000 vectors with 384 float32 components use about 150 megabytes. Each vector is 1,536 bytes raw, 384 bytes with 8-bit codes, or 288 bytes packed at 6 bits.

Four candidates went in:

Q8, the default the index already shipped, stores each component as an 8-bit integer, mapped back through a scale factor. It is plain and well understood.

E2M3 trades exponent range for a third mantissa bit. E3M2 favors exponent range instead. I built the E3M2 path that week with int16 decoding and AVX2 accumulation.

F8 E5M2 uses 1 sign bit, 5 exponent bits, and 2 mantissa bits. It has more range and less precision.

All four formats used the same 10,000 by 384 fixture and the same exhaustive float32 reference. I left the float32 rerank pass out so the table shows what each stored format does on its own.

Q8 and E2M3 took 124.7 and 129.2 microseconds. Q8 had about three points more recall and was the only format above 99%. E3M2 and E5M2 were several times slower and both stopped at 90.62% recall. I had built E3M2 days earlier to try to improve on E5M2.

Why two formats stopped at 90.62%

The 10k compact fixture made the 90.62% result easier to explain. Its 3.84 MB E5M2 table falls below Astral's 16 MiB compact-index cutoff, so the benchmark chose exact flat search and scored every stored vector. Graph routing did not participate. The stored E5M2 scores still reproduced only 90.62% of the exhaustive float32 top results.

The encoding explains the result. E5M2 has 2 mantissa bits, so it represents four values between each pair of powers of two. Between 1.0 and 2.0, those values are 1.0, 1.25, 1.5, and 1.75. Stored components snap to that grid. Close neighbors can then tie or swap after rounding. A larger candidate pool cannot recover precision that was lost during storage.

E3M2 tested the same limit with another layout and an int16 decode path with AVX2 accumulation. Its exact 10,000-vector flat result also reached 90.62%. Both E3M2 and E5M2 have 2 mantissa bits.

E2M3 has 3 mantissa bits and reached 96.48%. That extra bit provides eight values between adjacent powers of two instead of four. On these vectors, the finer grid produced nearly six more recall points.

On this 10k fixture, two different two-mantissa implementations both measured 90.62% under exact flat scoring. That is the observed ceiling for these lanes and quantizers here, not a universal limit of every two-mantissa format. Both missed the 99% floor.

The formats I kept

Q8 shipped as the default. It was the only raw format above the floor and the fastest of the four: 99.61% recall at 124.7 microseconds, measured against the exhaustive reference.

E2M3 had the highest measured recall among the six-bit candidates and used 25% fewer component bits than the eight-bit lanes. Its 96.48% still missed the floor.

I reverted the E3M2 work because it reached only 90.62% recall and took 455.1 microseconds per query, more than three times Q8's 124.7 microseconds on the same fixture.

E3M2 and E5M2 remained available with float32 reranking. Their compact scores route the search, then full-precision scores reorder the final candidates. E2M3 with reranking and bare E5M2 were not promoted because neither configuration had measured 99% recall.

Reranked formats receive three times the query budget of plain quantized storage. Float32 needs a wider candidate pool to improve the final order. Too small a pool misses useful candidates. Too large a pool wastes query time.

The latency column is from that comparison. Later kernel work pulled E3M2 to roughly 205 microseconds and E5M2 to roughly 483, while both stayed at 90.62% recall. Their raw formats still failed the recall requirement.

Tuning against a floor

After the format decision, doubling graph connectivity let the required efSearch budget fall from 4096 to 1536 while holding the recall target, a 2.67x parameter reduction paid for with more memory and build cost. The same test checks the configured storage format against the fixed query set.

During diagnosis, I doubled base-layer connectivity relative to the upper layers. It looked fine at small scale, then dropped recall to 50% and increased latency at 100,000 vectors.

After rebuilding the graph, I doubled connectivity across every layer and reduced the required query budget by 2.7x. Both the change and the graph were different, so I measured the new configuration instead of carrying the older result forward.

The retune produced a high-recall starting point for this fixture: 64 neighbors, 128 build-time search, and 1536 query-time search at 100k vectors and 384 dimensions. That configuration cleared the 99% recall test.

Choosing the recall and latency limits

Each configured storage format is compared with exhaustive float32 search over the same queries. These results use 10,000 or 100,000 vectors at 384 dimensions from one embedding distribution, so a new model or data distribution needs a new comparison.

I chose 99% because the rebuilt index could reach it. The insertion ceiling remains optional.

Query latency never received a hard threshold. A ten-point sweep at 100k ran from 69.1% recall at 1.8 milliseconds to 99.2% at 8.8. The top point showed that 99% was reachable in the offline reference sweep, not that it met the production retrieval budget. The sweep also contained a dominated setting: budget 3840 repeated budget 3584's 98.83% recall at 8.366 milliseconds instead of 7.996. A recall floor cannot reject it because both points fail together.

The recall requirement is clear. Query latency and the optional insert-p99 budget still come from the recall-latency curve. I have not found one latency rule that rejects the dominated plateau without also cutting away useful points near the knee.