Astral's memory index stores each 384-component benchmark vector in either 1,536 bytes of f32 or 384 bytes of E5M2. The smaller form saves memory, but the CPU must turn each byte back into a number before it can score a candidate.
On x86, AVX2 and F16C can widen bytes, shift bits, and convert half floats. Neither can decode E5M2. The hot loop therefore has to build that missing conversion from instructions the CPU does provide.
A prompt through Astral follows the broader runtime.
An E5M2 byte
E5M2 is an 8-bit floating-point format: 1 sign bit, 5 exponent bits, 2 mantissa bits, packed in that order from the top of the byte. E5M2 puts its mantissa in bits 0 and 1, exponent in bits 2 through 6, and sign in bit 7. Extraction is one shift and mask. Its exponent bias is 15 and its largest finite value is 57,344. A normal E5M2 value decodes as
value = (-1)^sign * 2^(exponent - 15) * (1 + mantissa / 4)Take 0x3C and pull it apart. In binary it is 0 01111 00: sign 0, exponent 01111, which is 15, mantissa 00, which is 0. The formula gives 2^(15 - 15) * (1 + 0/4) = +1.0. So 0x3C is the value one.
Flipping the sign of 0x3C produces 0xBC and -1.0. Incrementing the mantissa produces 0x3D and 1.25. Incrementing 0x3F, the code for 1.75, rolls the mantissa into the next exponent and produces 0x40, the code for 2.0. Positive finite codes therefore increase with their decoded values, a property the quantizer uses when it compares codes.
Two mantissa bits give four representable values per power of two. Between 1 and 2 they are 1.0, 1.25, 1.5, and 1.75. The largest finite value is 57,344, encoded as 0x7B. The four codes above it, 0x7C through 0x7F, carry the all-ones exponent used for infinity and NaN. Astral clamps those inputs to a finite score.
Int8 uses its 256 steps evenly and needs a shared scale. One outlier can stretch that scale for every component that shares it. E5M2 gives each component its own exponent. It handles range better but has less precision. Astral supports both and compares them with the same acceptance tests.
Recall is the fraction of true nearest neighbors returned by the search. The pure E5M2 lane measured 90.62% on my fixtures, below the 99% default target. A second lane keeps the original f32 vectors beside the bytes. E5M2 routes the graph walk, then f32 reranks the survivors. That lane reached 100% recall after I tuned the rerank scaling, at the cost of storing both representations.
The conversion runs inside the innermost query loop. Every implementation must produce the same finite values for bytes written by Astral and leave recall unchanged. The scalar table remains the portable reference. Four exponent-31 codes need special care because the table clamps them while the hardware half converter returns infinity or NaN.
Five attempts
The loop converts 8 E5M2 bytes into 8 f32 lanes for every candidate visited by the graph. A 384-component candidate needs 48 of those conversions before the dot product is complete. A query over 10,000 candidates therefore passes 3.84 million stored bytes through the conversion loop. Two dot kernels use it, one with an f32 query and one with an E5M2 query.
I measured five implementations in one day. The measurements use different fixtures, so each result identifies whether it came from 10,000 or 100,000 vectors.
Scoring stored f32 directly would avoid conversion, but each row would grow from 384 to 1,536 bytes. The graph visits far more candidates than the reranker keeps, so loading f32 for every visit would give up much of the compact lane's cache advantage.
1. A 256-entry table
The baseline was a 256-entry lookup table. It decodes every byte once at startup, then uses one indexed load per component. The test covers all 256 values, including zero, subnormals, exponent-31 values, infinity, and the chosen NaN policy. The table occupies 1 KiB.
The table decodes each byte, clamps codes 0x7C through 0x7F to plus or minus 57344, and works without vector instructions. Every replacement has to preserve the values produced for Astral's own data.
2. Gather the table
Step two vectorized the lookup. AVX2 gather fetches eight table entries into one register, after the eight bytes have been widened into 32-bit indices. The table still supplies the same decode and clamp. At 100k vectors, query time fell from 8.961 to 5.436 milliseconds, 1.65x with recall unchanged. The gather still performs eight table reads per conversion, competing with vector data for cache, so the next attempts replaced those reads with arithmetic.
3. Shift into half-float position
E5M2 and FP16 put the sign and five-bit exponent in the same positions with the same bias. Only the mantissa width differs. A left shift can place the E5M2 bits into an FP16 value, which the hardware already knows how to convert to f32.
FP16 has one sign bit, five exponent bits, and ten mantissa bits. Write the two layouts next to each other:
E5M2: S EEEEE MM (8 bits)
FP16: S EEEEE MM00000000 (16 bits)half_bits = e5m2_byte << 8 produces an FP16 value with the same value because the extra mantissa bits become zero. F16C's _mm256_cvtph_ps converts 8 half words into 8 f32 lanes. For example, E5M2 0x3C becomes FP16 0x3C00, which is 1.0.
On a 10k flat-search measurement, the first implementation took 0.492 milliseconds per query at 90.62% recall. The existing gather version took 0.481 milliseconds at 91.15% on a different 10k fixture, and memory-mapped snapshot search also became slower. These were not paired timings, but every result rejected the new version. The bit mapping was right. The surrounding data movement was still too expensive.
4. Build f32 directly
Step four built f32 directly with masks and shifts: extract the fields, change the exponent bias from 15 to 127, and place each field in its f32 position. It removed the table and the half-float conversion. Query time rose from 0.483 to 0.585 milliseconds with identical recall, about 21% slower. The table won on this machine, so the direct upcast was reverted.
5. Keep the shift, change the movement
Step five, the same evening, was the F16C trick again with the data movement done right:
// widen 8 E5M2 bytes into 8 f32 lanes
bytes = load_8_bytes(p); // one small load
lanes = _mm256_cvtepu8_epi32(bytes); // zero-extend each byte into its own 32-bit lane
lanes = _mm256_slli_epi32(lanes, 8); // each lane now reads S EEEEE MM00000000
halves = pack_to_16bit(lanes); // squeeze back down to 8 half words
floats = _mm256_cvtph_ps(halves); // the hardware half decoder does the restThis version loads only the 8 source bytes. The gather also reads eight table entries. Everything between the load and decode is register arithmetic. GCC and Clang also needed -mf16c so the compiler could emit the conversion.
In a standalone 384-component dot-product probe, compiled with GCC 13.3.0 at -O3 -march=native, the full loop fell from 296.32 to 61.62 cycles per dot with matching sinks, a 4.81x improvement. In the paired 10,000-vector search, query time fell from 0.479 to 0.226 milliseconds, 2.12x, with recall at 90.62% in both measurements.
A separate four-accumulator measurement reached 0.225 milliseconds. I kept the 0.226-millisecond F16C version as the x86 fast implementation because it more than halved the paired query time without changing recall. The gather and table implementations remain as fallbacks.
That left three paths computing the same conversion in the source, and the question of what the two slower ones are still for.
The fallback paths
The slower implementations remain as portable fallbacks. Compile time decides which kernels the binary contains. At runtime, Astral selects the F16C path only when AVX state, AVX2, and F16C are all available. AVX2 builds without runtime F16C support keep the gather, and other builds use the scalar table.
The decoders disagree on four inputs. Shift codes 0x7C through 0x7F into FP16 and the hardware returns infinity or NaN. The table clamps them to plus or minus 57344. Astral's quantizer stops at 0x7B, so its own write path cannot produce the disagreement. Normalizing the fast path would add a compare and blend to every conversion for four unreachable codes, while changing the table would remove its finite-value guarantee. Neither shipped. Bytes from another writer remain the residual risk, and the decoder comment states that boundary.
F16C, gather, and scalar conversion all begin with loads. Their cost depends on where the bytes sit.
The bytes around the instruction
The first F16C attempt lost because conversion was only part of the cost. The kernel still had to load packed bytes, widen them, and handle row boundaries. Better data movement turned the same idea into the 0.226-millisecond version.
This AVX2 loop loads 32 bytes per iteration. A 32-byte-aligned load cannot straddle a 64-byte cache line, while an arbitrary address can. Astral's 384-byte E5M2 rows are exactly six cache lines when they start on a line boundary, so the allocator aligns the row and the loop can issue twelve 32-byte steps without crossing a row or leaving a tail. The six-line and twelve-step counts are size arithmetic. I did not measure the isolated cost of a straddling load.
Other dimensions need an explicit tail policy. At 300 components, nine full steps cover 288 and leave 12. A kernel can finish those 12 scalarly, use a masked load, or pad the row. The current fixture avoids the choice because 384 divides by 32, but the kernel contract still states its stride and alignment so a future dimension cannot silently drop values.
Cache-line ownership and LL/SC reservation granules are separate alignment problems. They are covered where they affect the actual design: cache-line layout for shared state and reservation alignment for synthesized atomics.
When the toolchain hides an instruction you have
The fastest measured implementation was still absent from one Windows binary because MSVC exposed the F16C intrinsics without defining the feature macro the guard expected. Tests stayed green because the fallback returned identical values. My routine inspection of the MSVC loop exposed the mistake: the expected vcvtph2ps instruction was absent. One internal build macro fixed compile-time selection.
The compiler fix and the runtime guard answer different questions. The build macro says the compiler may emit the intrinsics. Startup verifies OS AVX state, AVX2, and F16C before selecting the kernel. A legal AVX2-without-F16C machine therefore reaches the gather fallback and never executes an instruction it did not advertise.
In another case, the CPU really lacked the operation and I built an atomic from weaker instructions. That is a separate post.
After conversion: one more bottleneck
Once the paired 10k query fell from 0.479 to 0.226 milliseconds, the running sum became easier to see. A dot product with one accumulator is one dependency chain: each FMA waits for the previous FMA's result. The widening instructions are independent of that chain, so the loop can overlap more work by carrying several partial sums and reducing them once at the end.
The E5M2 kernels moved from two vector accumulators to four. On the 100,000-vector insert fixture, p95 fell from 1.926 to 1.899 milliseconds and p99 from 2.079 to 2.049 milliseconds. The median and recall did not move. A wider f32 loop measured as no change and was reverted. The SIMD post follows those dependency chains, the emitted loop, and both verdicts.
The conversion comparison used the pure E5M2 10,000-vector lane, where recall stayed at 90.62%. The 99% release check applies to named high-recall lanes, including reranked storage. The 100,000-vector insert run around 2.05 milliseconds measured the later accumulator change.
E5M2's quarter-step precision still needs an f32 rerank to recover recall. The quantizer avoids the four infinity and NaN codes by stopping at 0x7B, so every writer must use that rule. F16C also has to pass both the compile-time and runtime feature checks.
On the Intel Core Ultra 7 265, F16C v2 reduced query time from 0.479 to 0.226 milliseconds with recall unchanged. The fallbacks remain compiled for other CPUs. The Compiler Explorer reproduction includes both accumulator loops and the quantizer.