Astral stores each 384-component benchmark vector in either 1,536 bytes of f32 or 384 bytes of E5M2. The smaller form saves memory, but the CPU must turn each byte back into a number before it can score a candidate.
On x86, AVX2 and F16C can widen bytes, shift bits, and convert half floats. Neither can decode E5M2. The hot loop therefore has to build that missing conversion from instructions the CPU does provide.
The broader runtime is mapped in A prompt through Astral. This article stays on that one conversion path, starting with the format itself.
What an E5M2 byte is
E5M2 is an 8-bit floating-point format: 1 sign bit, 5 exponent bits, 2 mantissa bits, packed in that order from the top of the byte. In astral the constants are: sign shift 7, exponent shift 2, exponent bias 15, mantissa bits 2, max finite 57344.0f. The shifts are the field positions restated: the mantissa lives in bits 0 and 1, the exponent in bits 2 through 6, the sign in bit 7, so extracting a field is one shift and one mask. A normal E5M2 value decodes as
value = (-1)^sign * 2^(exponent - 15) * (1 + mantissa / 4)Take 0x3C and pull it apart. In binary it is 0 01111 00: sign 0, exponent 01111, which is 15, mantissa 00, which is 0. The formula gives 2^(15 - 15) * (1 + 0/4) = +1.0. So 0x3C is the value one.
Each field has one job. Flipping the sign of 0x3C produces 0xBC and -1.0. Incrementing the mantissa produces 0x3D and 1.25. Incrementing 0x3F, the code for 1.75, rolls the mantissa into the next exponent and produces 0x40, the code for 2.0. Positive finite codes therefore increase with their decoded values, a property the quantizer uses when it compares codes.
The quarter steps are the whole story of the format's precision. Two mantissa bits means four representable values per power of two, and nothing in between. Between 1 and 2 the format can say 1.0, 1.25, 1.5, 1.75. Between 2 and 4 it can say 2.0, 2.5, 3.0, 3.5. The pattern continues all the way up, so between 32768 and 65536 the representable values sit 8192 apart: 32768, 40960, 49152, 57344. That last value is the top of the format, and now the max finite constant explains itself: 57344 = 2^15 * 1.75, the biggest mantissa on the biggest normal exponent, which is the byte 0x7B = 0 11110 11. The four codes above it, 0x7C through 0x7F, have an all-ones exponent field, and in an IEEE-shaped format an all-ones exponent means infinity and NaN.
On the other end, the smallest normal value is 2^(1 - 15) = 2^-14, about 0.000061. So one byte covers roughly nine decimal orders of magnitude.
What the rough precision costs is visible the moment you store an ordinary number. Take 0.1. The nearest exponent is 2^-4 = 0.0625, and the four mantissa steps there are 0.0625, 0.078125, 0.09375, 0.109375. The value 0.1 falls between the last two, about 6% from one neighbor and 9% from the other, so the stored byte starts life several percent wrong whichever neighbor it lands on (illustrative arithmetic from the formula, not a measured astral error). A dot product then sums hundreds of these small errors, some canceling, some not. For steering a search toward the right neighborhood, that is fine. For deciding which of two close neighbors is truly closer, it is not.
It is fair to ask why a float byte at all, when int8 also fits in one. An int8 grid is uniform: 256 evenly spaced steps, so it needs a shared scale factor to place the grid over your data, and one outlier component stretches the grid for every component sharing the scale. E5M2 carries an exponent in every byte, so each component picks its own magnitude and an outlier costs only itself. The price is resolution: int8 spends all eight bits on positions inside the grid, E5M2 spends five of them on range. Astral ships both kinds of lane and lets the acceptance numbers rank them.
That last clause is why the lane exists in two forms. Recall is the fraction of the true nearest neighbors an approximate search actually returns, and I care about 99% recall as the default target. Score everything through quarter-step bytes and near neighbors blur together at the cutoff: the pure E5M2 lane measured 90.62% recall on my fixtures, which makes it a useful compact lane and rules it out as a 99% default. Keep the original f32 vectors alongside the bytes and the roles split: the bytes steer the graph walk, where being roughly right is enough, and the f32 copies rerank the survivors at the end, where being exactly right is the whole point. That paired lane reached 100% recall after I tuned the rerank scaling, and it pays for the number in memory and cache pressure, because both representations are now resident. So the byte does the approximate work and f32 does the exact ranking.
That split only works if converting bytes back to f32 costs almost nothing, because the conversion sits inside the innermost loop of every query. For bytes written by Astral's quantizer, every candidate path must produce the same finite f32 value for each code, preserve search ordering well enough to leave recall unchanged, and keep the scalar path as the portability oracle. The four exponent-31 codes need separate treatment because the table clamps them while the hardware half converter produces infinity or NaN.
What I tried, in order
The operation is to take 8 stored E5M2 bytes and produce 8 f32 lanes inside the innermost loop, for every candidate the graph walk touches. The lane has two dot kernels, one scoring stored bytes against an f32 query and one against a query quantized to the same byte format, and both stand on that conversion. I built and measured five paths in one day. The captures do not all use the same fixture, so the numbers below identify 10k and 100k runs instead of placing unlike query timings on one scale.
One shortcut I ruled out before the first step: skip the conversion by scoring against stored f32 directly. Even the paired rerank lane, which has f32 copies sitting right there, steers the graph walk with the bytes and touches f32 only for the final rerank of the survivors, because the walk visits far more candidates than it keeps. Pull a 1,536-byte f32 row through the cache for every candidate visited and the cache pressure eats the win the compact format exists to buy. So the conversion has to happen at scoring time, and it has to be fast there.
1. A 256-entry table
Step one was a 256-entry lookup table, and it became the baseline. A byte has 256 possible values, so decode every one of them once, at startup, into an array of floats, and the conversion is one indexed load per byte. Even this baseline was not naive: the first scalar kernels already kept four partial sums running. The table is correct by construction: all 256 entries can be checked against the decode formula by hand. It is also small: 256 floats is 1 KB, sixteen cache lines, so after a few queries the whole table sits in L1 and the indexed load almost always hits (cache arithmetic, not a measured hit rate).
The table is doing more than converting, and every replacement has to be audited against each of its jobs:
1. It decodes: byte in, f32 out, per the formula. 2. It clamps: for the four codes 0x7C through 0x7F it does not answer what the format says, which is infinity and NaN. It answers +57344 or -57344, the max finite. Whatever byte arrives, the score stays finite and orderable. 3. It is the portability floor: it needs no vector instructions at all, so it runs on anything astral compiles for.
2. Gather the table
Step two vectorized the lookup. AVX2 gather fetches eight table entries into one register, after the eight bytes have been widened into 32-bit indices. The table still supplies the same decode and clamp. At 100k vectors, query time fell from 8,960,779 to 5,436,180 nanoseconds, 1.65x with recall unchanged. The gather still performs eight table reads per conversion, competing with vector data for cache, so the next attempts replaced those reads with arithmetic.
3. Shift into half-float position
Step three is where the trick enters, and to see it you need one more format on the table. FP16, the IEEE half float, is: 1 sign bit, 5 exponent bits, 10 mantissa bits. Same sign position, same exponent width, same bias 15 as E5M2. Write the two layouts next to each other:
E5M2: S EEEEE MM (8 bits)
FP16: S EEEEE MM00000000 (16 bits)E5M2 is basically the top 8 bits of a half-float-shaped value. So half_bits = e5m2_byte << 8 produces a genuine FP16 number with the same value, because the two extra mantissa bits come in as zeros. And x86 does have a hardware half-to-float decoder: F16C, an instruction set extension whose _mm256_cvtph_ps turns 8 half words into 8 f32 lanes in one instruction. Worked example, with the byte from the primer: 0x3C shifted left by 8 is 0x3C00, and FP16 0x3C00 is exactly 1.0. The instruction x86 does not have, decode E5M2, becomes one shift plus an instruction it does have.
On a 10k flat-search capture, the first implementation measured 492,446 nanoseconds per query at 90.62% recall. The existing gather path measured 480,930 nanoseconds at 91.15% in its own 10k capture, and memory-mapped snapshot search also regressed. Those are not a paired recall-parity comparison, but they were enough to reject the first implementation: it had not improved latency, and it made the mapped path worse. The bit mapping was right; the surrounding data movement was not yet good enough.
4. Build f32 directly
Step four built f32 directly with masks and shifts: extract the fields, change the exponent bias from 15 to 127, and place each field in its f32 position. It removed the table and the half-float conversion. Query time rose from 482,963 to 584,678 nanoseconds with identical recall, about 21% slower. The table won on this machine, so the direct upcast was reverted.
5. Keep the shift, change the movement
Step five, the same evening, was the F16C trick again with the data movement done right:
// widen 8 E5M2 bytes into 8 f32 lanes
bytes = load_8_bytes(p); // one small load
lanes = _mm256_cvtepu8_epi32(bytes); // zero-extend each byte into its own 32-bit lane
lanes = _mm256_slli_epi32(lanes, 8); // each lane now reads S EEEEE MM00000000
halves = pack_to_16bit(lanes); // squeeze back down to 8 half words
floats = _mm256_cvtph_ps(halves); // the hardware half decoder does the rest-O3 -mavx2 -mf16c -mfma · open in compiler explorerTwo properties of that shape matter. The only memory it touches is the 8 bytes themselves, once; the gather touched the table eight more times per conversion. And everything between the load and the decode is register arithmetic the core can schedule as it likes. There was also plumbing, because the fast path only exists if the compiler is allowed to emit F16C at all: the GCC and Clang builds gained the -mf16c flag alongside the code.
In a standalone 384-component dot-product probe, compiled with GCC 13.3.0 at -O3 -march=native, the conversion moved from 296.32 to 61.62 cycles per dot with matching sinks: 4.81x on that kernel. In the paired 10k graph-search capture, query cost moved from 478,806 to 225,704 ns/op, 2.12x, with recall at 90.62% in both runs. A later four-accumulator run measured 225,340 ns/op, but that belongs to the unroll experiment rather than this before-and-after pair. The accepted path shipped with runtime F16C detection and a branchless AVX2 f32-to-E5M2 quantizer on the insert path. The represented finite values did not change. The load, widen, pack, and conversion sequence did.
That left three paths computing the same conversion in the source, and the question of what the two slower ones are still for.
What the slower paths are for
The slower paths keep the portability floor. Compile time decides which kernels the binary contains. At runtime, Astral selects the F16C path only when AVX state, AVX2, and F16C are all available; AVX2 builds without runtime F16C support keep the gather, and other builds use the scalar table.
The decoders disagree on four inputs. Shift codes 0x7C through 0x7F into FP16 and the hardware returns infinity or NaN; the table clamps them to plus or minus 57344. Astral's quantizer stops at 0x7B, so its own write path cannot produce the disagreement. Normalizing the fast path would add a compare and blend to every conversion for four unreachable codes, while changing the table would remove its finite-value guarantee. Neither shipped. Bytes from another writer remain the residual risk, and the decoder comment states that boundary.
Every path in that tree, F16C, gather, scalar, begins the same way: with loads. And I have been treating a load as free and uniform, which it is not. What a load costs depends on where the bytes sit.
The bytes around the instruction
The first arithmetic-conversion attempt lost because the instruction was never the whole cost. The kernel still had to load packed bytes, widen them into lanes, and keep each row and loop boundary valid. Changing that data movement is what turned the same conversion idea from a loss into the 225,704 nanosecond path.
An AVX2 load is 32 bytes wide. A 32-byte-aligned load cannot straddle a 64-byte cache line, while an arbitrary address can. Astral's 384-byte E5M2 rows are exactly six cache lines when they start on a line boundary, so the allocator aligns the row and the loop can issue twelve 32-byte steps without crossing a row or leaving a tail. The six-line and twelve-step counts are size arithmetic; I did not measure the isolated cost of a straddling load.
Other dimensions need an explicit tail policy. At 300 components, nine full steps cover 288 and leave 12. A kernel can finish those 12 scalarly, use a masked load, or pad the row. The current fixture avoids the choice because 384 divides by 32, but the kernel contract still states its stride and alignment instead of letting a future dimension silently drop values.
Cache-line ownership and LL/SC reservation granules are separate alignment problems. They are covered where they affect the actual design: cache-line layout for shared state and reservation alignment for synthesized atomics.
There is a third state between an instruction that exists and one that does not. The CPU has the instruction, and the toolchain reports it missing.
When the toolchain hides an instruction you have
The winning path was still absent from one Windows binary because its compiler exposed the F16C intrinsics without defining the feature macro the guard expected. One internal build macro fixed the selection. Tests had stayed green because the fallback returned identical values; only the missing speedup exposed the mistake.
The compiler fix and the runtime guard answer different questions. The build macro says the compiler may emit the intrinsics. Startup checks AVX, XSAVE, OSXSAVE, XMM/YMM state in XCR0, AVX2, and the separate F16C bit before selecting the kernel. A legal AVX2-without-F16C machine therefore reaches the gather fallback instead of executing an instruction it did not advertise.
The macro fix worked because the instruction was there all along. In low-level game-engine runtime work I also met the other case, where the silicon itself lacked the operation and the atomic had to be built from weaker ones; that is a separate post.
After conversion: one more bottleneck
Once the paired 10k query capture fell from 478,806 to 225,704 nanoseconds, the running sum became easier to see. A dot product with one accumulator is one dependency chain: each FMA waits for the previous FMA's result. The widening instructions are independent of that chain, so the loop can overlap more work by carrying several partial sums and reducing them once at the end.
The E5M2 kernels moved from two vector accumulators to four. On the 100,000-vector insert fixture, p95 fell from 1,926,399 to 1,899,105 nanoseconds and p99 from 2,079,024 to 2,048,914; p50 and recall did not move. A wider f32 loop measured as no change and was reverted. The SIMD post follows those dependency chains, the emitted loop, and both verdicts.
The acceptance gates made both decisions. Every lane had to hold at least 99% recall and keep add p99 below 600,000 nanoseconds on the scripted fixture. They admitted the F16C conversion with recall unchanged and rejected a narrower rerank width when one compact lane fell below the recall floor.
The remaining risks are specific. E5M2 moves in quarter steps between powers of two, so the f32 rerank keeps a second representation in memory to recover recall. The quantizer stops at 0x7B, which makes the four infinity and NaN codes unreachable only while that quantizer remains the sole writer. Compile-time feature availability and runtime CPU permission are separate checks, and both must admit the F16C path.
So the five conversion results belong to one Intel Core Ultra 7 265, GCC 13.3.0, Ubuntu 24.04, and the fixture named beside each result. A core with faster gathers or slower conversion can rank them differently. The slower paths remain in the tree, and the Compiler Explorer reproduction keeps the emitted conversion inspectable.