The conversion path astral's byte kernels ship on today was rejected on a measurement first. The trick, shift an 8-bit float into half-float position and let the hardware converter finish the job, lost its first benchmark to a 256-entry lookup table, 492,446 nanoseconds per query against the table's 482,963, and the verdict went into the log that day: the tiny table currently beats arithmetic conversion on this hardware. The same evening, the same trick shipped at 225,704, better than two to one against the table that had just beaten it. Nothing about the trick changed; the data movement around it did. Getting from the first verdict to the second took five attempts in one day, and all five are below, with their numbers.
Astral is my from-scratch C++ LLM inference runtime with an embedded vector index. The index is the memory part: you insert embedding vectors, you query with another vector, and it returns the nearest stored neighbors. Nearest-neighbor search at that scale means dot products in the hot loop. Every query walks a graph of stored vectors and scores candidates against the query at every step of the walk, so the dot product kernel is the floor the whole index stands on.
The broader runtime is mapped in A prompt through Astral. This post takes one much smaller path through it: the byte conversion under the vector index.
The index is built to hold millions of vectors in memory, and f32 vectors are heavy. My benchmark fixtures use 384-dimensional vectors, which is 1,536 bytes per vector as f32. A million of those is about 1.5 GB before any graph structure goes on top. Store the same vectors at one byte per component and a million of them fits under 400 MB. So astral has storage lanes: policies that decide how vectors are stored and how they are scored. The lane in this story stores every component as a single byte, in an 8-bit floating-point format called E5M2.
Scoring a stored vector means multiplying its components against the query's components and accumulating, and no instruction on my CPU knows what an E5M2 byte is. "x86 has no native E5M2" means something specific: AVX2 and F16C, the vector instruction sets astral's x86 path targets, have no primitive that loads 8 E5M2 values and dots them against 8 floats. There is not even an instruction that converts one E5M2 byte into one float. The format comes out of the ML accelerator world, and newer hardware does speak it natively, but the machine on my desk treats it as a byte with no meaning attached.
What an E5M2 byte is
E5M2 is an 8-bit floating-point format: 1 sign bit, 5 exponent bits, 2 mantissa bits, packed in that order from the top of the byte. In astral the constants are: sign shift 7, exponent shift 2, exponent bias 15, mantissa bits 2, max finite 57344.0f. The shifts are the field positions restated: the mantissa lives in bits 0 and 1, the exponent in bits 2 through 6, the sign in bit 7, so extracting a field is one shift and one mask. A normal E5M2 value decodes as
value = (-1)^sign * 2^(exponent - 15) * (1 + mantissa / 4)Take 0x3C and pull it apart. In binary it is 0 01111 00: sign 0, exponent 01111, which is 15, mantissa 00, which is 0. The formula gives 2^(15 - 15) * (1 + 0/4) = +1.0. So 0x3C is the value one.
Each field has one job. Flipping the sign of 0x3C produces 0xBC and -1.0. Incrementing the mantissa produces 0x3D and 1.25. Incrementing 0x3F, the code for 1.75, rolls the mantissa into the next exponent and produces 0x40, the code for 2.0. Positive finite codes therefore increase with their decoded values, a property the quantizer uses when it compares codes.
The quarter steps are the whole story of the format's precision. Two mantissa bits means four representable values per power of two, and nothing in between. Between 1 and 2 the format can say 1.0, 1.25, 1.5, 1.75. Between 2 and 4 it can say 2.0, 2.5, 3.0, 3.5. The pattern continues all the way up, so between 32768 and 65536 the representable values sit 8192 apart: 32768, 40960, 49152, 57344. That last value is the top of the format, and now the max finite constant explains itself: 57344 = 2^15 * 1.75, the biggest mantissa on the biggest normal exponent, which is the byte 0x7B = 0 11110 11. The four codes above it, 0x7C through 0x7F, have an all-ones exponent field, and in an IEEE-shaped format an all-ones exponent means infinity and NaN.
On the other end, the smallest normal value is 2^(1 - 15) = 2^-14, about 0.000061. So one byte covers roughly nine decimal orders of magnitude.
What the rough precision costs is visible the moment you store an ordinary number. Take 0.1. The nearest exponent is 2^-4 = 0.0625, and the four mantissa steps there are 0.0625, 0.078125, 0.09375, 0.109375. The value 0.1 falls between the last two, about 6% from one neighbor and 9% from the other, so the stored byte starts life several percent wrong whichever neighbor it lands on (illustrative arithmetic from the formula, not a measured astral error). A dot product then sums hundreds of these small errors, some canceling, some not. For steering a search toward the right neighborhood, that is fine. For deciding which of two close neighbors is truly closer, it is not.
It is fair to ask why a float byte at all, when int8 also fits in one. An int8 grid is uniform: 256 evenly spaced steps, so it needs a shared scale factor to place the grid over your data, and one outlier component stretches the grid for every component sharing the scale. E5M2 carries an exponent in every byte, so each component picks its own magnitude and an outlier costs only itself. The price is resolution: int8 spends all eight bits on positions inside the grid, E5M2 spends five of them on range. Astral ships both kinds of lane and lets the acceptance numbers rank them.
That last clause is why the lane exists in two forms. Recall is the fraction of the true nearest neighbors an approximate search actually returns, and I care about 99% recall as the default target. Score everything through quarter-step bytes and near neighbors blur together at the cutoff: the pure E5M2 lane measured 90.62% recall on my fixtures, which makes it a useful compact lane and rules it out as a 99% default. Keep the original f32 vectors alongside the bytes and the roles split: the bytes steer the graph walk, where being roughly right is enough, and the f32 copies rerank the survivors at the end, where being exactly right is the whole point. That paired lane reached 100% recall after I tuned the rerank scaling, and it pays for the number in memory and cache pressure, because both representations are now resident. So the byte does the approximate work and f32 does the exact ranking.
That split only works if converting bytes back to f32 costs almost nothing, because the conversion sits inside the innermost loop of every query. On a CPU with no E5M2 instructions, nothing about the conversion is cheap by default. Getting it cheap took me roughly a day.
What I tried, in order
The operation: take 8 stored E5M2 bytes and produce 8 f32 lanes, inside the innermost loop, for every candidate the graph walk touches. The lane has two dot kernels, one scoring stored bytes against an f32 query and one against a query quantized to the same byte format, and both stand on that one conversion. I built and measured five paths. The path I kept in step five had already been rejected on measurement in step three, and that is most of the reason to trust the rest of the numbers here. The protocol stayed constant across all five: the same machine and the same benchmark harness, with recall reported next to latency every time, so a fast path that broke recall would have been visible immediately.
One shortcut I ruled out before the first step: skip the conversion by scoring against stored f32 directly. Even the paired rerank lane, which has f32 copies sitting right there, steers the graph walk with the bytes and touches f32 only for the final rerank of the survivors, because the walk visits far more candidates than it keeps. Pull a 1,536-byte f32 row through the cache for every candidate visited and the cache pressure eats the win the compact format exists to buy. So the conversion has to happen at scoring time, and it has to be fast there.
1. A 256-entry table
Step one was a 256-entry lookup table, and it became the baseline. A byte has 256 possible values, so decode every one of them once, at startup, into an array of floats, and the conversion is one indexed load per byte. Even this baseline was not naive: the first scalar kernels already kept four partial sums running. The table is correct by construction: all 256 entries can be checked against the decode formula by hand. It is also small: 256 floats is 1 KB, sixteen cache lines, so after a few queries the whole table sits in L1 and the indexed load almost always hits (cache arithmetic, not a measured hit rate).
The table is doing more than converting, and every replacement has to be audited against each of its jobs:
1. It decodes: byte in, f32 out, per the formula. 2. It clamps: for the four codes 0x7C through 0x7F it does not answer what the format says, which is infinity and NaN. It answers +57344 or -57344, the max finite. Whatever byte arrives, the score stays finite and orderable. 3. It is the portability floor: it needs no vector instructions at all, so it runs on anything astral compiles for.
2. Gather the table
Step two vectorized the lookup. AVX2 gather fetches eight table entries into one register, after the eight bytes have been widened into 32-bit indices. The table still supplies the same decode and clamp. At 100k vectors, query time fell from 8,960,779 to 5,436,180 nanoseconds, 1.65x with recall unchanged. The gather still performs eight table reads per conversion, competing with vector data for cache, so the next attempts replaced those reads with arithmetic.
3. Shift into half-float position
Step three is where the trick enters, and to see it you need one more format on the table. FP16, the IEEE half float, is: 1 sign bit, 5 exponent bits, 10 mantissa bits. Same sign position, same exponent width, same bias 15 as E5M2. Write the two layouts next to each other:
E5M2: S EEEEE MM (8 bits)
FP16: S EEEEE MM00000000 (16 bits)E5M2 is basically the top 8 bits of a half-float-shaped value. So half_bits = e5m2_byte << 8 produces a genuine FP16 number with the same value, because the two extra mantissa bits come in as zeros. And x86 does have a hardware half-to-float decoder: F16C, an instruction set extension whose _mm256_cvtph_ps turns 8 half words into 8 f32 lanes in one instruction. Worked example, with the byte from the primer: 0x3C shifted left by 8 is 0x3C00, and FP16 0x3C00 is exactly 1.0. The instruction x86 does not have, decode E5M2, becomes one shift plus an instruction it does have.
The first implementation measured 492,446 nanoseconds per query at 90.62% recall, against 480,930 for the table it was meant to replace. Memory-mapped snapshot search also regressed. The shift was correct, but this implementation lost, so it was rejected.
4. Build f32 directly
Step four built f32 directly with masks and shifts: extract the fields, change the exponent bias from 15 to 127, and place each field in its f32 position. It removed the table and the half-float conversion. Query time rose from 482,963 to 584,678 nanoseconds with identical recall, about 21% slower. The table won on this machine, so the direct upcast was reverted.
5. Keep the shift, change the movement
Step five, the same evening, was the F16C trick again with the data movement done right:
// widen 8 E5M2 bytes into 8 f32 lanes
bytes = load_8_bytes(p); // one small load
lanes = _mm256_cvtepu8_epi32(bytes); // zero-extend each byte into its own 32-bit lane
lanes = _mm256_slli_epi32(lanes, 8); // each lane now reads S EEEEE MM00000000
halves = pack_to_16bit(lanes); // squeeze back down to 8 half words
floats = _mm256_cvtph_ps(halves); // the hardware half decoder does the rest-O3 -mavx2 -mf16c -mfma · open in compiler explorerTwo properties of that shape matter. The only memory it touches is the 8 bytes themselves, once; the gather touched the table eight more times per conversion. And everything between the load and the decode is register arithmetic the core can schedule as it likes. There was also plumbing, because the fast path only exists if the compiler is allowed to emit F16C at all: the GCC and Clang builds gained the -mf16c flag alongside the code.
The microbenchmark moved from 296 to 62 cycles per dot product: 4.8x on the kernel. End to end it shipped with a companion on the insert path, a branchless AVX2 f32-to-E5M2 quantizer, so that encoding vectors into bytes vectorizes the same way decoding them does. Query cost went from 478,806 to 225,704 ns/op, 2.12x, at recall unchanged: 90.62% before, 90.62% after. The unchanged recall is not luck, it is the check: every path decodes the same bytes to the same floats (four edge codes excepted), so a recall shift anywhere in these five steps would have meant a decode bug, not a tradeoff. Same trick as step three. What changed was the data movement.
That left three paths computing the same conversion in the source, and the question of what the two slower ones are still for.
What the slower paths are for
The slower paths keep the portability floor. Builds with F16C use the 4.8x conversion; AVX2 builds without it keep the 1.65x gather; other builds use the scalar table. This selection is still made at compile time. The index runtime-dispatches its VNNI kernel through CPUID, but the conversion tree does not yet use that mechanism.
The decoders disagree on four inputs. Shift codes 0x7C through 0x7F into FP16 and the hardware returns infinity or NaN; the table clamps them to plus or minus 57344. Astral's quantizer stops at 0x7B, so its own write path cannot produce the disagreement. Normalizing the fast path would add a compare and blend to every conversion for four unreachable codes, while changing the table would remove its finite-value guarantee. Neither shipped. Bytes from another writer remain the residual risk, and the decoder comment states that boundary.
Every path in that tree, F16C, gather, scalar, begins the same way: with loads. And I have been treating a load as free and uniform, which it is not. What a load costs depends on where the bytes sit.
The bytes around the instruction
The first arithmetic-conversion attempt lost because the instruction was never the whole cost. The kernel still had to load packed bytes, widen them into lanes, and keep each row and loop boundary valid. Changing that data movement is what turned the same conversion idea from a loss into the 225,704 nanosecond path.
An AVX2 load is 32 bytes wide. A 32-byte-aligned load cannot straddle a 64-byte cache line, while an arbitrary address can. Astral's 384-byte E5M2 rows are exactly six cache lines when they start on a line boundary, so the allocator aligns the row and the loop can issue twelve 32-byte steps without crossing a row or leaving a tail. The six-line and twelve-step counts are size arithmetic; I did not measure the isolated cost of a straddling load.
Other dimensions need an explicit tail policy. At 300 components, nine full steps cover 288 and leave 12. A kernel can finish those 12 scalarly, use a masked load, or pad the row. The current fixture avoids the choice because 384 divides by 32, but the kernel contract still states its stride and alignment instead of letting a future dimension silently drop values.
Cache-line ownership and LL/SC reservation granules are separate alignment problems. They are covered where they affect the actual design: cache-line layout for shared state and reservation alignment for synthesized atomics.
There is a third state between an instruction that exists and one that does not. The CPU has the instruction, and the toolchain reports it missing.
When the toolchain hides an instruction you have
The winning path was still absent from one Windows binary because its compiler exposed the F16C intrinsics without defining the feature macro the guard expected. One internal build macro fixed the selection. Tests had stayed green because the fallback returned identical values; only the missing speedup exposed the mistake.
The remaining issue is separate: AVX2 and F16C have different CPUID bits, while this path still trusts the build target. A legal AVX2-without-F16C machine would fault instead of reaching the fallback. The VNNI kernel already uses runtime dispatch; this path does not yet.
The macro fix worked because the instruction was there all along. In low-level game-engine runtime work I also met the other case, where the silicon itself lacked the operation and the atomic had to be built from weaker ones; that is a separate post.
After conversion: one more bottleneck
Once conversion fell from 492,446 to 225,704 nanoseconds, the running sum became easier to see. A dot product with one accumulator is one dependency chain: each FMA waits for the previous FMA's result. The widening instructions are independent of that chain, so the loop can overlap more work by carrying several partial sums and reducing them once at the end.
The E5M2 kernels moved from two vector accumulators to four. On the 100,000-vector insert fixture, p95 fell from 1,926,399 to 1,899,105 nanoseconds and p99 from 2,079,024 to 2,048,914; p50 and recall did not move. A wider f32 loop measured as no change and was reverted. The SIMD post follows those dependency chains, the emitted loop, and both verdicts.
The acceptance gates made both decisions. Every lane had to hold at least 99% recall and keep add p99 below 600,000 nanoseconds on the scripted fixture. They admitted the F16C conversion with recall unchanged and rejected a narrower rerank width when one compact lane fell below the recall floor.
The remaining risks are specific. E5M2 moves in quarter steps between powers of two, so the f32 rerank keeps a second representation in memory to recover recall. The quantizer stops at 0x7B, which makes the four infinity and NaN codes unreachable only while that quantizer remains the sole writer. The build also treats F16C as part of the AVX2 baseline even though the CPU reports them separately.
So the five conversion results belong to one Intel Core Ultra 7 265, GCC 13.3.0, Ubuntu 24.04, and this row layout. A core with faster gathers or slower conversion can rank them differently. The slower paths remain in the tree, the Compiler Explorer reproduction keeps the emitted conversion inspectable, and the runtime-dispatch gap remains open.