One memory-order change in a reference-count decrement made two ARM machines 37% and 39% faster. The same source change made x86 exactly 0% faster because the compiler emitted the same binary before and after.
The code came from a cross-platform primitives library used across game-runtime targets. Those machines do not pay the same price for acquire, release, and sequential consistency, even when the C++ operation looks identical.
The useful question is which ordering the correctness argument actually needs. The answer differs by operation and by path through an operation.
Ordering controls surrounding memory
An atomic variable always has one modification order that every thread agrees on. Ten threads can each increment a relaxed counter one million times and still finish at exactly ten million.
The ordering argument governs the ordinary loads and stores around the atomic, including when another thread may rely on seeing them. A relaxed operation still updates the atomic variable indivisibly and keeps that variable's modification order consistent. It promises nothing about nearby memory.
Both the compiler and CPU may reorder work around a relaxed operation. The compiler schedules instructions to keep registers busy. The CPU can drain stores late, start loads early, and expose effects to other cores in a different order. Relaxed permits both.
Relaxed is enough when the value is read for its own sake and says nothing about other memory. A counter of slow-path calls, read after every thread has joined, is a clean example. No thread uses that number to decide that a buffer is ready or an object is dead. Atomicity keeps the count exact. On the ARM core above, one million relaxed increments cost 42 ms, or 42 nanoseconds per increment. Every other measurement in this post starts there.
Relaxed stops being enough when another thread treats the atomic as a signal about nearby memory. One thread fills a buffer and sets a flag. Another sees the flag and reads the buffer. With relaxed operations, the flag may become visible before the buffer writes. Source order alone does not make that inference safe.
Publish and subscribe
A release store publishes. Every write before it becomes visible to a thread that reads the stored value with an acquire load. Once that acquire reads the released value, the consumer can use the preceding writes. The relationship is between two operations through one value:
// producer // consumer
payload = build(); while (!ready.load(acquire)) { }
ready.store(true, release); use(payload);The consumer exits only after reading the value written by the release store. That release and acquire synchronize. The payload written before release is visible after acquire.
Two boundaries of that guarantee matter later. First, the pairing happens through a value, not merely because two operations touch the same variable. An acquire can synchronize with the release that supplied the value it read, including through the release-sequence rules for atomic read-modify-writes. A value written only by a relaxed store does not create that edge. Second, release constrains order, not timing. It does not flush data or make another core observe the value by a deadline.
A release-acquire pair is a handoff through the value the acquire actually reads. It does not create a general ordering among unrelated atomics.
Splitting the refcount barrier
A refcounted object has one atomic counter. Every holder decrements it on the way out. The thread that changes the count to zero runs the destructor. That decrement is hot because it runs once for every released reference.
A read-modify-write can apply acquire to its read, release to its write, or both. On the ARM core measured later in the article, acquire-release cost 50% more than relaxed.
The library splits the decrement because its two outcomes need different guarantees.
A thread that decrements from five to four will not read the object again. It needs release so its earlier writes become visible to the eventual destroyer. It does not need acquire because it performs no later reads through the object.
The thread that decrements from one to zero is different. Its destructor may read fields written by every earlier holder before freeing the memory. That path needs acquire to observe their release operations.
The code follows those two paths with the standard release-decrement pattern. Every atomic read-modify-write participates in the counter's modification order. When the final fetch_sub reads the value that makes it the destroyer, the acquire fence after that read-modify-write synchronizes with the earlier release sequence before destruction begins:
if (count.fetch_sub(1, release) == 1) {
atomic_thread_fence(acquire);
destroy(object);
}The acquire now runs on one thread per object, the one about to read. The common-path decrements skip it, because they read nothing through the object.
On two ARM cores, the split decrement was 37% and 39% faster than acquire-release. On x86 both versions compiled to the same locked decrement, and the acquire fence emitted nothing. The result was exactly 0%:
; x86-64: the ordering is free on the RMW; identical instruction either way
dec_release: lock dec dword ptr [count] ; release
dec_acqrel: lock dec dword ptr [count] ; acq_rel: byte-for-byte the sameOn the weakly ordered core the acquire is a real, separate bit of the opcode, one a between the release-only form and the acquire-release form, which is exactly the guarantee the common path is dropping:
; ARM64 (Apple Silicon, LSE): the acquire is one letter of real work
dec_release: ldaddl w9, w8, [count] ; load-add, release
dec_acqrel: ldaddal w9, w8, [count] ; load-add, acquire + releaseThe cost of a wrong argument
The split is correct only while the common path publishes without reading and the zero path performs the acquire before destruction.
A later refactor could read the object after the decrement without revisiting that argument. x86 would still supply acquire behavior through the locked instruction, so its tests could keep passing.
On a weakly ordered core, the missing acquire is really missing, and the failure is the destructor racing another holder's last writes: a use-after-free that needs a narrow reordering window to open, on the zero path, under contention.
This bug would cluster on weakly ordered targets that receive less testing. Stress tests may miss the narrow reordering, while x86 tests receive a stronger hardware guarantee. The library shipped the split because its ordering argument was short enough to review and the decrement was hot enough to justify it. A model or litmus test can check a bounded version of that argument.
Sequential consistency adds one total order
Sequential consistency is acquire-release plus one property: a single total order over all the seq_cst operations in the program, one order, agreed by every thread.
Acquire-release does not provide that, and the gap shows in a four-line experiment. Two threads, two flags, both initially false. Each thread raises its own flag, then checks the other's:
// thread A // thread B
a.store(true, release); b.store(true, release);
saw_b = b.load(acquire); saw_a = a.load(acquire);With acquire-release, both threads may read false. Each load read the initial value, so neither synchronized with the other thread's store. With seq_cst, the four operations must fit one agreed order. There is no valid order where each store precedes its own load while both loads precede the stores they missed. At least one thread must see the other's flag.
Flag-based mutual exclusion can depend on that agreed order. Most synchronization in this library uses pairwise publication through one atomic value. Sequential consistency earns its cost only where the argument needs one total order across otherwise independent atomic operations.
Measured cost by memory ordering
One million atomic increments on one 64-bit ARM core, the same loop every time, only the ordering argument changed. The increment is the right operation for this experiment because it is a read-modify-write, so it can take every rung:
| ordering | one million increments |
|---|---|
| relaxed | 42 ms, baseline |
| acquire | +14% |
| release | +17% |
| acquire-release | +50% |
| sequential consistency | +86% |
On this core, acquire added 14%, release added 17%, acquire-release added 50%, and sequential consistency added 86% over relaxed. The acquire-release result is an independent measurement, not the sum of the previous two. The table implies about 78 ms for the sequentially consistent run. The loop and final count stayed the same. These percentages belong to this compiler, operation, and core. Another target may show a different order or no measurable separation.
x86 produces a different result because several source-level orderings compile to the same instructions.
The machine with the guarantees bundled in
On x86-64, aligned atomic acquire loads and release stores commonly compile to plain mov instructions because TSO already supplies those ordering properties. Cache state and surrounding dependencies still dominate latency. In the compiler output below, the ordering argument changes no instruction for those two operations, but it records what the source relies on.
The same three operations, compiled for each machine, show the two cost models directly. On x86 the release and acquire are plain moves and only seq_cst pays, with a locked exchange:
; x86-64 (clang-18 -O2): the ordering rides in the instruction choice
store_release: mov dword ptr [g], edi ; plain store; TSO already orders it
load_acquire: mov eax, dword ptr [g] ; plain load
store_seqcst: xchg dword ptr [g], edi ; locked exchange, the seq_cst taxOn the ARM core the ordering is encoded in the opcode instead. (These are the instructions this compiler picks for Apple Silicon. A generic AArch64 target often shows ldar for the acquire load, and older cores without the atomics extension use load-linked / store-conditional loops.)
; ARM64 (clang, Apple Silicon: LSE + LRCPC): the ordering rides in the opcode
store_release: stlr w0, [x8] ; store-release
load_acquire: ldapr w0, [x8] ; load-acquire (RCpc form)
store_seqcst: stlr w0, [x8] ; same opcode as release on this targetThe store-buffer outcome is the one exercised by the two-flags test. On x86, a standalone full fence or a locked operation can participate in preventing that outcome, but the exact instruction and cost depend on the source operation and compiler. The roughly 100-cycle fence and 15-cycle locked-operation figures below are measurements from the tested machine, not ISA constants.
On this ARM core, stronger ordering changes both the instruction and measured time. On x86, acquire loads and release stores remain ordinary moves, while seq_cst may need a locked instruction or fence. The stronger x86 behavior can hide an ordering bug until the code reaches a weaker machine.
The combination the API refuses
Compare-exchange takes an expected value and a desired value. On success, it installs the desired value. On failure, it writes the current value back into expected. Retry loops use that fresh value for the next attempt.
The loop is worth one concrete pass. To push a node onto a lock-free list, you read the current head, write it into your node's next pointer, and compare-exchange the head from that value to your node. Success means your node is installed and the list is intact behind it. Failure means someone else pushed first, and the fresh head the operation handed back is exactly what the next attempt needs.
C++ accepts separate orderings for success and failure. Success is a read-modify-write. Failure is only a load and cannot use release. A common pairing is acquire-release on success and acquire on failure.
The library forbids release on success with acquire on failure, even though C++ allows it. A failed compare-exchange synchronizes only if the value it reads came from a release operation. If the winning writer used relaxed, acquire on failure synchronizes with nothing. The names can make that condition easy to miss.
The success half does not rescue the belief either, because release on success carries no acquire: the winning path, the one about to act on what it displaced, has subscribed to nothing by construction.
The library's compare-exchange does not accept that combination. C++ permits it, but the acquire failure ordering is easy to misread as synchronization with whichever thread won. Ordinary x86 testing will not expose the missing guarantee, and weak-core stress testing may still miss the execution. A caller who needs the asymmetry can write the operations and fences at the call site, beside the reasoning that supports them.
The review rule
One sentence in the shared header made the review rule explicit: never use sequential consistency without a written justification.
A justification for seq_cst has to identify the independent operations whose order every thread must agree on. If the code never consumes that total order, acquire and release are enough.
When the code moves, the sentence naming the property is sitting next to the operation, and whoever touches it can check whether the place it points at still exists.
The rule outlived the library. Astral's production epoch manager uses a total-order handshake rather than one isolated strong operation. Reader entry loads the global epoch, stores that value into its participant slot, and reloads the global epoch. All three operations are seq_cst. The collector publishes the next global epoch with a seq_cst increment. If publication wins the race, the reader sees the changed epoch and retries before loading a protected pointer. Otherwise the collector's later scan must see the announcement or conservatively delay reclamation. A cold unregister also stores the inactive epoch with seq_cst before its slot can be reused. The retirement queues and ordinary participant scans stay on acquire and release.
When synchronization code changes, its ordering argument must be checked again. A bounded weak-memory model or architecture litmus test can exercise that pattern more directly than an application stress run.