The snapshot tax_

Astral's model executor copies up to thirty-two conversation pointers at the start of each scheduling tick. Keeping that snapshot alive cost 10 nanoseconds with one active conversation and 243 with thirty-two, before the executor decoded a token.

The old bookkeeping used two reference-count updates per pointer. It was correct, but its cost grew with the number of conversations even though the executor needed one lifetime guarantee for the whole snapshot.

This article follows the replacement: one epoch announcement around the snapshot, deferred destruction outside it, and the race fixes required before that design could run in production. The isolated timings were measured on an Intel Core Ultra 7 265 with GCC 13.3 at -O3 -march=native, pinned to one core.

What the snapshot owes the destroyer

The problem is destruction racing use. The executor thread loads a conversation pointer from the slot table and dereferences it for the rest of the tick. Any other thread can call the destroy function for that conversation at any moment, because destruction comes from the embedding application, and the application does not know when the executor is mid-tick. Something has to keep the pointer alive from the load until the executor is done looking, or the destroy has to wait until nobody is looking. Every scheme for this is a way of answering one question: has the last reader finished?

The slot table itself is protected by a lock, and that lock is not the answer. The executor holds it only long enough to copy the pointers out, because holding it for the whole tick would block conversation creation and destruction behind decode work that can take milliseconds. So the snapshot exists precisely so the lock can be short, and the snapshot is what creates the lifetime question: the executor is now using pointers outside any lock.

The reference count answers the question per object. Snapshot acquisition walks the active slots and increments each conversation's counter; release walks them again and decrements. A destroyer that finds the counter nonzero waits. This is correct, it needs no scheme-wide coordination, and the cost model is easy to state: two atomic read-modify-writes per active slot per snapshot. With N active conversations, that is 2N locked operations before any useful work, and the executor takes a snapshot every tick, including the idle ticks, because idling is implemented as scanning for work.

Nothing in that design is wrong until you ask what the cost is proportional to. The executor asks "may I look?" once per tick; how many conversations exist at that moment is the workload's business, not the design's. Paying per object hands the safety bill to the workload, and the bill compounds with the tick rate, because a per-tick tax is a floor under everything the executor does.

Measuring the bill

To see the cost without the noise of real decode work, I wrote a standalone benchmark that isolates the lifetime bookkeeping: the reference path increments and decrements each active conversation's atomic counter around a slot scan, the way the executor did, and nothing else. Each configuration gets eleven measurements of two million snapshots each, and the reported numbers are medians across three process runs. It does not model provider decode time or the slot-table lock, so everything it measures is the snapshot overhead alone; whether the change moves end-to-end decode throughput is a different measurement, and this post does not make that claim.

The reference path compiles to exactly what the cost model predicted. The acquire loop:

15f0:  f0 83 00 01    lock add DWORD PTR [rax],0x1
15f4:  48 03 70 08    add    rsi,QWORD PTR [rax+0x8]
15f8:  48 83 c0 40    add    rax,0x40

One locked add per conversation, then the scan's own load, then step to the next slot, 0x40 bytes away. The stride is the tell: each conversation's counter sits on its own cache line, which is the right layout for a counter multiple threads touch, and it means the 2N locked operations also touch N distinct lines. The release loop is the mirror image, and the compiler unrolled it into pairs:

1620:  f0 83 2a 01    lock sub DWORD PTR [rdx],0x1
1624:  f0 83 6a 40 01 lock sub DWORD PTR [rdx+0x40],0x1

A locked read-modify-write on this core is on the order of twenty cycles when the line is already local, a number I measured from a different angle in the allocator post, and here the protocol issues 2N of them per snapshot. The arithmetic and the measurement agree: cost grows linearly in active slots, from 10.1 nanoseconds at one slot to 243.3 at thirty-two. The full table is in the results section, after the design that replaces it.

Paying per question instead

The executor's actual need is narrower than per-object accounting. There is only one executor thread, and it needs all of the snapshot's pointers alive for the same interval. The question "has the last reader finished?" has one answer for the whole snapshot, because the reader is the snapshot. A protocol that answers once per snapshot, at a cost independent of how many pointers the snapshot covers, fits the shape of the problem. That protocol is epoch-based reclamation.

Walk one pointer through it. A global epoch counter sits at 41. The executor, before touching the slot table, announces "I am reading in epoch 41" and copies its pointers out. A destroyer on another thread removes one conversation from the table and retires its pointer with the current epoch instead of freeing it. A collector advances the epoch and scans the announcements. The pointer stamped 41 stays untouchable while any announcement at or before 41 is still standing; once the collector proves a safe frontier past it, the retirement callback may run. The executor never touches a per-object counter. Astral's synchronous destroy API may still park until that callback reports quiescence, but the read-side bookkeeping remains one announcement for the whole snapshot.

A reader pays one announcement per critical section no matter how much it reads, which is the property the snapshot wants. What the scheme costs in exchange comes later, because I did not get the working version on the first try, and the failures are more instructive than the concept.

The manager I already had

An epoch manager already existed in astral's tree as a design sketch with a single-threaded test. Its documentation failed first: while writing the concurrency reference, I claimed that nothing could be freed early. Checking that sentence against the implementation falsified it three ways, and each failure forced part of the replacement design.

1. Retirement raced itself

The first break: retirement raced itself. The retirement ring's head cursor was updated with a plain load, a write to the slot, and a store of head plus one. Two threads retiring concurrently could both read the same head, both write the same slot, and both store the same new head: one retirement silently overwritten, a leak with no error path. I had already met this failure shape from the other side, in the queue that reported zero retries, and the fix there transfers: a shared cursor needs a reservation protocol, or there must be nothing shared. The redesign chose nothing shared. Each registered participant owns a fixed-size retirement queue as its single producer, the collector is the single consumer of all of them, and each entry carries its own epoch stamp and deleter, so no ring is tied to any epoch:

struct RetiredEntry {
  void*    ptr     = nullptr;
  Deleter  deleter = nullptr;
  uint64_t epoch   = 0;
};

2. Overflow freed too early

The second break: a full ring freed the pointer immediately. The overflow fallback called the deleter on the spot, bypassing the grace period entirely. The condition that triggers it, a retirement burst, is exactly the condition under which a reader is most likely to be inside a critical section holding the pointer, so the scheme's one job failed under load. The consequence for the redesign: overflow cannot have a silent fallback. Retirement now returns a bool, a full queue returns false, and the caller keeps ownership. What a caller does with a false became a visible policy decision, and the executor's own policy is part of the wiring section.

3. Three rings aliased after three epochs

The third break is pure arithmetic. The old design kept three rings, one per recent epoch, and the collector advanced the global epoch first, then drained the ring three epochs back. With three rings, three epochs back and right now are the same ring index: N minus three, modulo three, equals N modulo three. So the collector published a new epoch, every concurrent retirement started stamping into the ring for that new epoch, and the collector then walked that very ring freeing entries while they were being appended. The consequence: advancing the epoch and consuming retirements cannot alias, structurally. In the redesign, each collect first drains everything at or below the safe frontier it proved on the previous call, and only then advances the epoch and proves the next frontier from the announcements. A retirement that races the advance gets stamped with the new epoch, and the drain in progress only consumes entries at or below a frontier proven before that epoch existed.

None of the three could fire in the single-threaded test, because every one of them needs two threads interleaving. The redesign got the test the first version never had: destruction churn from four threads while the executor runs.

One more piece of the reader side needed care. Announcing an epoch has a classic race with the collector publishing a new one: read the global, get delayed, announce a stale epoch after the collector has already scanned you as absent. The entry protocol closes it by storing and then re-checking:

do {
  epoch = global_epoch_.load(std::memory_order_seq_cst);
  participant->epoch.store(epoch, std::memory_order_seq_cst);
} while (global_epoch_.load(std::memory_order_seq_cst) != epoch);

If the collector's publication won the race, the loop notices before the caller can load a protected pointer, and re-announces the newer epoch. Writes and reads crossing over two variables, with correctness depending on a total order over them: this is the same shape that justified the one strong ordering in the refcount post, and it takes seq_cst here for the same reason.

Wiring it into the executor

The executor instantiates the manager with two participants and a sixty-four-entry queue per participant. Participant one is the executor thread, the only reader: snapshot acquisition enters the epoch before taking the slot lock, release leaves it after the local copies are dropped, and the per-conversation increments, decrements, and destroy-side waiting are gone. Participant two is the retirement producer. Destruction can be called from any application thread, but every retirement happens while holding the slot-table lock, so the lock serializes all destroyers onto the single-producer contract, and the manager's assumption holds by construction.

Destruction needed one trick to stay synchronous, and it is the piece I would keep even if everything else changed. The API promises that when destroy returns, the conversation is gone, and a deferred-free scheme does not naturally promise that. So the retirement's deleter frees nothing: it flips a flag on the conversation and signals. The destroying thread, having retired the pointer, parks until the flag flips, then tears the conversation down itself, on its own thread. The epoch machinery serves purely as a quiescence barrier, an answer to "no reader can be holding this anymore," and the free stays exactly where the API promise needs it.

The full-queue path is the overflow policy the manager refused to invent. When retirement returns false, the destroyer drops the slot lock, waits for a wake, and retries. An exact count of outstanding retirements tells the executor to keep running collection until the queue drains, so the destroyer's wait keeps making progress as long as the executor is alive; it is progress under an assumption, not a time bound. A source-scan gate in the build now fails compilation if the old refcount field reappears in the executor or the conversation runtime, the same enforcement move as the allocation gate.

The numbers

The full sweep, both paths under identical conditions:

The table establishes two things: the refcount column grows linearly in active slots, matching the 2N cost model, and the epoch column does not move, 6.4 to 8.2 nanoseconds across a thirty-two-fold change in slot count. The question the flat line raises is where the work went.

Hardware counters answer from a different angle than the clock. A perf capture of the same binary, same pinning, with both paths running the identical schedule: 92.9% of user cycles land in the refcount path against 7.0% in the epoch path, a 13.2x split, where summing the two wall-clock columns across the sweep predicts 13.1x. Retired instructions split almost evenly, 52.4% against 47.6%. So the two paths execute nearly the same number of instructions, and one of them spends thirteen times the cycles on them; the difference is stalling, not extra work.

The disassembly shows where the stalling stops. The epoch entry compiles to the announce-and-recheck loop:

1678:  49 8b 00    mov  rax,QWORD PTR [r8]
167b:  48 89 c7    mov  rdi,rax
167e:  48 87 39    xchg QWORD PTR [rcx],rdi
1681:  49 8b 38    mov  rdi,QWORD PTR [r8]
1684:  48 39 f8    cmp  rax,rdi
1687:  75 ef       jne  1678

The seq_cst store became an xchg, x86's way of making a store globally ordered, and it is the only locked operation in the entire snapshot path. The slot scan after it is plain loads; leaving the critical section is a plain release store, an ordinary mov. The scheme did not remove atomic read-modify-writes from the design, it removed the multiplication: one xchg on the reader's own cache line, once per snapshot, where the refcount issued 2N locked operations across N lines. My reading of the flat line is that the xchg usually hits a line no other core is writing, so its cost does not depend on N; the near-equal instruction counts next to the 13x cycle split are consistent with that reading, though they do not pin the stall to a specific mechanism.

What the grace period costs

The refcount did some things better, and they are worth stating plainly.

It freed memory immediately. An epoch scheme frees when the collector has proven a frontier past the retirement, which takes multiple collect calls, and retired conversations sit in their queue meanwhile. For thirty-two slots' worth of conversation objects this is noise; for a scheme guarding millions of small nodes it would be a sizing decision, and the fixed sixty-four-entry queue would be the wrong shape.

It could not refuse. Retirement against a bounded queue can, and the false return pushes a policy decision onto every retirement site. The executor's policy, park and retry while collection drains, is right when destroys are occasional; a workload that retires in bursts would need bigger queues or a spill path, and the manager deliberately refuses to invent one, because the one it invented last time was the use-after-free.

It had no roles. Anyone could drop the last reference and free; in the epoch scheme, one thread collects, the design says so, and a flag enforces it. In the executor that role lands on the executor thread itself, between snapshots, where it can never overlap a read-side section of its own. A system with many independent readers would have to decide who collects and how often, and that decision does not exist in the refcount design at all.

Where this leaves the executor

The snapshot now costs the same whether one conversation is active or thirty-two, around 7 nanoseconds of bookkeeping instead of up to 243, with the caveat from the top still attached: that is the bookkeeping in isolation, and the end-to-end effect on decode is a measurement I have not made yet. The three races in the first manager are closed by structure rather than by care: queues with one producer each, a retirement contract that never frees on overflow, a collector that only consumes frontiers proven a full cycle earlier, and a churn test that destroys from four threads while the executor runs.

One edge stays open. Destruction after the executor has stopped assumes a live executor thread to run collection and flip the quiescence flag. Today the teardown order guarantees conversations die before their executor, so the path is unreachable, but that guarantee is a convention, not a check. The next change here is the check: destroy should fail loudly when the executor is gone, instead of waiting politely on a thread that no longer exists.