The snapshot tax_

With thirty-two active conversations, Astral spent 243.3 nanoseconds on bookkeeping before decoding a token. One epoch announcement reduced that to 8.2 nanoseconds. The executor copied up to thirty-two conversation pointers at the start of every scheduling tick, and the per-pointer refcount protocol was correct. Its cost grew with the number of pointers copied into each snapshot even though I needed one lifetime guarantee for the whole snapshot.

I replaced those per-pointer updates with one epoch announcement around the snapshot. Astral already had an epoch manager, but its retirement and collection code needed changes before the executor could use it.

The isolated timings were measured on an Intel Core Ultra 7 265 with GCC 13.3 at -O3 -march=native, pinned to one core.

The refcounted snapshot

The race is between destruction and use. The executor loads a conversation pointer from the slot table and keeps using it for the rest of the tick. Another application thread can destroy that conversation at any moment. It does not know whether the executor is halfway through a tick.

The pointer has to stay alive until the executor finishes with it. Destruction can proceed only after the last reader has left.

I held the slot-table lock only long enough to copy the pointers. Keeping it for the whole tick would block creation and destruction behind decode work that can take milliseconds.

The short lock is why the snapshot exists. It is also why I had a lifetime problem. The executor uses those copied pointers after releasing the lock.

My reference count answered the lifetime question per object. Snapshot acquisition incremented every active conversation's counter. Release walked them again and decremented each one. A destroyer waited when it found a nonzero counter.

That gave me simple correctness with no scheme-wide coordination. It also gave me two atomic read-modify-writes per active slot. With N conversations, the executor performed 2N locked operations before useful work. It took a snapshot on every tick, including idle ticks that only scanned for work.

The executor needed one lifetime guarantee per tick, but refcounting paid once per pointer.

Measuring the bookkeeping

I wrote a standalone benchmark so decode work could not hide the cost. The reference path increments and decrements each active conversation's counter around a slot scan, exactly as the executor did, and nothing else.

Each configuration gets eleven measurements of two million snapshots. I report the median across three process launches. The benchmark does not model provider decode time or the slot-table lock. It measures snapshot bookkeeping, not end-to-end decode throughput.

The acquire loop contains one locked add per conversation:

15f0:  f0 83 00 01    lock add DWORD PTR [rax],0x1
15f4:  48 03 70 08    add    rsi,QWORD PTR [rax+0x8]
15f8:  48 83 c0 40    add    rax,0x40

The loop performs one locked add per conversation, loads the scan value, then steps to the next slot 0x40 bytes away. That stride matters. Each counter sits on its own cache line, which is the right layout for a counter touched by several threads. It also means the 2N locked operations touch N distinct lines.

The release loop is the mirror image. The compiler unrolled it into pairs:

1620:  f0 83 2a 01    lock sub DWORD PTR [rdx],0x1
1624:  f0 83 6a 40 01 lock sub DWORD PTR [rdx+0x40],0x1

I had already measured a locked read-modify-write on this core at roughly twenty cycles when the line was local in the allocator work. This protocol issued 2N of them per snapshot.

The model predicts two locked refcount operations per pointer, or 64 at thirty-two slots. The measured slope was about 7.5 nanoseconds per additional pointer, close to the expected cost of two local locked operations on this machine. The benchmark recorded elapsed time rather than a fixed core frequency, so this is a consistency check rather than a cycle-accurate model.

One epoch for the whole snapshot

One executor thread reads every pointer in the snapshot during the same interval. Epoch-based reclamation lets it announce that interval once for the whole snapshot.

Suppose the global epoch is 41. Before touching the slot table, the executor announces that it is reading in epoch 41 and copies the pointers. Another thread can remove a conversation from the table, but it retires the pointer with epoch 41 rather than freeing it.

The collector advances the epoch and scans the reader announcements. It cannot reclaim the pointer while any reader remains at epoch 41 or earlier. Once the collector proves a safe frontier past 41, the retirement callback can run.

The executor never updates a per-object counter. Astral's synchronous destroy API may still wait for the callback, but the reader pays for one announcement per snapshot.

Preparing the epoch manager

The executor did not use Astral's existing epoch manager. Its retirement and collection paths needed concurrency fixes first.

Before the executor could use it, I found races in retirement, an overflow path that skipped the grace period, and collection draining a ring while producers still wrote to it.

1. Retirement raced itself

The retirement ring updated its head with a plain load, a slot write, and a store of head plus one. Two retiring threads could read the same head, write the same slot, and publish the same next value. One retirement disappeared and leaked.

A shared cursor needs a reservation protocol. I removed the sharing instead. Each participant is the only producer of its fixed-size retirement queue, and the collector is the only consumer. Each entry carries its own epoch and deleter. The queue that reported zero retries covers the related reservation problem.

struct RetiredEntry {
  void*    ptr     = nullptr;
  Deleter  deleter = nullptr;
  uint64_t epoch   = 0;
};

2. Overflow freed too early

A full ring called the deleter immediately. It skipped the grace period when retirements arrived in a burst, which is also when readers are likely to be active. The manager failed its one job under load.

I removed the hidden fallback. Retirement now returns a bool. A full queue returns false and leaves ownership with the caller. Overflow policy is visible at the call site instead of silently freeing protected memory.

3. Three rings aliased after three epochs

The third failure was arithmetic. The old manager kept three rings. It advanced the global epoch, then drained the ring from three epochs ago. With three rings, those are the same index: N minus three, modulo three, equals N modulo three.

I separated publication from draining by one collection cycle. Each call drains the frontier proved by the previous call, then publishes and proves the next. A retirement racing the advance receives the new epoch and cannot enter the frontier currently being consumed.

One more piece of the reader side needed care. Announcing an epoch has a classic race with the collector publishing a new one: read the global, get delayed, announce a stale epoch after the collector has already scanned you as absent. The entry protocol closes it by storing and then re-checking:

do {
  epoch = global_epoch_.load(std::memory_order_seq_cst);
  participant->epoch.store(epoch, std::memory_order_seq_cst);
} while (global_epoch_.load(std::memory_order_seq_cst) != epoch);

If the collector publishes first, the loop notices before the caller can load a protected pointer and announces the newer epoch. Correctness depends on a total order across two variables, so this path uses seq_cst. The refcount work covers the same ordering requirement in detail.

How I wired it into the executor

I gave the executor two epoch participants and a sixty-four-entry queue for each. The first participant is the executor thread, which is the only reader. It enters the epoch before taking the slot lock and leaves after dropping the copied pointers. The per-conversation increments and decrements are gone.

The second participant is the retirement producer. Any application thread may call destroy, but retirement always happens while holding the slot-table lock. That lock serializes every destroyer onto the queue's single-producer contract. The ownership rule is enforced by the surrounding design rather than left as a comment.

I still wanted destruction to remain synchronous. The API promises that the conversation is gone when destroy returns. Deferred reclamation does not naturally provide that promise.

I made the retirement callback free nothing. It flips a flag and signals. The destroying thread parks until the flag changes, then tears down the conversation on its own thread. I use the epoch manager only as a quiescence barrier. It answers "no reader can still hold this," while the actual free remains where the API contract needs it.

A full queue returns false. The destroyer drops the slot lock, waits for a wake, and retries. An exact outstanding-retirement count tells the executor to keep collecting until the queue drains. The wait makes progress while the executor is alive, but it does not have a time bound.

Snapshot cost

I ran both paths under identical conditions:

The refcount column follows the 2N cost model. The epoch column stays between 6.4 and 8.2 nanoseconds while the active-slot count grows thirty-two-fold.

I then used hardware counters to see where the difference came from.

With the same binary, core pinning, and schedule, perf attributed 92.9% of measured user cycles to the refcount path and 7.0% to the epoch path, a 13.2x split. Summing the timing columns predicts 13.1x.

Retired instructions split almost evenly at 52.4% and 47.6%. The paths execute nearly the same number of instructions, but one spends thirteen times the cycles. The difference is stalls, not extra instructions.

The disassembly shows where the stalling stops. The epoch entry compiles to the announce-and-recheck loop:

1678:  49 8b 00    mov  rax,QWORD PTR [r8]
167b:  48 89 c7    mov  rdi,rax
167e:  48 87 39    xchg QWORD PTR [rcx],rdi
1681:  49 8b 38    mov  rdi,QWORD PTR [r8]
1684:  48 39 f8    cmp  rax,rdi
1687:  75 ef       jne  1678

The seq_cst store became xchg, x86's globally ordered store. It is the only locked operation in the snapshot path. The slot scan uses plain loads, and leaving the critical section is an ordinary mov.

I replaced 2N locked operations across N lines with one xchg on the reader's own cache line. My reading is that the line is normally uncontended, so its cost does not depend on N. The nearly equal instruction counts and 13x cycle split support that explanation, but they do not identify a specific stall mechanism.

Costs of deferred reclamation

Refcounts free memory immediately. The epoch manager waits several collection calls until it proves a safe frontier, so retired conversations stay queued for longer. That retained memory is small for thirty-two slots. A system protecting millions of nodes would need different queues and explicit sizing.

A bounded retirement queue can be full. Every retirement site needs a policy for that result. The executor parks and retries while collection drains, which fits occasional destruction. A burst-heavy workload would need larger queues or a spill path.

Epoch reclamation also needs a collector. The executor collects between snapshots, where collection cannot overlap its own read section. A system with many independent readers would need to decide who collects and how often.

Conversations are currently destroyed before the executor stops. If that order changes, destruction should return a lifecycle error instead of waiting for a collector thread that no longer exists.