Stale on purpose_

Astral uses a single-producer, single-consumer ring to carry decoded tokens and stream metadata from the inference worker to the caller that drains the conversation. The ring is bounded, so a caller that falls behind creates backpressure without growing memory.

Every push must still answer one question: is there room? A naive producer reloads the consumer's cursor on every push, even when the answer has not changed.

Astral keeps a local copy instead. The producer owns the head cursor and caches the consumer's tail, while the consumer owns the tail and caches the producer's head. Because both cursors only move forward, an old copy can make the ring look fuller or emptier after the remote cursor advances, but it cannot invent room or work. When that local check fails, the thread refreshes the remote cursor and tries again.

The local fast path

// standalone illustration, clang++ -O2 -std=c++17 (x86-64), not the astral source
bool spsc_push(SpscRing& r, void* item) {
    uint64_t h    = r.head.load(std::memory_order_relaxed);
    uint64_t next = h + 1;
    if (next - r.cached_tail > r.capacity) {                  // check the local copy first
        r.cached_tail = r.tail.load(std::memory_order_acquire); // refresh only at the boundary
        if (next - r.cached_tail > r.capacity)
            return false;                                    // genuinely full
    }
    r.slots[h & (r.capacity - 1)] = item;
    r.head.store(next, std::memory_order_release);           // publish
    return true;
}

The standalone push loads the producer-owned head, compares next against the local cached_tail, and jumps straight to publication when the cached view proves there is room. The acquire load of the consumer-owned tail executes only when that first check fails.

; clang++ 18 -O2 (x86-64), standalone spsc_push
    mov     rcx, qword ptr [rdi]          ; h = producer-owned head
    lea     rax, [rcx + 1]                ; next = h + 1
    mov     r8, rax
    sub     r8, qword ptr [rdi + 8]       ; next - cached_tail
    mov     rdx, qword ptr [rdi + 88]     ; capacity
    cmp     r8, rdx
    jbe     .push                         ; cached view proves room
    mov     rdx, qword ptr [rdi + 64]     ; acquire-load remote tail
    mov     qword ptr [rdi + 8], rdx      ; refresh cached_tail
    ; subtract and compare again; return false if still full
.push:
    mov     r8, qword ptr [rdi + 80]      ; slots base
    dec     rdx                            ; capacity - 1
    and     rdx, rcx                      ; h & (capacity - 1)
    mov     qword ptr [r8 + 8*rdx], rsi   ; publish payload
    mov     qword ptr [rdi], rax          ; head.store(next, release)
    ret

Across one million items, the isolated loop sustained 0.48 nanoseconds per push-or-pop operation in the single-item case and 0.32 nanoseconds per item with batches of 64. These are amortized throughput values from the cursor loop, not single-call latency or full queue transit time. The consumed-value sum stays observable through do_not_optimize.

I checked the probe's disassembly because sub-nanosecond numbers are easy to misread. Its local path is ordinary loads, arithmetic, a compare, and publication, with no lock-prefixed instruction. A lock addl elsewhere in the binary belongs to harness bookkeeping, so the review had to inspect the surrounding symbol instead of counting lock prefixes.

Astral's production ring adds transition-sensitive wake logic around this cursor protocol. It signals when a push changes empty to nonempty and when a pop changes full to not full. On ARM, the transition check refreshes the remote cursor before deciding whether to signal, so a stale cached value cannot suppress the wake.

The cached cursor can only lag. A push may report full and refresh after the consumer has already made room, but it cannot overwrite an unread item. The ring capacity is a power of two, the cursors are unsigned 64-bit totals, and their distance is bounded by the ring capacity. Unsigned subtraction therefore remains unambiguous across wrap.