A game sends Astral one chat turn. Before a model can answer, that request crosses an engine wrapper, a C ABI, native prompt state, a bounded executor slot, a backend provider, sampling, and a byte stream back to the frame loop.
The runtime has to decide where the bytes and history live, which thread may cancel, what happens when every decode slot is busy, and how output returns without allocating or calling back for every token.
Astral is a C++17 inference runtime for Unity, Unreal, and native applications. It handles scheduling, memory, conversation state, and streaming around a model provider.
Start with one engine call
Suppose an Unreal character has a native Astral agent handle. The game sends one user turn: "What did you see near the bridge?" The wrapper constructs an AstralSpanU8, points it at the UTF-8 bytes, and calls astral_agent_chat_enqueue.
The span is borrowed for one call.
Astral copies the user text into native storage before returning. Copying the borrowed span before the call returns prevents later reads from the caller's temporary message buffer.
typedef struct AstralSpanU8 {
const uint8_t* data;
uint32_t len;
uint32_t _padding;
} AstralSpanU8;
AstralAgentChatDesc chat = {0};
chat.size = sizeof(chat);
chat.user_message = prompt;
AstralErr err = astral_agent_chat_enqueue(agent, &chat);Astral exposes the runtime through a C ABI. Calls use plain-data structs and 64-bit tagged handles, return status through out-parameters, and never allow C++ exceptions across the boundary.
Extensible configuration and descriptor structs carry their byte size. Fixed spans and small value types keep fixed layouts. Compile-time assertions catch accidental ABI drift on 32-bit and 64-bit targets.
Astral is still pre-1.0. The C API is designed for long-lived compatibility, but documented breaking changes can still happen before 1.0 and will include matching binding updates.
The agent owns the conversation
A native agent stores the system prompt, rolling summary, retrieved memory context, chat history, optional toolset, prompt cache, and the conversation handle used for active decode. Unity and Unreal wrappers remain thin owned handles over this state. The character can outlive a scene object, change engine wrappers, or sit idle for minutes without moving its conversational state into managed collections.
On enqueue, the agent assembles one bounded prompt in a defined order:
- system prompt
- rolling summary
- retrieved memory context
- prior messages
- the current user turn
- the assistant prefix.
For the bridge question, suppose the agent's stable prefix contains the character rules, a short summary of the previous scene, and a retrieved memory saying that Mara saw torchlight near the bridge. The new suffix contains only "What did you see near the bridge?" plus the assistant marker. This is an illustrative conversation passed through the implemented ownership rules, not a recorded game session.
History uses small POD records that point into one agent-owned byte arena. The agent caches the stable prefix after its system prompt, summary, memory, or history changes. It appends only the current suffix for the next turn. With a prompt cache bound, a stable-prefix hit can reuse cached tokens and tokenize only the new part.
The "reused prompt tokens" counter measures native prompt setup. Backend KV-prefix reuse is a separate optimization and needs separate measurement.
max_messages and max_prompt_bytes bound the agent. By default an oversized turn is rejected. A caller can instead discard the oldest history until the prompt fits. The choice is made before decoding starts.
Idle agents do not use decode slots
A game may have hundreds of characters but only a few active conversations. Astral separates an agent's durable state from a model executor slot. An idle agent keeps its prompts and history without occupying KV-backed decode capacity, so many characters can share a smaller executor.
The first chat request acquires a conversation slot from the model-scoped executor. If every slot is busy, enqueue may reclaim a completed, canceled, failed, or idle agent slot, but only after its stream bytes have been drained. Otherwise the call returns ASTRAL_E_BUSY. Unread output is part of ownership, so a "finished" producer is not automatically a reusable slot.
Assume the bridge agent acquires slot zero. Its native prompt tokens now belong to that conversation. The agent handle still owns Mara's history. Slot zero owns only the active decode state. When the answer finishes and its stream is drained, Astral can release the slot without deleting what Mara remembers.
Slot affinity is optional. A caller can prefer a one-based slot when backend locality matters. Affinity does not permit stealing a live conversation. If that slot cannot be safely reclaimed, enqueue stays busy.
The executor turns conversations into one batch
The executor belongs to a model because batching only makes sense across requests that share that model and provider context. It scans active slots, adds bounded prompt work and decode work to a provider batch, evaluates that batch, then maps each logits output back to the conversation that requested it.
Prompt ingestion has a per-slot cap per tick. Without it, one long new prompt could fill a batch repeatedly while already-decoding conversations wait for their next token. The cap limits how much prompt work one rotation accepts. A latency bound would also need provider evaluation time, batch size, active-slot count, and cancellation behavior.
Each active model executor owns one dedicated thread, separate from Astral's bounded worker pool. Provider session state, slot masks, and batch buffers stay single-owner there, so the control path does not pay for several threads contending over the same per-token state. Parallelism still exists across model executors and inside backend evaluation, while embeddings, memory-index work, and bounded runtime jobs retain the full configured worker budget.
Provider execution and sampling
Backends implement an operations table that engine code never sees. A local logits provider loads the model, tokenizes input, keeps session and KV state, evaluates tokens, and exposes a read-only logits view. Astral applies penalties and grammar constraints, samples the next token, detokenizes it, and sends the UTF-8 bytes toward the engine. A remote provider keeps the same handles and streaming API while handling its own transport.
On one turn of the worked trace, the provider returns logits for slot zero. Astral applies Mara's grammar and sampling configuration, selects the next token, and detokenizes it. If the piece is the byte I, that byte goes into slot zero's stream ring while the chosen token returns to model evaluation on the next executor tick.
For logits providers, Astral keeps grammar, stop rules, log probabilities, tools, and sampling in the core. The game can move between local CPU execution, optional CUDA offload, deterministic mock runs, and remote transport while keeping the same handles, cancellation rules, and stream consumer.
The built-in registry includes CPU, mock, and remote providers. CUDA offload is optional at build time. The engine-facing handles and stream calls stay the same when the provider changes.
The engine reads output when it is ready
Generated bytes land in a bounded single-producer, single-consumer ring. The decode worker is the producer. One engine-side consumer calls astral_agent_chat_stream_read or the lower-level conversation/session equivalent with a caller-owned buffer.
The engine polls with a zero timeout during Update or a tick, fills its own buffer up to the UI budget, and returns to the frame. This requires no callback onto an arbitrary game thread and no managed allocation for each token. A positive result is a byte count, zero is end of stream, and a negative value carries an error such as timeout.
The engine therefore receives the bridge answer as ordinary UTF-8 fragments in its own buffer. It may reveal "I saw torchlight..." over several frames without moving prompt assembly, history mutation, or model scheduling onto the game thread.
The ring also makes backpressure real. If the consumer stops draining, the producer cannot overwrite unread output. The conversation becomes safely reclaimable only after those bytes have an owner. Slot release therefore checks stream state. A stopped model can still have a live request.
The lower-level concurrency work is covered in The weakest ordering that is correct. The state-transfer cost behind long-lived sessions is in The snapshot tax.
Memory is assigned before the hot path
Astral reserves and allocates control structures when the runtime, model, conversation, or agent is created. Conversation prompt tokens, sampling candidates, and stream storage are bounded buffers. Agent history uses an arena. The decode loop reuses those structures. The memory-index path follows the same discipline with fixed-capacity workspaces and architecture-dispatched kernels. Each query reuses explicit scratch storage.
Model loading, agent creation, prompt replacement, and other control operations may allocate. Steady-state decode, sampling, and streaming reuse storage allocated during setup. A release test fails if the selected allocator sees a heap call in those loops.
Cache lines are the API explains the queue layout. The test that rejected my quantizer covers recall checks, and The per-query spread follows the graph rebuild.
The bridge answer returns to the game
The engine reads the final UTF-8 fragment into its own buffer during a frame tick. The next read returns zero, which marks the end of the stream. Slot zero can now return to the executor because no generated bytes remain unread.
Mara's agent still holds the turn and its updated history. The UI can finish displaying the bridge answer, and the next question can reuse that state without keeping a decode slot occupied between conversations.