A prompt through Astral_

Astral is a C++17 inference control plane for real-time engines. Unity and Unreal cross one C ABI using native handles and byte spans. Astral owns prompts, durable agent state, continuous batching, sampling, retrieval, cancellation, and token delivery; a backend provider owns model execution and KV state. The result is one runtime contract instead of a different integration for every engine, model backend, and feature.

That middle layer exists because "call an LLM from the engine" becomes several systems problems as soon as the model is part of a frame loop. Who owns a prompt after a C# string moves across P/Invoke? Where does chat history live? Which thread may cancel generation? What happens when twelve characters want tokens but the model has four decode slots? How do bytes return to the engine without a managed allocation or callback on every token?

A thin wrapper can hide those questions for a demo. It cannot answer them for a runtime. Astral answers them once, in native code, with bounded ownership and behavior that can be tested below the engine layer. That is the difference between exposing a model call and building inference into a game.

One runtime, several kinds of work

Chat is the easiest path to demonstrate, but it is not the whole runtime. The same ABI covers asynchronous text embeddings; model-dependent image, audio, and multimodal embeddings; exact and bounded-graph native memory search with filters and persistent snapshots; GBNF and JSON-schema output constraints; tool parsing, log probabilities, prompt caches, LoRA adapters, deterministic session save and restore, and media chunks inside a conversation. These are runtime capabilities, not wrapper-specific conveniences.

The leverage comes from the common contract. Every input crosses the same sized C structs and generation-tagged handles. Every asynchronous operation gets bounded storage, explicit cancellation, and caller-owned collection. Model-specific work stays behind the provider table. Adding retrieval or structured output does not force a second scheduler, allocator policy, or engine lifetime model:

The rest of this article follows chat because one generated answer crosses almost every boundary: wrapper memory, agent state, scheduler capacity, provider execution, sampling, and streaming. The embedding and memory paths reuse the same design vocabulary, but they do not all pass through the token decoder.

Start with one engine call

Suppose an Unreal character has a native Astral agent handle. The game sends one user turn: "What did you see near the bridge?" The wrapper constructs an AstralSpanU8, points it at the UTF-8 bytes, and calls astral_agent_chat_enqueue.

The span is borrowed only for that call. Astral copies the user text into native storage before returning. The game can release or move its string without leaving the decode worker with a dangling pointer. That single ownership rule removes a large class of language-boundary bugs.

typedef struct AstralSpanU8 {
    const uint8_t* data;
    uint32_t len;
    uint32_t _padding;
} AstralSpanU8;

AstralAgentChatDesc chat = {0};
chat.size = sizeof(chat);
chat.user_message = prompt;

AstralErr err = astral_agent_chat_enqueue(agent, &chat);

The public surface is C even though the runtime is C++. Structs are plain data. Handles are 64-bit tagged values. Results travel through return codes and out-parameters. Exceptions stop at the boundary. Every public struct begins with its byte size, and compile-time layout assertions catch accidental ABI drift on 32-bit and 64-bit targets.

The word "stable" needs a qualifier here: Astral is pre-1.0. The C shape is designed for long-lived compatibility, but breaking changes are still allowed before 1.0 when they are documented and accompanied by binding updates. Pretending otherwise would make the version field decorative.

The agent, not the wrapper, owns the conversation

A native agent stores the system prompt, rolling summary, retrieved memory context, chat history, optional toolset, prompt cache, and the conversation handle used for active decode. Unity and Unreal wrappers remain thin owned handles over this state. The character can outlive a scene object, change engine wrappers, or sit idle for minutes without moving its conversational state into managed collections.

On enqueue, the agent assembles one bounded prompt in a defined order:

• system prompt;
• rolling summary;
• retrieved memory context;
• prior messages;
• the current user turn;
• the assistant prefix.

For the bridge question, suppose the agent's stable prefix contains the character rules, a short summary of the previous scene, and a retrieved memory saying that Mara saw torchlight near the bridge. The new suffix contains only "What did you see near the bridge?" plus the assistant marker. This is an illustrative conversation passed through the implemented ownership rules, not a recorded game session.

History is not a vector of heap-owning strings. Small POD records point into one agent-owned byte arena. The agent caches the stable prefix after its system prompt, summary, memory, or history changes, then appends only the current suffix for the next turn. With a prompt cache bound, a stable-prefix hit can reuse cached tokens and tokenize only the new part.

Those counters are deliberately narrow. "Reused prompt tokens" describes native prompt setup. It does not claim that the backend reused the matching KV prefix. Token reuse and KV reuse are related optimizations with different owners.

Growth has a policy rather than an accident. max_messages and max_prompt_bytes bound the agent. The default overflow behavior rejects an oversized turn. A caller may instead choose to truncate the oldest history until the prompt fits. Either way, the behavior is known before decode starts.

Agents wait outside the expensive slots

A game may have hundreds of characters but only a few active conversations. Astral therefore separates an agent's durable state from a model executor slot. An idle agent owns prompts and history without occupying KV-backed decode capacity. Character count is no longer forced to equal the model's concurrency limit.

The first chat request acquires a conversation slot from the model-scoped executor. If every slot is busy, enqueue may reclaim a completed, canceled, failed, or idle agent slot, but only after its stream bytes have been drained. Otherwise the call returns ASTRAL_E_BUSY. Unread output is part of ownership, so a "finished" producer is not automatically a reusable slot.

Assume the bridge agent acquires slot zero. Its native prompt tokens now belong to that conversation. The agent handle still owns Mara's history; slot zero owns only the active decode state. When the answer finishes and its stream is drained, Astral can release the slot without deleting what Mara remembers.

Slot affinity is optional. A caller can prefer a one-based slot when backend locality matters. Affinity does not permit stealing a live conversation; if that slot cannot be safely reclaimed, enqueue stays busy.

The executor turns conversations into one batch

The executor belongs to a model because batching only makes sense across requests that share that model and provider context. It scans active slots, adds bounded prompt work and decode work to a provider batch, evaluates that batch, then maps each logits output back to the conversation that requested it.

Prompt ingestion has a per-slot cap per tick. Without it, one long new prompt could fill a batch repeatedly while already-decoding conversations wait for their next token. The cap makes the trade visible: prompt throughput is allowed to yield so token latency across active conversations remains bounded by the scheduler's rotation.

Each active model executor owns one dedicated thread, separate from Astral's bounded worker pool. Provider session state, slot masks, and batch buffers stay single-owner there, so the control path does not pay for several threads contending over the same per-token state. Parallelism still exists across model executors and inside backend evaluation, while embeddings, memory-index work, and bounded runtime jobs retain the full configured worker budget.

The provider computes; the core decides

Astral's backend boundary is an operations table, not a C++ inheritance tree exposed to engine code. A local logits provider loads the model, tokenizes input, owns session and KV state, evaluates tokens, and exposes a read-only logits view. The core applies penalties and grammar constraints, samples the next token, detokenizes it, and moves the resulting UTF-8 bytes toward the engine. Remote execution uses the same engine-facing ownership and streaming contract while transport-specific work stays behind its provider.

On one turn of the worked trace, the provider returns logits for slot zero. Astral applies Mara's grammar and sampling configuration, selects the next token, and detokenizes it. If the piece is the byte I, that byte goes into slot zero's stream ring while the chosen token returns to model evaluation on the next executor tick.

This split is useful for more than swapping CPU and GPU implementations. For logits providers, grammar, stop rules, log probabilities, tools, and sampling remain core-owned semantics. The game can move between local CPU execution, optional CUDA offload, deterministic mock runs, and remote transport without replacing its handles, cancellation rules, or stream consumer. Backend selection becomes an execution decision instead of an engine rewrite.

The built-in registry includes CPU, mock, and remote providers. The remote path has loopback coverage for health, authentication, tokenization, streaming, and embeddings. CUDA offload is optional at build time. Its harness and parity gates exist, but I do not yet have current real-hardware CPU-versus-CUDA measurements. Source support and measured parity are different claims.

The engine pulls bytes on its own clock

Generated bytes land in a bounded single-producer, single-consumer ring. The decode worker is the producer. One engine-side consumer calls astral_agent_chat_stream_read or the lower-level conversation/session equivalent with a caller-owned buffer.

There is no callback into an arbitrary game thread and no required managed allocation per token. The engine can poll with a zero timeout during Update or a tick, fill its own buffer up to the UI budget, and return to the frame. A positive result is a byte count, zero is end of stream, and a negative value carries an error such as timeout. The engine owns pacing instead of inheriting the backend's thread and callback behavior.

The engine therefore receives the bridge answer as ordinary UTF-8 fragments in its own buffer. It may reveal "I saw torchlight..." over several frames without moving prompt assembly, history mutation, or model scheduling onto the game thread. After the final zero-length read, the stream is drained and slot zero can return to the executor.

The ring also makes backpressure real. If the consumer stops draining, the producer cannot overwrite unread output. The conversation is not safely reclaimable until those bytes have an owner. This is why slot release checks stream state instead of equating "model stopped generating" with "request no longer exists."

Memory is assigned before the hot path

Astral reserves and allocates control structures when the runtime, model, conversation, or agent is created. Conversation prompt tokens, sampling candidates, and stream storage are bounded buffers. Agent history uses an arena. The decode loop reuses those structures. The memory-index path follows the same discipline with fixed-capacity workspaces and architecture-dispatched kernels rather than hiding scratch allocation inside each query.

The project's claim is intentionally "allocation-gated hot paths," not "the process never allocates." Model loading, agent creation, prompt replacement, and other control operations may allocate. Release gates instrument steady-state decode, sampling, and streaming so a new heap call in those paths fails evidence rather than disappearing inside an average.

The same specificity applies to concurrency. The code has bounded SPSC, MPSC, and MPMC structures because their ownership patterns differ. Atomics carry explicit memory orders. Public rings and queues isolate producer and consumer control fields onto cache lines. "Lock-free" is not used as a synonym for "fast"; the implementation and generated instructions still need inspection.

The surface at a glance

So what is Astral?

For a game, Astral is one native API for generation, durable agents, embeddings, retrieval, structured output, deterministic continuation, and multimodal input. For the runtime, it is the control plane that should not be duplicated in C#, Unreal modules, and backend implementations: prompt state, bounds, scheduling, sampling, cancellation, streaming, persistence, and the rules that join their lifetimes.

One prompt exposes the whole design. Engine bytes cross a size-checked C struct and become native-owned state. An idle agent acquires a bounded slot only when work begins. The model executor folds active conversations into a provider batch. The provider evaluates; the core samples. Generated bytes enter a bounded ring and return when the engine is ready to read them.

Astral already owns the hard middle layer end to end: bytes cross one ABI, durable agents acquire expensive decode capacity only when needed, active conversations batch by model, providers remain replaceable, and results return on the engine's clock. The remaining work is concrete validation breadth: more editor and device runs, CUDA parity and performance measurements, multimodal verification across supported models, and ABI stabilization on the road to 1.0.