Rocket

The compiler, the engine it emits, and the rules both follow.

A rocket is a spark igniting fuel in an engine.

term	meaning
spark	the hardware that ignites it (DGX Spark, GB10)
booster	one spark, with its own memory and its own tank
chemistry	a model’s architecture, shapes, and weight format
fuel	a model and its weights. Has a chemistry.
engine	compiled for one chemistry on one rocket. Burns any fuel that matches.
rocket	boosters, fuel, and an engine assembled and launchable

Fuels exist without engines. Engines are compiled for one fuel. Measurements live in the log. Source is in the repository: engines/, fuels/, boosters/.

Doctrine

General-purpose engines pay for generality at runtime: dynamic shapes, runtime dtype dispatch, autotuned kernel selection, a general allocator, and a Python step loop on a 20-core Arm host. Their results on a spark are recorded in the log. An engine recovers that cost by refusing to be general.

One model. One cluster topology. One weight format. Every shape known ahead of time. That binds each engine, not the compiler. A second model means a second engine, never a branch in this one.

Structure

The compiler runs offline on the target hardware. It consumes the model checkpoint, a hardware profile, and an expert co-activation profile gathered from a representative corpus. It emits a plan: a pre-swizzled weight blob, a static memory map with literal byte offsets, an expert-to-node placement, a kernel selection with pinned tile parameters, the execution graph topology, and the cache arena geometry.

The engine maps that plan and executes it. No dynamic allocator, no autotuner, no dtype or shape dispatch, no kernel fallback paths, no Python in the step loop. Startup either maps the plan or refuses to start.

Changing the model is a compiler run producing a new engine.

What specialization buys

memory	residency is decided offline down to byte offsets and validated once. No utilization fraction to tune, nothing discovered by allocating until failure.
launch overhead	fixed shapes make the decode step one graph launch per batch bucket. Buckets are a compiled closed set (1, 2, 4, 8, 16, 32) with no shape wildcard.
weight layout	experts are pre-swizzled offline into the layout the GEMM wants, 64 KiB aligned, so a top-k gather is a few aligned contiguous reads. No runtime transposition or padding discovered at load time.
expert placement	the compiler places experts across nodes from measured co-activation. A general engine cannot, because it learns the model at load time.
TLB coverage	the weight arena is one hugepage-backed mapping, 512 MiB pages over a per-node working set in the tens of GiB.

What we give up

Arbitrary models, runtime-selectable dtypes, plugin backends, dynamic adapters, arbitrary parallelism degrees, and degradation onto unsupported hardware. These are out of scope rather than pending. A proposal to add one has to argue against the throughput it costs.

A reference implementation (HF/PyTorch) is kept as an offline parity oracle and for golden-vector regression. It never appears in the serving path.

Boosters

A rocket is one or more boosters of the same kind. A Falcon Heavy is three. Booster capacity and bandwidth are measured in the log.

arrangement	meaning	cost
split	one engine spans boosters	capacity and single-stream latency, pays fabric on every token
replicated	each booster runs a full engine on the same fuel	throughput, no fabric coupling
independent	boosters burn different fuel	no coupling

Capacity decides which are available, not preference. Fuel that does not fit in one booster forces split. Fuel that fits leaves all three open.

Booster count is part of a rocket’s identity. The plan bakes in expert placement and the all-to-all schedule for one topology, so a rocket built for two boosters does not launch on three.

Rules an engine must satisfy

Each rule follows from a measurement in the log.

Chemistry match

Build depends on architecture, topology, and hardware. It does not depend on weight values, so an engine burns any fuel whose chemistry matches: same layer shapes, same expert count and routing width, same weight format. A fine-tune of a supported fuel needs no rebuild.

Launch verifies chemistry, not fuel identity, and refuses on mismatch.

Expert placement is the exception. It is fit to one fuel’s co-activation statistics, so a different fuel of the same chemistry stays correct and loses locality. Placement quality is a performance property, not a correctness one.

Memory budget

Take an absolute byte budget from measured MemAvailable minus an OS floor, never a fraction of MemTotal (why).

usable = measured_cuda_free_after_registration
       - registration_reserve      (probed at startup)
       - os_floor                  (hard minimum 5 GiB)

Refuse to start if requested residency exceeds usable. MemTotal may appear in logs, never in an arithmetic path that sizes an allocation.

Alignment

64 KiB is the universal quantum: O_DIRECT I/O, mmap offsets, cache block rows, copy descriptors, shared-region layout. Constants inherited from 4K-page code are wrong until re-derived.

Host registration

Serialize ranks with a file lock, register in bounded row-aligned chunks, roll back to unpinned on failure, unregister on every exit path, and never register a regular-file mapping (protocol).

Cluster

Engines require the 64K kernel, which returns 2.10 GiB per node. Page size is a compiler input baked into the plan’s alignment constants. A node whose page size disagrees is refused at handshake. Both rails are required; single-rail is a startup refusal, because the compiled all-to-all schedule assumes two.

Kernels

Engines own their attention and expert kernels. Every kernel declares its dynamic shared-memory requirement and is rejected at build time above the GB10 ceiling (limits).

Reusable state

Whatever a layer needs to resume a sequence is one cache entry, written and restored atomically at a shared token boundary. An attention KV cache alone is not sufficient for models with recurrent state.

Numerics

Precision is fixed at compile time per layer. No runtime dtype selection, no mixed-precision fallback. A quality gate runs on every numerics change.

Observability

Telemetry goes through OpenTelemetry. Required signals: per-step time split (attention, expert, all-to-all, sampling), bytes read per token, cache hit rates, prefix restore latency, registration reserve against measured CUDA free, and per-rail fabric utilization.

Cardinality is bounded by construction. Request IDs, prompt hashes, and expert IDs are never metric dimensions. Per-request detail lives on spans.

First engine status

glm5-moe-nvfp4-2b, burning GLM-5.3-Flash across the pair. Updated as entries land; each row links its evidence.

milestone	state	measured
decode end to end, one booster	done	21.05 GiB/token at M=1
M-stream batching, token parity	done	grouped GEMM, crossover M=1
pair split + static fire	done, gate green	6.1x one booster; 5.60 tok/s M=8, 7.50 M=32
radix KV, fork sharing	done	93 streams at 262k vs 8 unshared
KDA kernel at roofline	kernel shipped, engine adoption in flight	47 to 215 GB/s
FP8 on the BF16 half	telemetry capturing	plan in fuel.yaml
c8..c64 forked-prefix vs 132.8 tok/s c32	not yet run	workload: scripts/scheduler/agent-workload.py
OpenAI-compatible API	not started

Known handicap: the head booster boost-caps near 728 MHz (thermal); lockstep pair decode runs at the slower rank’s pace.

Launch

rocket launch <name> runs the countdown. Each phase has its own failure mode, and no traffic is accepted before static fire passes.

phase	what happens	fails when
upright	resolve boosters, map the plan, verify the memory budget	boosters missing, topology mismatch, or requested residency exceeds usable
fuel	fetch the weights, condition them to the map, cache the result	first load only; later launches fetch or find it already on the booster
static fire	register host memory, bring up both rails, warm the graphs, run one token and check it	registration stalls, a rail is down, a kernel exceeds the shared-memory ceiling
launch	accept traffic

API

OpenAI-compatible /v1/chat/completions and /v1/completions with streaming, tool calling, and reasoning-content separation. Image input in scope, video out of scope.

Per-request knobs that would force a shape or kernel choice the plan did not compile are rejected. Serving-level controls are plan parameters, set at compile time and reported read-only.