Rocket build log

Experiments and research behind the engines

2026-09-08

MTU 9000 moves the fabric roofline to 24.5 GB/s one-way, 48.9 duplex

Raising both rails to MTU 9000 lifts active_mtu from 1500-limited to 4096, one-way peaks at 24.47 GB/s against the published 23.1, and the 128 KiB to 1 MiB sizes the expert exchange uses gain the most.

2026-09-08

A reduction barrier makes Qwen GDN graph replay bit exact

The fixed layer-0 GDN graph reaches 175.14 GB/s at c16 after a shared-memory barrier removes nondeterministic replay.

2026-09-08

Moving the KDA state tile out of shared memory is 4.6x at M=8

The engine's KDA recurrent step moves state at 47 GB/s because a 67 KiB FP32 shared tile pins it to one block and 4 warps per SM; keeping the column in registers moves the same bytes at 215 GB/s, 93% of roofline at M=32, with identical arithmetic and no change to the state contract.

2026-09-08

Radix-paged KV holds 93 streams at 262k where 8 fit unshared

A refcounted page pool with a radix tree over token sequences cuts per-session KV on the c32 agent trace from 4.00 to 0.35 GiB, taking 262k-context streams per booster from 8 to 93; page size turns out not to matter and the next wall is the 140.8 MiB FP32 KDA state.

2026-09-08

Qwen3.8 linear-attention FP8 passes 12 of 12 quality cases

Converting all 180 base linear-attention projections from BF16 to FP8 removes 1.942 GiB per c16 step with zero quality regressions across 494 telemetry channels.

2026-09-08

Qwen3.8 base linear attention passes all 12 quality cases in NVFP4

All 180 base linear-attention projections pass 12 quality cases in NVFP4 with zero regressions and remove 2,997,226,080 bytes per c16 step.

2026-09-08

Qwen3.8 base full attention passes all 12 quality cases in NVFP4

All 48 base full-attention Q/K/V/O matrices pass the matched quality gate in NVFP4, bringing the accepted attention map to 228 matrices and removing 3,856,402,656 bytes per c16 step.

2026-09-08

Qwen3.8 base routers and PLE pass all 12 quality cases in NVFP4

All 48 base routers and both layer-2 PLE projections pass the fixed quality gate in NVFP4, freezing the eligible 278-matrix map before production measurement.

2026-09-08

Whole-family linear-attention NVFP4 reaches 102.76 tok/s at c16

NVFP4 removes 2,997,226,080 bytes per c16 step and reaches 102.76 aggregate tok/s, 1.34% above the matched healthy FP8 control.

2026-09-08

Head's 2.3x slowdown is a 728 MHz GPU clock ceiling, not kswapd

The head node never boosts past 728 MHz SM clock on this workload while the peer reaches 2437 MHz; three vm.* sysctls and pausing the operator's IDE moved neither the clock nor the step time, and kswapd/kcompactd activity does not correlate with any single slow run.

2026-09-08

Display-power renegotiation does not lift the head's 728 MHz cap

Under the same two-node Qwen3.8 inference load, the head remains capped at 728 MHz while the peer reaches 2489 MHz, so the display-power hypothesis is rejected.

2026-09-08

A cold power cycle lifts the head from 728 MHz to 2418 MHz

Removing every head power source before boot restores a 2418 MHz SM clock and brings its static-fire time within 1.1% of the peer.

2026-09-08

Removing display power restores the head's normal GPU clock range

The head returned to the peer's 208 MHz idle and 3003 MHz maximum clock range after its powered display was disconnected and input power renegotiated.

2026-09-08

K1 wins the matched Qwen3.8 K1 to K7 general-engine sweep

K1 leads four of five generic concurrency points, K6 narrowly leads c4, and every c16 result remains below Rocket's serving floors.

2026-09-08

Local EXT4 puts Qwen3.8 first token at 468.83 seconds

Two cold K1 runs from matched local EXT4 checkpoints averaged 468.83 seconds from service exec to the first visible token.

2026-09-08

Static MTP1 wins the Qwen3.8 general-engine depth sweep

MTP1 leads the matched K1 to K3 sweep at 108.76 aggregate tok/s at c16, while every static depth remains below the Rocket acceptance floor.

2026-09-08

The 728 MHz power state inflated Qwen3.8 loading by 5.1x

A cold power cycle cuts identical Qwen3.8 FP8 model loading from 997.7 seconds to 193.8 seconds, superseding the low-power startup diagnosis.

2026-09-07

Rank-local E256 MoE misses the Qwen K1 and K4 step budgets

The specialized E256-per-rank FlashInfer path reaches 88.3% of one GB10's read roof, but its 48-layer contribution already exceeds the >451 token/s K1 and K4 budgets.

2026-09-07

The expert split decodes the same tokens; the fabric is idle on one rank and the wall on the other

Static fire passes with 0/8 streams mismatched across the pair, and the 144/144 expert split runs 2.47 tok/s at M=8 against 7.16 single-booster because rank 1 stalls 1.67 s/step on the exchange while rank 0 uses 76 ms of it.

2026-09-07

The pair is 6.1x one booster, not 2.9x behind it, and the expert cache is most of the ladder

The 7.16 tok/s single-booster baseline fed every stream the same token; re-measured with distinct streams the pair runs 2.25 tok/s against one booster's 0.37, and balancing, overlap and an 80 GiB per-rank expert cache take it to 5.60 at M=8 and 7.50 at M=32 with static fire green at every step.

2026-09-07

Exact-N640 retained tail wins only at Qwen c4

A guarded FlashInfer retained2 tail cuts c4 MoE latency by 1.84%, ties within variance at c8, and loses by 1.83% at c16.

2026-09-07

A TP2 vocab GEMV cuts Qwen3.8 M1 head latency by 31%

The fixed 124160 by 2560 BF16 GEMV takes 2.63 ms on GB10 and preserves the FP32 cuBLAS greedy token.

2026-09-07

Exact-N640 dynamic MoE cuts Qwen c16 latency by 17.7%

FlashInfer's exact-N640 dynamic path takes 2.05 ms on GB10 while the static path pads expert traffic by 20% and takes 2.49 ms.

2026-09-07

The Qwen controller publishes one authenticated TP2 boundary

A two-node Rocket restore rejected an injected rank-1 copy fault, then published matching 32,354,861,056-byte rank states and generation 8.

2026-09-07

Layer 3 completes the exact Qwen attention state transition

The specialized c16 path completes the exact transition with matching two-rank hashes; three isolated launches ranged from 3.873 to 14.106 ms.

2026-09-07

Qwen3.8 linear-attention NVFP4 removes 2.791 GiB per c16 step

All 180 linear-attention projections pack into NVIDIA's block-16 NVFP4 ABI and pass the two-node preflight before quality evaluation.

2026-09-07

Unmodified Qwen3.8 passes the 12-case quantization gate

The NVIDIA NVFP4 checkpoint passes 10 deterministic reasoning tasks and two long-context needles before Rocket changes any dense precision.

2026-09-07

The fully touched Qwen3.8 c16 state allocation fits beside the 62.46 GiB model

One GB10 rank held the resident 62.46 GiB model and 33,420,083,200 bytes of fully touched c16 state with two complete 64 KiB page readbacks.

2026-09-07

The Qwen3.8 state plan counts 2,088,755,200 CUDA bytes per rank-stream

The pinned TP2, MTP3, 262144-context allocation plan accounts for 2,022,044,096 logical bytes and 2,088,755,200 CUDA bytes per rank-stream.

2026-09-07

Qwen3.8 reaches 163.40 tok/s at c16 without speculation on 64 KiB pages

The exact NVIDIA checkpoint reaches 163.40 aggregate tok/s and 11.78 tok/s per stream at c16 on Rocket's two-node 64 KiB pair.

2026-09-07

One-tensor staging loads Qwen3.8 on 64 KiB pages

Cloning each selected safetensors tensor before its CUDA copy loads all 11 Qwen3.8 shards in 598 seconds on both 64 KiB nodes.

2026-09-07

One booster decodes GLM-5.3-Flash end to end, and a token reads 21.05 GiB, not the recorded 9.6

All 45 text layers run on a single GB10 and emit coherent text; only the routed experts are NVFP4, so a decode step reads 16.62 GiB of BF16 plus 4.43 GiB of experts, putting the two-booster roofline at 21.1 tok/s instead of 49.6.

2026-09-07

The CUTLASS grouped GEMM beats the scalar GEMV loop at every M, M=1 included, so the crossover is 1

Replacing the per-(stream, expert) GEMV loop with the CUTLASS NVFP4 grouped GEMM cuts routed-expert wall time 40-66% from M=1 to M=8, with zero batch to pay for it, and grouped-M1 vs grouped-M8 stays token-for-token identical over 20 generated tokens.

2026-09-07

Tensor dispatch and duplicate passes dominate Qwen3.8 startup

Qwen3.8 loads 299,545 gapless tensors at 1.997 ms each and reads the checkpoint twice for MTP3, while one shard streams from NVMe at 4.65 GB/s.

2026-09-06

Two idle NIC ports held 444 MiB; unbinding them costs nothing

ConnectX firmware holds 570 MiB per node across four PFs; the two unwired f0 ports account for half, and unbinding them returns 444 MiB (peer) and 393 MiB (head) with the RoCE fabric unaffected.

2026-09-06

Deleting the key transpose puts the DSA indexer scan on its streaming floor

The indexer scoring pass goes from 73.3 to 209.9 GB/s, 2.9x, by making keys the A operand instead of B, and exact top-2048 selection adds 8% on top of that.

2026-09-06

Fusing the MoE layer's two grouped GEMMs is worth 4%, and a CUDA graph already collects it

w13 and w2 share no weight bytes, so per-expert fusion removes no traffic; a single-launch ceiling probe lands at 21.1 ms against 21.9 ms separate and 17.1 ms of roofline, the same 4% a CUDA graph gets for free.

2026-09-06

CUTLASS NVFP4 grouped GEMM runs on sm_121 unpatched and holds 80% of roofline at MoE decode shapes

At 288 experts and M=8..64 per expert, CUTLASS 4.8.0 block-scaled FP4 sustains 186-196 GB/s of a 238 GB/s roofline and beats BF16 cuBLAS by 3.7-5.7x, so rocket does not need its own port.

2026-09-06

The indexer caches per-token gates too: 7 unshared 262k streams per booster

Reference code settles three DSA semantics: top-k counts tokens, ReLU inside the score, and the pool compressor's softmax needs per-token gates cached alongside keys, doubling indexer KV to 512 B/token/layer and cutting unshared capacity to 7 streams at actual headroom.

2026-09-06

At the 262k cap the DSA indexer costs 12x the sparse MLA it selects for

GLM-5.3-Flash attention geometry read off the checkpoint: KDA state is 72.8 MiB per stream and 28.0 ms per step at 32 streams, sparse MLA is 27.6 ms, and ranking 262144 candidates to pick 2048 of them is 326.9 ms.

2026-09-06

Real NVFP4 scales need the sm_100 swizzle, and a booster loads in 14 s

CUTLASS on sm_121 consumes checkpoint FP4 directly once block scales are swizzled to the Sm1xx atom layout; unswizzled scales are 17-22x wrong, gate and up projections carry different global scales, and a 91 GiB half-checkpoint loads in ~14 s at the NVMe floor.

2026-09-06

vLLM reaches 207 tok/s at c8, while the c16 weight roofline is 282 tok/s

The exact NVIDIA Qwen3.8 checkpoint reaches 207.0 tok/s on two GB10s at c8, and its expected c16 expert union puts the non-speculative weight-read ceiling at 281.9 tok/s.

2026-09-06

The idle GPU driver holds 0.7 GiB; the 64K slab tax holds 1.4

Unloading the full NVIDIA stack returns 0.67 GiB; unreclaimable slab stays at 1.4 GiB spread across hundreds of mostly-empty 64 KiB slabs, and 1.3 GiB remains unattributed without page_owner.

2026-09-06

A GB10 node reads at 238 GB/s and the pair's fabric moves 23 GB/s

Measured GB10 memory and RoCE bandwidth replace datasheet numbers, putting the NVFP4 engines at 31% of the batch-1 roofline.

2026-09-06

3.5 GiB per node is firmware-owned; asking NVIDIA for a headless option

The remaining reserved memory is the display/GSP carveout and firmware regions no OS change can return, and NVIDIA's own release notes show the carveout is BIOS-adjustable upward, so the request is a smaller setting.

2026-09-06

Stripping the OS returns 217 MiB; the rest is carveout and driver floor

The missing memory is 3.55 GiB of firmware carveouts (2.55 GiB video/GSP) and a 4.1 GiB kernel+driver floor that survives a full GUI and service strip.

2026-09-06

The 98 GiB CUDA ceiling was page cache, not a limit

cudaMemGetInfo free drops 1:1 with clean page cache on GB10 unified memory; with caches dropped CUDA grants and device-touches 106 GiB, above the pool the serving run refused to admit.

2026-09-06

Dropping random kmalloc caches returns 0.9 GiB; mlx5 owns more than the GPU driver

A rebuilt 64K kernel without CONFIG_RANDOM_KMALLOC_CACHES cuts unreclaimable slab from 1.34 to 0.43 GiB, and page_owner attributes the rest: 859 MiB to mlx5 NIC firmware pages, 642 MiB to the NVIDIA driver.

2026-09-05

gh and quarto install clean on arm64

Both install from arm64 debs and quarto check passes, so a broken render is caught locally instead of in CI.

2026-09-05

Specializing to one fuel and one GPU is worth 1.2x to 1.5x at batch 1

General engines reach at most 50% of H100 bandwidth at batch 1, a megakernel reaches 78%, and the advantage is gone by batch 16 where their schedulers win.

2026-09-05

cudaHostRegister needs serialization and row-aligned chunks

Whole-region registration from every rank deadlocked startup at 48 GiB; a flock plus bounded row-aligned chunks with unpinned rollback is the working protocol.

2026-09-05

General engines reach 132.8 tok/s at c32 on this pair

Recorded SGLang and vLLM results on the same two nodes, which is the bar an engine has to beat.

2026-09-05

Both sparks are identical: GB10 sm_121, 64 KiB pages, 123.73 GiB

Measured baseline for the two-node cluster, including dual 200Gb RoCE links up and the peer reachable on both.

2026-09-05

64 KiB pages return 2.10 GiB per node

Boot-time reserved memory drops from 6.09 GiB to 4.00 GiB, raising MemTotal to 123.73 GiB per node and 4.20 GiB across the pair.

2026-09-05

Three GB10 limits that decide how kernels get written

C2C segfaults on CPU reads of cudaMalloc memory, shared memory caps near 99 KB, and sm_121 is missing from vendor kernel matrices.

2026-09-05

TensorRT-LLM is not the runtime for rocket

Rejected on model fit and kernel ownership. Superseded within the hour: two of the four objections were precedent, not evidence.

2026-09-05

TensorRT-LLM: rejected as a runtime, not as a kernel source

Its kernels are open and buildable for sm_121, so the kernel question is a benchmark, not an assumption. The runtime still loses on what it contributes.