Whole-family linear-attention NVFP4 reaches 102.76 tok/s at c16

experiment
baseline
numerics
attention
runtime
NVFP4 removes 2,997,226,080 bytes per c16 step and reaches 102.76 aggregate tok/s, 1.34% above the matched healthy FP8 control.
Author

agent

Published

2026-09-08

Question. Does the accepted whole-family linear-attention NVFP4 map retain production throughput on healthy hardware while reducing c16 weight traffic?

setup
nodes gx10-5e36 head and gx10-2a13 worker, GB10, 6.17.13-rocket64k, 64 KiB pages
checkpoint nvidia/Qwen3.8-Flash-Next-NVFP4 at fc694b54fb0174e0913e6adf86691ef85a4ead47
Rocket commit 422cde89c250d3fa8a8c0bd7800ebbbb52b085c6
vLLM image vllm/vllm-openai:qwen38-flash-next at sha256:d464f3b466fa9c45ddbff8a812e80564503b6879a9fd95c1a47514f3f0df5a4a
runtime source MiaAI-Lab Qwen3.8 dual-Spark at c2325b22602b51a5faf55fc2bebccc34f3f80b9f
serving map 180 base linear-attention projections in W4A4 NVFP4
artifact 64539e4a...56c4e5
runtime TP2, EP, MTP3, FP8 KV cache, 262,144-token context, vLLM

This path required a model-specific 180-tensor overlay, ModelOpt block-16 packing, and separate FP8 dispatch for MTP experts.

scripts/numerics/qwen38-expanded-calibration.sh \
  --output-dir /home/glwillen/calibration/qwen38-linear-nvfp4-production-20260907-06 \
  --nvfp4-artifact-dir /home/glwillen/calibration/qwen38-linear-nvfp4-artifacts/64539e4a5a4b533aa143d20e055ae73ebd10b21f05dbd422847a9be30e56c4e5 \
  --production

jq -r '.[] | [.concurrency,.aggregate_tok_s,.per_stream_tok_s,.mean_ttft_s] | @tsv' \
  /home/glwillen/calibration/qwen38-linear-nvfp4-production-20260907-06/throughput.json
1   41.216281042312566  41.216281042312566  5.85900435000076
2   72.77877253307481   36.97938550701514   5.90815700299936
4   70.11521222444112   28.345566091688937  5.998136556499958
8   93.19815126912113   19.738601543822398  9.338284739500068
16  102.7641806133714   11.711004318437539  16.682225616625374
concurrency aggregate throughput per-stream throughput mean TTFT
1 41.22 tok/s 41.22 tok/s/stream 5.859 s
2 72.78 tok/s 36.98 tok/s/stream 5.908 s
4 70.12 tok/s 28.35 tok/s/stream 5.998 s
8 93.20 tok/s 19.74 tok/s/stream 9.338 s
16 102.76 tok/s 11.71 tok/s/stream 16.682 s 
nvidia-smi --query-gpu=clocks.current.sm,power.draw,utilization.gpu --format=csv,noheader
ssh glwillen@192.168.100.11 'nvidia-smi --query-gpu=clocks.current.sm,power.draw,utilization.gpu --format=csv,noheader'
head:   2483 MHz, 36.01 W, 95 %
worker: 2463 MHz, 35.94 W, 95 %
matched healthy c16 result FP8 NVFP4 NVFP4 delta
aggregate throughput 101.40 tok/s 102.76 tok/s +1.34%
per-stream throughput 11.58 tok/s/stream 11.71 tok/s/stream +1.10%
mean TTFT 16.822 s 16.682 s -0.140 s
traffic removed from BF16 2,085,027,840 bytes/step 2,997,226,080 bytes/step +912,198,240 bytes/step

One NVFP4 ladder and one matched healthy FP8 ladder were measured. Run-to-run variance was not measured.

Verdict. Accepted for the serving-weight map. All 180 base linear-attention projections remain NVFP4 because the map removes 2,997,226,080 bytes per c16 step and holds c16 throughput within a 1.34% gain over the healthy FP8 control.

This vLLM result is a comparison baseline. At c16 it reaches 32.1% of Rocket’s greater-than-320 aggregate tok/s target and 58.6% of the greater-than-20 generated tok/s/stream target.

Next.

Reopen if.