Skip to content

Qwen3 Dense

The complete Qwen3 dense family — 0.6B, 1.7B, 4B, 8B, 14B, and 32B — is the default pegainfer model line: pure Rust + CUDA, no Python at build time or runtime, full-attention GQA, paged KV cache, prefix caching, CUDA Graph decode, optional pegaflow KV offload, and DFlash or DSpark speculative decoding for Qwen3-4B.

From the pegainfer workspace root:

Choose a supported checkpoint by setting MODEL once. The same download and launch commands work across the family:

MODEL=Qwen3-4B
huggingface-cli download Qwen/$MODEL --local-dir models/$MODEL
export CUDA_HOME=/usr/local/cuda
cargo run --release -- --model-path models/$MODEL

pegainfer-server is the workspace default member. Qwen3-4B also launches with plain cargo run --release because its default path is models/Qwen3-4B.

SizeMODEL valueDeployment note
0.6BQwen3-0.6B
1.7BQwen3-1.7B
4BQwen3-4BDefault; DFlash and DSpark support
8BQwen3-8B
14BQwen3-14BDecode reroutes to batched eager because GQA group 5 has no compiled decode kernel
32BQwen3-32BAbout 63 GB of BF16 weights; use a large-VRAM GPU

To change the port:

cargo run --release -p pegainfer-server -- \
--model-path models/Qwen3-4B \
--port 8000

The server exposes an OpenAI-compatible /v1/completions endpoint:

curl -s http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{"model": "models/Qwen3-4B", "prompt": "The capital of France is", "max_tokens": 32}'

Streaming:

curl -N http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{"model": "models/Qwen3-4B", "prompt": "Write a haiku about Rust:", "max_tokens": 64, "stream": true}'

Useful Qwen3 flags:

# Disable CUDA Graph for debugging
cargo run --release -- --cuda-graph=false
# Pure host-tier KV offload benchmark mode
cargo run --release -- \
--kv-offload \
--kv-offload-host-gib 16 \
--no-prefix-cache
# DFlash or DSpark speculative decoding (single-GPU)
cargo run --release -- \
--model-path models/Qwen3-4B \
--dflash-draft-model-path models/dspark_qwen3_4b_block7

Tool calling goes through /v1/chat/completions with a tools array; a get_weather round-trip returns:

{"choices":[{"message":{"role":"assistant","tool_calls":[{"function":{"name":"get_weather",
"arguments":"{\"city\": \"Paris\"}"}}]},"finish_reason":"tool_calls"}]}

Two serving-benchmark suites are reported below — an engine comparison against vLLM on a consumer RTX 5090, and GH200 family benchmarks covering all six dense sizes — followed by KV offload and speculative-decoding results.

Measured on 1x RTX 5090 32GB, driver 590.48.01, CUDA 13.1 build, Qwen3-4B BF16 weights, TP1. pegainfer main 70888b2, vLLM 0.24.0, same vllm bench serve client, same host, same GPU, prefix cache on, seed 42, input 1024 / output 128 for the QPS sweep.

These tables were measured with the bench harness of the time: one seed for every sweep point against one long-lived server with the prefix cache on, so later points could replay prompts earlier points had already cached. Both engines saw the identical prompt stream, so the side-by-side is like-for-like; the absolute numbers are order-dependent, and the bench script has since moved to per-point seeds, so re-runs are not directly comparable to these tables.

MetricpegainfervLLM 0.24.0
RSS before stress, loaded and idle771 MB3814 MB
RSS after stress1064 MB3863 MB
Startup to HTTP ready, cold2.99 s70.0 s
Startup, warm compile cache~3.0 s32.7 s
GPU memory, default utilization28832 MiB30290 MiB

pegainfer is a single process; vLLM RSS is summed over its process tree. The pegainfer RSS peak during load is transient while reading safetensors through mmap; steady-state settles at 771 MB after load.

Poisson arrivals, 1024-token prompts, 128-token outputs, greedy (--temperature 0):

QPSpegainfer out tok/svLLM out tok/spegainfer TTFT p50vLLM TTFT p50pegainfer TPOT p50vLLM TPOT p50
1126.3126.245.2 ms54.9 ms6.53 ms6.71 ms
2252.3252.230.3 ms38.4 ms6.93 ms7.08 ms
4504.1503.348.8 ms38.7 ms8.30 ms7.95 ms
81007.81006.951.1 ms66.9 ms11.39 ms11.97 ms
101258.31256.353.4 ms76.3 ms13.55 ms14.11 ms
121507.71506.260.0 ms106.0 ms16.75 ms18.36 ms
161979.91687.9203.8 ms3832.3 ms46.92 ms79.42 ms

Low load (QPS 1–4) is comparable. At QPS 8–12 pegainfer leads on both TTFT and TPOT. At QPS 16 both systems are overloaded, but pegainfer edges ahead on throughput (1980 vs 1688 output tok/s) and stays 19× lower on TTFT.

Same harness, Qwen3-8B BF16, single RTX 5090 (32 GB). The 8B model is 2× the weights of 4B; throughput scales accordingly until the GPU saturates around QPS 8:

QPSpegainfer out tok/svLLM out tok/spegainfer TTFT p50vLLM TTFT p50pegainfer TPOT p50vLLM TPOT p50
1125.1125.082.2 ms97.4 ms11.55 ms11.63 ms
2249.9250.054.1 ms61.5 ms11.46 ms11.57 ms
4498.6498.588.1 ms103.6 ms16.08 ms16.24 ms
8991.9990.4148.0 ms235.1 ms30.97 ms35.56 ms

For multi-turn chat and agent workloads, most of the prompt often lands as a warm prefix-cache hit. In this sweep, the same prompt group is sent cold once to populate GPU KV cache, then sent warm:

Input lengthpegainfer coldpegainfer warm p50pegainfer warm p99vLLM warm p50vLLM warm p99
25616.2 ms8.5 ms8.8 ms14.5 ms19.1 ms
51224.6 ms8.6 ms8.8 ms16.0 ms16.4 ms
102444.0 ms9.2 ms9.5 ms18.4 ms19.0 ms
204892.0 ms10.4 ms10.8 ms23.7 ms24.4 ms
4096211.5 ms12.7 ms13.4 ms34.1 ms36.2 ms
8192460.0 ms21.6 ms22.8 ms58.6 ms59.9 ms
163841143.9 ms26.3 ms27.9 ms95.6 ms98.2 ms

pegainfer wins warm TTFT at every measured length; the 16k warm-cache path is 3.6× faster than vLLM p50.

All six dense sizes were measured on 1x GH200 120GB (aarch64, sm_90), BF16, TP1, with random 1024-token prompts and 128-token greedy outputs. The 0.6B and 1.7B runs used pegainfer main aea46ed and vllm bench serve 0.23.0 with range ratio 0 and no seed. The 4B through 32B runs used main c116077b and seed 42; each benchmark point started a fresh server so it could not reuse prefixes cached by an earlier point. Their low-load p99 therefore includes first-use process costs. c=N holds N requests in flight; QPS n uses Poisson arrivals.

The c=8 column is the one measured at the same load for all six sizes. The 0.6B and 1.7B ladders stop at QPS 16; 4B through 32B run to QPS 32.

ModelDecode pathc=8 out tok/sSaturationQueueing knee
0.6BCUDA Graph2805not reachednot reached
1.7BCUDA Graph2094not reachednot reached
4BCUDA Graph1204not reached, ≥3.7k tok/s at QPS 32QPS 24→32
8BCUDA Graph836~2.6k tok/sQPS 20→24
14Bbatched eager reroute504~1.5k tok/sQPS 12→16
32BCUDA Graph253~660–675 tok/sQPS 4→6

The model loads in 0.12 s and the server is ready in 3.4 s. The profiled KV budget is 84885 MiB (48506 blocks).

loadreq/sout tok/sTTFT p50 / p99TPOT p50 / p99
c=14.1052412.0 / 22.8 ms1.81 / 1.84 ms
c=411.58148314.1 / 973 ms2.09 / 2.17 ms
c=821.92280512.7 / 725 ms2.31 / 2.42 ms
QPS 10.991269.5 / 19.0 ms1.81 / 1.99 ms
QPS 21.972529.1 / 16.7 ms1.81 / 1.99 ms
QPS 43.935039.5 / 105 ms1.89 / 2.02 ms
QPS 87.87100711.1 / 344 ms1.98 / 2.53 ms
QPS 109.84126010.4 / 492 ms1.99 / 2.26 ms
QPS 1211.35145310.1 / 835 ms2.03 / 2.27 ms
QPS 1614.94191310.1 / 920 ms2.09 / 2.49 ms

QPS 16 is not saturated; c=8 reaches 2805 output tok/s. A 4097-token long-context run completes 4/4 with TTFT p50 36.7 ms. Under c=120 with 4096-token prompts, 120/120 requests complete with TTFT p50 2.86 s, TPOT p50 15.0 ms, and no server errors.

The HF logits golden gate passes all six execution paths (sequential bs=1 eager: mean 0.0387, p99 0.136, max 0.498). Greedy output is token-identical to HF on 2/6 24-token prompts; the other continuations remain coherent. Tool calling returns a valid get_weather call.

The model loads in 0.25 s and the server is ready in 3.9 s. The profiled KV budget is 82714 MiB (47265 blocks).

loadreq/sout tok/sTTFT p50 / p99TPOT p50 / p99
c=13.0138516.0 / 21.9 ms2.49 / 2.51 ms
c=48.14104218.5 / 965 ms2.77 / 2.86 ms
c=816.36209419.0 / 979 ms3.06 / 3.29 ms
QPS 10.981269.4 / 15.4 ms2.48 / 2.66 ms
QPS 21.9625010.0 / 17.3 ms2.52 / 2.67 ms
QPS 43.9150110.2 / 14.7 ms2.58 / 2.69 ms
QPS 87.82100113.1 / 25.6 ms2.70 / 3.41 ms
QPS 109.78125211.4 / 888 ms2.70 / 3.27 ms
QPS 1211.20143411.5 / 974 ms2.73 / 3.14 ms
QPS 1615.02192311.5 / 760 ms2.89 / 3.61 ms

QPS 16 is not saturated. A 4097-token long-context run completes 4/4 with TTFT p50 46.3 ms. Under c=120, 120/120 requests complete with TTFT p50 3.48 s, TPOT p50 19.7 ms, no server errors, and no unknown token IDs.

The HF logits golden gate passes all six execution paths (sequential bs=1 eager: mean 0.0332, p99 0.123, max 0.250). Greedy output is token-identical to HF on 5/6 24-token prompts. Tool calling returns a valid get_weather call.

The profiled KV budget is 34783 blocks.

loadreq/sout tok/sTTFT p50 / p99TPOT p50 / p99
c=11.7422326 / 35 ms4.3 / 4.3 ms
c=45.4669928 / 891 ms4.9 / 5.1 ms
c=89.41120429 / 1120 ms5.6 / 5.7 ms
c=1614.19181764 / 1829 ms7.1 / 7.4 ms
c=3220.52262648 / 2130 ms10.6 / 11.1 ms
c=6426.98345453 / 2054 ms17.4 / 17.6 ms
QPS 10.9712429 / 40 ms4.5 / 5.0 ms
QPS 21.9224627 / 37 ms4.6 / 5.0 ms
QPS 43.8449230 / 99 ms4.9 / 5.5 ms
QPS 87.6998530 / 412 ms5.6 / 6.6 ms
QPS 109.43120730 / 425 ms5.8 / 7.1 ms
QPS 1211.51147330 / 992 ms6.2 / 8.5 ms
QPS 1615.01192132 / 1103 ms7.2 / 8.9 ms
QPS 2018.95242536 / 1087 ms9.4 / 12.5 ms
QPS 2422.43287141 / 1059 ms12.5 / 14.5 ms
QPS 3228.963707239 / 2064 ms30.3 / 31.2 ms

QPS 32, the last measured point, still adds throughput over QPS 24 (2871 → 3707 out tok/s) while TTFT p50 jumps from 41 to 239 ms — the ladder ends at ≥3.7k tok/s without reaching a plateau. The interactive band ends around QPS 24, where TPOT p50 crosses from 12.5 to 30 ms.

The profiled KV budget is 31057 blocks.

loadreq/sout tok/sTTFT p50 / p99TPOT p50 / p99
c=11.1715034 / 43 ms6.4 / 6.5 ms
c=43.7047340 / 871 ms7.4 / 7.4 ms
c=86.5383662 / 1417 ms8.3 / 8.5 ms
c=1610.50134472 / 1248 ms10.6 / 10.9 ms
c=3214.42184676 / 2288 ms15.5 / 15.9 ms
c=6418.55237580 / 2373 ms25.3 / 29.0 ms
QPS 10.9612339 / 43 ms6.7 / 7.3 ms
QPS 21.8924240 / 58 ms7.0 / 7.6 ms
QPS 43.7748342 / 69 ms7.7 / 9.1 ms
QPS 87.5596744 / 327 ms9.2 / 11.7 ms
QPS 109.39120146 / 859 ms10.1 / 12.6 ms
QPS 1211.16142948 / 885 ms11.3 / 13.8 ms
QPS 1614.87190360 / 861 ms16.9 / 19.3 ms
QPS 2017.722268119 / 1122 ms31.5 / 38.7 ms
QPS 2419.9925590.6 / 2.0 s44.8 / 47.0 ms
QPS 3220.5926363.4 / 7.0 s45.5 / 47.6 ms

Throughput saturates around 20 req/s and ~2.6k out tok/s — QPS 24–32 hold 2559–2636 tok/s. The interactive band ends around QPS 16, where TPOT p50 crosses from 17 to 32 ms.

Decode runs on the batched eager reroute (no CUDA Graph at this size). Load to HTTP-ready is 8.7 s warm; the profiled KV budget is 57.5 GB (23020 blocks).

loadreq/sout tok/sTTFT p50 / p99TPOT p50 / p99
c=10.668457 / 68 ms11.5 / 11.5 ms
c=42.2428771 / 982 ms12.7 / 12.7 ms
c=83.94504130 / 848 ms14.2 / 14.7 ms
c=166.06776133 / 1771 ms17.8 / 24.0 ms
c=329.241182131 / 2552 ms25.2 / 25.8 ms
c=6411.511473136 / 4129 ms41.6 / 42.0 ms
QPS 10.9311970 / 104 ms12.2 / 13.2 ms
QPS 21.8123271 / 115 ms12.8 / 14.3 ms
QPS 43.6146375 / 165 ms14.6 / 18.7 ms
QPS 65.4369581 / 329 ms17.3 / 21.2 ms
QPS 87.2592887 / 254 ms21.2 / 27.0 ms
QPS 108.901140112 / 1157 ms27.9 / 32.4 ms
QPS 1210.511345165 / 980 ms41.9 / 48.9 ms
QPS 1611.5214751.4 / 3.3 s71.6 / 79.4 ms
QPS 2011.8815203.8 / 7.7 s74.4 / 78.8 ms
QPS 2411.8715196.0 / 12.8 s76.2 / 82.2 ms
QPS 3212.13155311.1 / 22.3 s78.9 / 79.2 ms

The eager batched-decode path saturates around 12 req/s and ~1.5k out tok/s — QPS 16–32 hold 1475–1553 tok/s while TTFT grows with queueing, and c=64 lands on the same plateau. The interactive band ends around QPS 10–12, where TPOT p50 crosses from 28 to 42 ms.

From the earlier ffb959c4 sweep on the same GPU class: long-context at in=4097 / out=32, c=1 holds TTFT p50 at 224 ms; a c=120 overload with 4096-token prompts (~507k aggregate demanded tokens against the 23020-block pool) completes 120/120 with zero server-side errors; a get_weather tool-call round-trip returns well-formed tool_calls. Greedy output matches HF transformers (bf16, same GPU class) token-for-token on 4 of 6 test prompts over the first 20 tokens. One conspicuous completion, web-forum mimicry on a malformed arithmetic prompt, is token-for-token identical in HF. The per-size HF logits golden gate passes at 14B.

Load to HTTP-ready is 46 s cold; the profiled KV budget is 21.4 GB (5360 blocks) next to the 63 GB of weights.

loadreq/sout tok/sTTFT p50 / p99TPOT p50 / p99
c=10.3545134 / 144 ms21.4 / 21.4 ms
c=41.16149163 / 1113 ms25.2 / 25.3 ms
c=81.98253289 / 1783 ms28.7 / 29.7 ms
c=163.05391291 / 2555 ms38.2 / 39.2 ms
c=324.20538292 / 4389 ms56.8 / 57.8 ms
c=645.18663303 / 8855 ms92.9 / 93.8 ms
QPS 10.87111156 / 276 ms25.2 / 29.2 ms
QPS 21.65211160 / 329 ms29.5 / 37.9 ms
QPS 43.24414261 / 561 ms51.6 / 66.5 ms
QPS 64.375600.7 / 2.0 s79.7 / 104.2 ms
QPS 84.816161.9 / 6.1 s94.0 / 105.6 ms
QPS 104.786125.7 / 11.3 s103.8 / 108.8 ms
QPS 125.006408.5 / 16.9 s104.7 / 106.2 ms
QPS 165.1065212.9 / 28.0 s105.7 / 106.0 ms
QPS 205.1365620.1 / 39.4 s106.2 / 106.5 ms
QPS 245.2667324.9 / 49.3 s106.0 / 106.4 ms
QPS 325.2767536.3 / 71.5 s105.8 / 106.6 ms

The single GPU saturates around 5.2 req/s and ~660–675 out tok/s — QPS 12–32 hold 640–675 tok/s while TTFT is pure queueing, and c=64, past the ≤32-batch SplitKv decode-attention boundary, lands on the same plateau. The interactive band ends around QPS 4–6 (TPOT p50 52 → 80 ms, TTFT p50 261 → 709 ms).

From the earlier 5959f05 run: greedy output matches HF transformers (bf16, same GPU) token-for-token on 4 of 5 test prompts over the first 20 tokens. The fifth diverges at the second generated token, where HF’s own top-4 logits sit within a 0.375 spread and pegainfer emits HF’s second-ranked token, 0.25 below the top.

With --kv-offload, sealed Qwen3 KV blocks can be restored from the pegaflow host tier instead of recomputing full prefill. Measured on the same RTX 5090 setup as the engine comparison above. The pure-L2 mode below disables cross-request HBM prefix reuse, so every prefix hit is restored from host DRAM:

cargo run --release -- \
--kv-offload \
--kv-offload-host-gib 16 \
--no-prefix-cache
Input lengthCold full prefillL2 warm p50, host restoreSpeedup
25625.4 ms9.8 ms2.6x
51225.6 ms11.6 ms2.2x
102445.3 ms15.4 ms2.9x
204892.5 ms22.9 ms4.0x
4096211.1 ms37.5 ms5.6x
8192461.3 ms71.4 ms6.5x
163841140.5 ms125.5 ms9.1x

At 16k, the tiering picture is: HBM hit about 26 ms, host-tier restore about 126 ms, cold prefill about 1.14 s.

Qwen3-4B supports both drafter formats through --dflash-draft-model-path: DFlash proposes each block in parallel, while DSpark adds a Markov head that conditions each later proposal on the preceding sampled token. The target model remains unchanged. The example below uses DSpark; a DFlash checkpoint uses the same command and flag.

# Download the released DSpark block7 drafter
huggingface-cli download deepseek-ai/dspark_qwen3_4b_block7 \
--local-dir models/dspark_qwen3_4b_block7
# Launch with speculative decoding (greedy, single-GPU)
cargo run --release -- \
--model-path models/Qwen3-4B \
--dflash-draft-model-path models/dspark_qwen3_4b_block7

In the measured greedy run, single-stream TPOT drops from 5.8 ms to 3.0 ms — roughly 2× decode speedup from amortizing target forwards over accepted drafts. Concurrency sweep on the same RTX 5090 setup as the engine comparison above, greedy, sharegpt + SPEED-Bench (coding) datasets:

ShareGPT:

Concurrencybaseline tok/sDSpark tok/sbaseline TPOT p50DSpark TPOT p50
11703815.83 ms2.96 ms
457612886.72 ms3.59 ms

SPEED-Bench (coding):

Concurrencybaseline tok/sDSpark tok/sbaseline TPOT p50DSpark TPOT p50
11643145.87 ms3.07 ms
45749886.73 ms3.77 ms

DSpark gains 2.2× throughput on ShareGPT and 1.7–1.9× on coding, roughly halving TPOT on both.

For greedy requests, verify commits only target-agreed tokens. Sampled requests are also supported: the regular target sampler selects each verify row, and accepted drafts preserve that target distribution.

  • Full attention with grouped-query attention: 32 query heads, 8 KV heads, head dim 128, 36 layers. Qwen3-14B widens to 40 query heads over 40 layers (GQA group 5 — batched eager decode reroute); Qwen3-32B scales to 64 query heads and 64 layers (GQA group 8).
  • Paged KV cache uses full-lifetime admission, so requests that cannot fit are rejected instead of hanging under memory pressure.
  • Prefix cache is on by default; --no-prefix-cache disables GPU prefix matching, or becomes pure-L2 host restore mode when combined with --kv-offload.
  • CUDA Graph decode uses pre-allocated buffers and can be disabled with --cuda-graph=false for debugging.
  • DFlash/DSpark speculative decoding is single-GPU, supports greedy and sampled requests, and forces prefix caching off because the drafter needs clean target hidden states. It cannot be combined with KV offload or LoRA.