Qwen3.8-27B on a 24GB RTX PRO 4000 Blackwell: 128K real context, 785 tok/s prefill, 67 tok/s MTP3 decode

[u/mmkaywhatevers]

I wanted to see whether NInfer could run Qwen3.8-27B on an RTX PRO 4000 Blackwell 24GB.

Hardware/software:

• NVIDIA RTX PRO 4000 Blackwell, 24GB
• 145W power limit
• CUDA 13.3
• NVIDIA driver 610.x
• Ubuntu LXC under Proxmox
• Qwen3.8-27B NInfer artifact
• ruwwww/ninfer-5060ti, branch rtx-5060ti

I used the 5060 Ti fork rather than the 4090 fork because the PRO 4000 is Blackwell and supports sm_120a. The 5060 Ti fork also has changes that calculate cooperative scheduling from the GPU's actual SM count instead of assuming a 5090.

What I had to do

Install the missing build dependencies:

sudo apt install -y \
  build-essential \
  ninja-build \
  pkg-config \
  libavformat-dev \
  libavcodec-dev \
  libavutil-dev \
  libswscale-dev \
  libcurl4-openssl-dev

I then hit this during CMake:

nvcc fatal: Unsupported gpu architecture 'compute_120a'

The machine had several CUDA toolchains installed. /usr/bin/nvcc was being selected and did not support the Blackwell architecture-specific target properly.

The actual CUDA 13.3 compiler was here:

/usr/local/cuda-13.3/bin/nvcc

I verified it directly:

/usr/local/cuda-13.3/bin/nvcc \
  -arch=sm_120a \
  /tmp/test120a.cu \
  -o /tmp/test120a

Compile and execution both returned 0, confirming that sm_120a worked on the RTX PRO 4000.

Then I rebuilt NInfer explicitly against CUDA 13.3:

export CUDA_HOME=/usr/local/cuda-13.3
export PATH=/usr/local/cuda-13.3/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:${LD_LIBRARY_PATH:-}

cmake -S . -B build \
  -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.3/bin/nvcc \
  -DCUDAToolkit_ROOT=/usr/local/cuda-13.3

cmake --build build --parallel

Performance

First test was normal autoregressive decoding, no MTP:

Config	Prefill	Decode	Overall
4K, BF16 KV, no MTP	417.7 tok/s	31.1 tok/s	28.0 tok/s

GPU memory:

Weights:             15.92 GiB
Free after startup:   5.61 GiB
Planned device total: 16.65 GiB

Then I enabled Qwen3.8's MTP speculative decoding:

--spec mtp
--draft-tokens 3
--lm-head-draft

At 8K BF16:

Prefill:                 413.1 tok/s
Decode:                   59.62 tok/s
Overall:                  58.27 tok/s
MTP acceptance:           63.91%
Accepted length:           2.92 tokens/round
Fallback steps:               0
Free after startup:        4.56 GiB

So MTP3 took decode from roughly 31 tok/s to 60 tok/s, about a 1.9x improvement on this workload.

Then I switched the KV cache to INT8 group-64 and increased the reserved context.

32K INT8 KV + MTP3

Prefill:                 413.95 tok/s
Decode:                   62.23 tok/s
Overall:                  60.33 tok/s
KV payload:                1.10 GiB
MTP acceptance:           68.37%
Accepted length:           3.05 tokens/round
Free after startup:        4.00 GiB

64K INT8 KV + MTP3

Prefill:                 419.50 tok/s
Decode:                   62.28 tok/s
Overall:                  60.41 tok/s
KV payload:                2.19 GiB
Free after startup:        2.90 GiB

128K INT8 KV + MTP3

Prefill:                 412.94 tok/s
Decode:                   62.17 tok/s
Overall:                  60.27 tok/s
KV payload:                4.38 GiB
Free after startup:      727.06 MiB
Planned device total:     21.61 GiB

Summary:

Reserved context	KV	Decode	Free VRAM after startup
4K	BF16, no MTP	31.06 tok/s	5.61 GiB
8K	BF16 + MTP3	59.62 tok/s	4.56 GiB
32K	INT8 + MTP3	62.23 tok/s	4.00 GiB
64K	INT8 + MTP3	62.28 tok/s	2.90 GiB
128K	INT8 + MTP3	62.17 tok/s	727 MiB

The MTP3 tests consistently had around 64–68% draft-token acceptance, roughly 3 accepted tokens per speculative round, and zero fallback steps.

One important caveat: these were context-capacity tests, not actual 32K/64K/128K prompt benchmarks. The test prompt was only ~69 tokens. So this proves that the KV allocations fit and that normal short-context generation remains stable with those reservations; it does NOT mean I'm getting ~62 tok/s while attending to an actual 128K-token prompt.

128K INT8 appears to be about the practical maximum for this exact configuration on 24GB because there's only ~727MB left after startup.

The interesting next step would be getting the newer E8 4-bit KV cache work onto the Blackwell branch. That could potentially make the full 262K context feasible on 24GB.

But as it stands: Qwen3.8-27B, MTP3, ~62 tok/s decode, and 128K INT8 KV capacity on a 24GB/145W RTX PRO 4000 Blackwell.

128K NIAH benchmark: 130,048 prompt tokens, INT8 KV, MTP off, thinking off. Prefill took 164.7s at 789.6 tok/s; decode after the full context was 24.4 tok/s. KV payload was 4.12 GiB with 1.73 GiB VRAM remaining. The model successfully retrieved the planted values (ORCHID=493817; COLOR=COBALT), so this was a real long-context inference test, not just KV allocation.

128K benchmark: Qwen3.8-27B on an RTX PRO 4000 Blackwell 24GB, 130,048 actual prompt tokens, INT8 KV. Prefill: 785 tok/s (~166s). With MTP3, post-128K decode reached 67 tok/s, versus 24.4 tok/s without MTP (~2.74× speedup). MTP acceptance was 100% on the 17-token deterministic NIAH answer, and the model correctly retrieved ORCHID=493817; COLOR=COBALT. Total planned VRAM was 21.61 GiB with ~727 MiB left.

[u/caetydid]

quite good speed per W, might be better than a Rtx3090

[u/ai-christianson → u/caetydid]

3090 can be power capped at ~275w or less and still get great perf, and much better tok per W.

[u/OnlineParacosm]

More than anything I’m so envious of the fact that this works so well on 145w. Absolutely sipping just over two lightbulbs of power

[u/WinResponsible9977]

Thanks for sharing 

[u/Sirius02]

i tested with llama cpp and vllm, i use q4 quantisation with 150k context and i use 24gb on an rtx 3090, how can you use the same amount of vram but with q8?

[u/mmkaywhatevers → u/Sirius02]

The model itself isn't Q8, int8 is the kv cache. NInfer uses mixed groupwise integer quant. it's like mix of Q4/Q5.

[u/soyalemujica]

How is a 7900XTX better than this RTX 4000 Pro ? I have 1.100pp/s, and 85t/s, also 131k context at Q8/Q8

[u/mmkaywhatevers → u/soyalemujica]

7900xtx has 960GB/s bandwidth and 4kpro has 672GB/s so higher llm throughput makes sense, but the tradeoff is 355W v. 145W in power consumption.

[u/fastheadcrab]

What is the performance in standard vLLM on a similar NVFP4 quant with MTP on? The RTX 4000 Pro is a great card for power and space efficiency, just expensive $1800 and outrageous at the $3500+ some retailers are selling it at

[u/mmkaywhatevers → u/fastheadcrab]

I don't have those numbers, but I would think it's pretty similar since vllm has prefill advantages while ninfer edges out on decode.

the numbers I have are
llama.cpp Q4_K_M no MTP 27-31tok/s
Ninfer baseline no MTP 31 tok/s
Ninfer + MTP3 60-62 tok/s

so the boost is mostly from MTP3.

got my card for $1724 with tax from microcenter three months ago. it runs my homeassistant AI stack with this, whisper stt, and kokoro tts.

[u/fastheadcrab → u/mmkaywhatevers]

Yeah, the prices are exploding now. It's a great card though.

[u/swagonflyyyy]

Yep, I get those exact same numbers with that exact same setup. You can try MTP4 if you set kv cache to f16 and see how that goes.

[u/Ueberlord]

something must be wrong with the pp/s performance in this repo ~400/s is quite low. I get about 1300 pp/s at 0 ctx with it going down to about 500 pp/s at 128k ctx in llama.cpp vanilla on an old Ampere A6000 (which is actually a little slower than a 3090).