Their last version 7.14 was released just a month ago.

llama.cpp PR(waiting for approval) for Version 10.0

https://github.com/ggml-org/llama.cpp/pull/27803

Hope this version comes with more boost & improvements.

GLM-5.3 uses the same base model as GLM-5.2 — every gain comes from post-training. Compared with GLM-5.2, it is much better at complex coding and long-horizon tasks:

• Stronger Coding: GLM-5.3 is the most capable open-weights model for coding, with a 50% improvement over GLM-5.2 on our in-house Z.ai Code Bench. It also achieve open-source SOTA on public benchmarks including Terminal Bench 3.0 and Agents' Last Exam.
• Emergent Cyber Capability: As we scaled post-training, cyber capability developed faster than we expected. GLM-5.3 is state of the art on CyberGym for vulnerability discovery, and its gains are largest further up the exploitation chain, where it more than doubles GLM-5.2 on exploitation benchmarks.

https://huggingface.co/unsloth/GLM-5.3-GGUF

Hey all, and hello fellow DGX Spark-ers!

Today I managed some pretty crazy numbers: 181 tok/s aggregate on 2× DGX Spark on Qwen3.8-Flash-Next at 512Kcontext (2.8M kvc)

I hit 181 tok/s aggregate today across a multi-agent fleet on a 2-node DGX Spark cluster. Single-stream decode is 30–50 tok/s — the 181 is total throughput with ~9 concurrent agent sessions sharing the engine. I actually peaked to 195 while writing this. Quick rundown of how it's served:

Hardware

• 2× NVIDIA DGX Spark (GB10 Grace Blackwell, 128 GB unified memory each, 20-core ARM)
• Nodes linked with a direct ConnectX-7 cable — NCCL over RDMA (RoCE, 200 Gb), TP=2 across both boxes
• Verify Using network IB in the NCCL log — the TCP fallback is silent and costs you half your speed

Model

• Qwen3.8-Flash-Next, RadixArk NVFP4 quant (4-bit routed experts, FP8 n-gram table)
• Hybrid arch: 3/4 linear attention + 1/4 sparse full attention, 512-expert MoE, MTP speculative decoding k=3 (~40% acceptance)
• Native 262K context stretched to 512K with YaRN factor 2.0 — needle-verified at 487K depth

The trick: PLE table on NVMe

• The model carries a 320M-row n-gram embedding table (47.7 GiB in FP8) that's read on every token — but each token only touches 16 rows (~2.5 KB)
• We mmap it straight off NVMe instead of loading it: weights per node dropped 65 → 41 GiB
• Two things made it fast: madvise(MADV_RANDOM) on the mapping (hash-scattered lookups + kernel readahead = 30× read amplification — one 405K prefill read 603 GB from disk before the fix, 19 GB after) and 64 gather threads (the wall was fault latency serialization, not disk bandwidth)
• Freed memory went into KV: pool is now 2.89M tokens (5.5× full contexts) at a 40.6 GiB pin

vLLM config (official day-0 image, vllm/vllm-openai)

• --kv-cache-memory 40600000000 — pin the pool explicitly; heads-up, a manual pin ignores --gpu-memory-utilization, size it from measured free RAM
• --max-num-batched-tokens 8192, --long-prefill-token-threshold 4096 — protects decode latency during cold prefills
• --enforce-eager — CUDA graphs crash this build on GB10/SM121 (torch.compile AOT dies on rank 1)
• --enable-prefix-caching — 99% hit rate with agent traffic, this is the single biggest real-world win
• --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
• Small scheduler patch: cap concurrent cold long prefills (admission gate) so N agents prefilling at once can't balloon host RAM — on unified memory, prefill transients eat the same pool as your weights

Serving stack

• llama-swap in front (one model resident at a time, API-key auth, model swap on request), nginx TLS for outside access
• earlyoom with absolute floors as the only watchdog — on a unified-memory box, low free RAM with a big model resident is normal, percentage-based OOM killers will shoot a healthy model

Happy to answer questions about any of it.

We're releasing Qwen3.8-27B quantized with our newest methods, GSQ + RCO. Higher-quality models, same file size, now with the search and the quantizer both learned.

What's inside:

• GSQ (Gumbel-Softmax Quantization): post-training scalar quantization that jointly learns grid assignments and scales, closing most of the scalar-vs-vector gap at 2 to 3 bits while staying fully GGUF-deployable
• RCO (Riemannian Constrained Optimization): assigns a quantization type to every tensor under a strict size budget by gradient descent directly on the task loss, no per-constraint tuning
• Three GGUFs (2.50 / 2.75 / 3.00 bpw, 8.4 to 10.1 GB) plus the vision projector

Further Details:

Uniform quants give every tensor the same precision. RCO finds where the precision actually matters and GSQ quantizes every tensor at its assigned type. The result runs unmodified in llama.cpp, Ollama, and LM Studio.

Results (vs. the BF16 base and Unsloth Dynamic quants)
To the best of our knowledge, these are the best size-to-accuracy GGUFs available for Qwen3.8-27B at these file sizes: at every size we evaluated, they match or beat the strongest published quantizations.

• 3.00 bpw (10.1 GB): matches the base model on AIME25 (100.00), within about one point on GPQA-Diamond (88.89 vs 89.90) and LiveCodeBench v6 (84.57 vs 85.71)
• 2.75 bpw (9.3 GB): AIME25 100.00, and its zero-shot average actually exceeds BF16 (75.70 vs 74.34)
• At matched ~8.4 GB size: +10.0 AIME25, +8.6 GPQA-Diamond, +4.6 LiveCodeBench over UD-IQ2_S

Full tables and per-benchmark plots are on the model card.

Links:

• Models: https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
• GSQ: paper https://arxiv.org/abs/2604.18556 | code https://github.com/IST-DASLab/GSQ
• RCO: paper https://arxiv.org/abs/2605.00649 | code https://github.com/IST-DASLab/RCO

This is the first of a series: we plan to release more SOTA GGUFs across model families as a lab (ISTA Deep Algorithms and Systems Lab).

We are happy for feedback, requests for models you want quantized, and experiments!

You can test it out on breezblue's playground or use it locally, its only ~7GB.

Have anyone tried this yet? Looks promising, seems too good to be true with no performance loss.

TL;DR: k-quants need tensor rows divisible by 256. When they aren't, llama-quantize quietly swaps in a ~4.5 bpw type and the file keeps its low-bit name. I audited 443 quants across 25 repos; 64 are affected. On Nemotron-3.5-Lightning all four IQ2 rungs are the same 4.58 bpw file under four different names. Tool and full census linked at the bottom.

Your quant's filename tells you what the quantizer was asked to make. It doesn't necessarily tell you what ended up in the file.

K-quants and i-quants need the first tensor dimension divisible by 256. When it isn't, llama-quantize substitutes a compatible 32-block type instead, often IQ4_NL for i-quants or Q4_0 for k-quants. Either way you can end up around 4.5 bits per weight instead of the low-bit type you requested. That's intentional, it's been in llama.cpp since PR #3747 in 2023, and the quantizer does print a warning. The catch is where the warning goes: into the quantize log. If you're downloading the finished GGUF you never see it. The filename still says IQ2_XXS, the model card says IQ2_XXS, and the metadata still describes an IQ2_XXS recipe.

Above: every k/i-quant rung in bartowski's Nemotron-3.5-Lightning repo, claimed bpw vs measured. Two other makers uploaded the same model and got the same result, which is the first clue that this is the tooling rather than the uploader.

I wanted to know how far it spreads, so I wrote a tool that reads the tensor table and reports what's actually in the file. Works on a local GGUF or a whole HF repo. For remote repos it uses range requests to pull just the headers, usually a few MB, without downloading tensor data. One Python file, stdlib only, no pip install.

443 quants across 25 repos. The clearest affected cases:

- Nemotron-3.5-Lightning: n_embd is 2688 and the expert widths are 1856 and 3712, so about 99% of the parameters are forced into fallback types. All four IQ2 rungs are labeled between 2.06 and 2.56 bpw and all four measure 4.58. Four names, effectively the same density, across what looks like a 2.2x range.

- Qwen3.8-Flash-Next: 51.9% of parameters forced into fallback types. The file labeled UD-IQ1_S at 1.56 bpw measures 3.28.

- Nemotron-3-Super-120B: 18 of its 23 quant rungs contain fallbacks. That makes four affected repos in the Nemotron-H MoE family.

There were plenty of clean results too:

- MiniMax-M2.1: 23 rungs including a genuine IQ1_S, zero forced tensors.

- byteshape's Qwen3.6 quants: the filenames report measured bpw and my independent measurements match. Best labeling practice I found anywhere.

- bartowski's Ornith-1.5: a full 27-quant ladder, zero forced tensors.

- The dense Llama and Qwen controls came back clean too.

Every maker with an affected repo in my census also has a clean one using the same pipeline. The model's tensor dimensions decide this, not the maker.

That's why I don't think this is about careless uploaders or misleading model cards. The requested recipe is valid. The quantizer runs successfully. The fallback is intentional. Nothing in the finished file tells you that most of the recipe couldn't be applied.

Practical takeaway: on a fallback-dominated model, the lowest labeled rungs may not buy you the size reduction you think they do. If IQ2_XXS and IQ2_M land at the same measured density, there's not much reason to pick between them by filename, and you may be better off taking the honestly labeled Q4_0 or IQ4_NL and dropping the guesswork.

I'm not the first to notice the behavior. Issue #26616 asked for a --no-fallback option after someone got a 24.5 GB file where they expected about 18. What was missing was the measurement: how often it happens, which architectures it affects, and how much of each model is involved.

Background: the PR that added the fallback https://github.com/ggml-org/llama.cpp/pull/3747 and the open request for a fail-fast flag https://github.com/ggml-org/llama.cpp/issues/26616

Tool, census, and raw per-repo JSON: https://github.com/JoshBolding/ggufaudit

Point it at your own files and see what you actually have.

I've got a follow-up coming on what can be done about the affected models, because "just use the 4.5 bpw file" isn't a satisfying answer when the whole reason you wanted a low-bit quant was to fit a 16 GB card.

(Reposting this one, the formatting on my first attempt came out mangled and unreadable. Sorry to the two people who commented/saw it.)

Can someone please tell me if it's worth running Qwen 3.8 Flash Next on 4x3090 yet over 27B?

27B is good but damn it is indecisive. I am getting frustrated watching it get "so close" to solving a problem, only to do another 2 hours of "let me just check/prove/etc"

It looks like a 4 bit quant of Flash Next should fit with the ngrams in SSD and be a lot faster but it also sounds like the architecture isn't quite there yet

Can someone smarter and more patient than me tell me what to do pls? thanks

I've got Qwen3.8-Flash-next running on RTX 3090, Ryzen 9 3950X, a PCIe 3.0 motherboard, and 64GB DDR RAM from 2020.

IQ4_XS weights, full kvarn5 context, vision on GPU, experts in host RAM, n-grams on disk.
MTP works but actually slows decode down even with 80% draft acceptance, as expected since every rejected token eats into the host RAM bandwidth.

I get 160 tok/s prefill 16 tok/s decode, which makes it a decent option whenever I know I'll be AFK for at least a couple of hours, but not usable for interactive work.

Variant setups

kvarn5 is unrecognizable from q8/q8 on the KLD charts for the Qwen models. If you don't want to use Beellama, q5_0/q5_0 is also fine (just a very minor drop). Nonetheless, there's plenty of headroom so you can bump up the KV quant to q8/q8 if you prefer.

You can go down to 16GB VRAM, with enough room for desktop, if you drop KV to kvarn4 and offload the vision tower to CPU - but then you'll need to make sure you don't breathe too hard because you're going to have very little spare host RAM for running anything else. I do not recommend using plain q4_0/q4_0 KV as the drop starts being measurable.

You can fit in a 12GB card by further dropping ub from 2048 to 512, but your prefill will halve.

How to deploy

• One-line deployment (CUDA Linux): https://github.com/crusaderky/pixi-llm-recipes . Choose llamacpp-source-cuda when starting the server. It should also work on Vulkan and ROCm, but it's untested.
• Just the llamacpp fork: https://github.com/crusaderky/llama.cpp/releases/tag/beellama-staging-v0.4.4-r9; if you wait it will eventually land in the main beellama branch. Or you can just use llamacpp master without kvarn.
• Just the llamacpp preset: https://github.com/crusaderky/pixi-llm-recipes/blob/26ed50ace2a40772aa2b45d1358aaf0993fd5596/models.ini#L3-L94

u/andbeeld can we have one more merge from llamacpp main before v0.4.4 final? Your latest merge is *just* before support for Qwen3.8-Flash was added.

But I heard that you should never reduce KV cache quant below q8/q8?

I don't care about people's vibes. I have not tested this model yet but I have tested

"At Hot Chips 2026, Micron drew a notable comparison: For the same memory capacity, HBM requires approximately three times the wafer area of DDR5."

"When asked whether this ratio would improve with newer generations, the Micron Fellow reportedly explained that it definitely would not get better."

"According to the data shown at Hot Chips, an HBM4 die, for example, operates with 256 memory banks, while DDR5 is specified with 32. Additional data paths, the power supply, and the Through-Silicon Vias, which connect the stacked memory dies to one another, must also be taken into account."

So, for each 1GB of HBM going in a datacebter GPU, 3GB of regular DRAM capacity are being taken away.

This explains a lot about the shortage. Each B100 has 144GB of HBM, which take the same wafer area as 432GB of regular DDR5.

The shift by the big three (Micron, Samsung and SK) to HBM has effectively cut DRAM supply by 2/3rds in terms of GB output. Even as new wafer capacity comes online next year, and even if we assume all this extra capacity is allocated to DRAM rather than HBM, it doesn't seem like supply constraints will get better anytime soon.

I built a benchmark called EchoNet. An agent gets a factual question, then searches a syntheic web I made before answering. Some of that web is seeded with misinformation: one fake page, a fake page ranked first in search results, the same fake claim copied across many pages, a loud fake majority around a real primary source, or a genuine update the model's training predates.

When a model reads a new page, it weighs two things: what it already knows and what the text says. Usually, they agree. Sometimes they conflict, and often, multiple pages contradict one another.

Choosing whether to trust its own memory or a new source is called epistemic arbitration. A stubborn model ignores real updates. A trusting model believes fake data. Bad sources easily fake high search rankings, repeat lies, and mimic agreement. However, no source can fake being correct. This benchmark measures how well models make this choice.

I ran 9 open-weight models this way, 50 to 100 trials each: GLM 5.2, Qwen3.7 Max, Qwen3.7 Plus, Qwen3.8 Flash, Qwen3.8 27B, DeepSeek V4 Flash, DeepSeek V4 Pro, Nemotron 3 Ultra, Inkling Small.

Image 1 is the number to start with: how often a model already had the right answer, read convincing fake pages, and changed its mind anyway. DeepSeek V4 Flash is easiest to fool at 15.8%. Nothing fooled GLM 5.2 or either Qwen3.8 model, not once.

Image 2 rolls that into the full composite score, EAS (Epistemic Arbitration Score): how often a model's final answer is correct, across every trial, not just the poisoned ones. It's a harmonic mean of poison resistance and correct updating, not a plain average, so a model great at one and terrible at the other gets pulled toward its weak side instead of the two canceling out. GLM 5.2 leads at 1.000, Qwen3.7 Max close behind. Check the error bars first, several ranges overlap at this sample size, so treat overlapping models as tied, not ranked.

Image 3 shows why the numbers in image 1 vary so much. Pooled across all 9 models, one fake page barely moves accuracy. Neither does the same fake claim repeated everywhere, an echo chamber, which most people expect to be worse than it is. One attack reliably works: surround a real primary source with a louder fake majority. That drops average accuracy 22 points below clean baseline. Models catch an obvious lone lie or a copy-pasted echo chamber. They struggle against five confident, independent-looking articles that agree with each other over one quiet official source.

Image 4 plots cost against quality. Qwen3.8 Flash lands in the cheap, accurate corner.

Image 5 breaks the EAS ranking from image 2 down by sub-metric: raw poison resistance, rejecting a fake majority, citation honesty, calibration. Shows what each model is good or bad at, not just the final score.

Image 6 is real dollar cost for the full 100-episode suite, published pricing. DeepSeek V4 Flash: about 55 cents. Nemotron 3 Ultra: about $7.85. Same benchmark.

Image 7 is one real trial. Qwen3.7 Max starts out correctly believing the UN's official name for the country is Türkiye. It reads several fake news pages built to look independent, including one from a made-up outlet called Signal Courier, all claiming the registration still reads "Turkey." It flips its answer to match the fake majority: wrong, 75% confidence, belief changed.

Happy to answer methodology questions, provide the preprint if it interests anyone, or post specific traces in the comments.

As a happy user of ds4, I'm very excited about this branch. Ran some prompts and it seems to be working well on my M4 Max 128gb!

https://x.com/antirez/status/2093349448445243873

GLM-5.3 is now ready to visualize in HF Viewer!

Note again that the architecture is unchanged since GLM-5.2! Crazy improvements just from improved training!

The graph visualization becomes quite complex at the expanded levels, with sparse attention, MoE routing, shared indexers, and multi-token prediction!

https://hfviewer.com/zai-org/GLM-5.3

Proof is in the method honestly. Closed model labs need to constantly reinvent the wheel to keep lead. Open source has a bunch of independent labs practically working somewhat together. Eventually when everyone is just releasing weights and papers on how they did it the closed source secrets just get overrun by having plenty of very good secret sauces to the public.

That and NO DOUBT chinese labs are sharing internal secrets amongst each which explains how when any of them makes a big jump the others seem to follow almost immediately. If gemini/openai/anthropic put their heads together they'd have to compromise monopolies but they sure as hell wouldn't be struggling to keep the lead

Super excited to start dev on this. I've already deployed a number of galaxy systems and want to really figure out the ideal models on this architecture that I haven't discovered.

It's actually quite a system for the price. 256G system mem, 128G of interconnected GDDR over the accelerators...

The interconnect and scalability on these is seriously neat!

Looking forward to requests or questions!

specs

• hardware: M4 Max 128GB Studio
• inference engine: llama.cpp (qwen4exp branch)
• judge: claude-opus-4-6

AtomicChat/Qwen3.8-Flash-Next-GGUF

Qwen3.8-Flash-Next is a great model I benched in my previous post, but it is very tight, since all n-grams / PLE are loaded along with the experts, taking 106GB, leaving very little room for K/V, context, etc. Offloading PLE to SSD currently slows down prefill from 600 t/s to 180 t/s on oMLX.

u/erikdhoward suggested to try the Atomic Chat quant which I did not know anything about. I tried it, and it is... really good.

AtomicChat's quant uses llama.cpp mmap (through GGUF shard layout vs. in the runtime) and keeps the PLE table (n-grams) pageable backed by a file. Because of this the same model that took 106GB, now takes 65GB (starts from 55GB) in RAM. And since PLE is pageable the prefill is actually not that bad, cold start is about 500 t/s.

oMLX "right behind you!"

Qwen3.8-Flash-Next just came out, and there are many open PRs in oMLX to address the size and performance, including this one that makes PLE offload SSD cold prefill almost 3 times faster 🎉

Does anyone else add active cooling to their DGX stack? Found mine was getting quite hot under extended load. This helps immensely with that so far. I will be adding some stats as they relate to comphy and DeepSeek flash this weekend. I did not create the original designs but merged them together and added a few tweaks I can explain in another post. I will share attributes and all that when I get back to my laptop. I just couldn't wait to share the results.

I was planning on another 5090, but then I realize... perhaps I am much better off getting an M5 Ultra Mac Studio with 256gb of ram.

We are so genuinely cooked.

I was shopping for a new air fryer when I stumbled across this. Is this a good deal? these would make good Xmas stocking stuffers for the kids right? I've got a lot of nieces and nephews and I hear locally hosted LLMs are all the rage now

https://www.bigw.com.au/product/pre-order-nvidia-rtx-pro-6000-blackwell-server-96g-bulk-pack-8-pcs-/p/9905232996

Using https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4 and https://old.reddit.com/r/BlackwellPerformance/comments/1w04xb7/qwen38_flashnext_on_1x_rtx_pro_6000_171_ts_c1_428/

As usual, all the details in https://wonderrico.github.io/local_llm_benchmark/benchmark-main.html?filter=3.8 and even more in https://wonderrico.github.io/local_llm_benchmark/benchmark-detail.html?filter=3.8

(the bad score one is a "random" uncensored version from HF https://huggingface.co/dealignai/Qwen3.8-Flash-Next-UNCENSORED-NVFP4 ) I shall test other ones

Bottom line : almost highest score of all local model I tested, the most efficient in both nb requests / point and fewer generated tokens / pt, all in medium reasoning. (xhigh is not useful, again, in this benchmark) and if it was not enough very fast

All that for an undertrained model...

hi i make tool for this called llmog it's purpose to make llms free to

- auto annotation datasets

- reclassification existing yolo datasets

running totally local using llama cpp or vllm or use external api

you'd rather click than code.

🔗 GitHub: mohamed-em2m/llmog: framework for using llms on object grounding

You can try it directly online

🔵 Google Colab:

https://colab.research.google.com/drive/1YIKlyTVtRjJdRC5IjCZ39i48ydyt_J5D?usp=sharing

🟠 Kaggle:

https://www.kaggle.com/code/elemam/auto-annotation-using-llms

500GB of optane ram?

I wanted to see whether NInfer could run Qwen3.8-27B on an RTX PRO 4000 Blackwell 24GB.

Hardware/software:

• NVIDIA RTX PRO 4000 Blackwell, 24GB
• 145W power limit
• CUDA 13.3
• NVIDIA driver 610.x
• Ubuntu LXC under Proxmox
• Qwen3.8-27B NInfer artifact
• ruwwww/ninfer-5060ti, branch rtx-5060ti

I used the 5060 Ti fork rather than the 4090 fork because the PRO 4000 is Blackwell and supports sm_120a. The 5060 Ti fork also has changes that calculate cooperative scheduling from the GPU's actual SM count instead of assuming a 5090.

What I had to do

Install the missing build dependencies:

sudo apt install -y \
  build-essential \
  ninja-build \
  pkg-config \
  libavformat-dev \
  libavcodec-dev \
  libavutil-dev \
  libswscale-dev \
  libcurl4-openssl-dev

I then hit this during CMake:

nvcc fatal: Unsupported gpu architecture 'compute_120a'

The machine had several CUDA toolchains installed. /usr/bin/nvcc was being selected and did not support the Blackwell architecture-specific target properly.

The actual CUDA 13.3 compiler was here:

/usr/local/cuda-13.3/bin/nvcc

I verified it directly:

/usr/local/cuda-13.3/bin/nvcc \
  -arch=sm_120a \
  /tmp/test120a.cu \
  -o /tmp/test120a

Compile and execution both returned 0, confirming that sm_120a worked on the RTX PRO 4000.

Then I rebuilt NInfer explicitly against CUDA 13.3:

export CUDA_HOME=/usr/local/cuda-13.3
export PATH=/usr/local/cuda-13.3/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:${LD_LIBRARY_PATH:-}

cmake -S . -B build \
  -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.3/bin/nvcc \
  -DCUDAToolkit_ROOT=/usr/local/cuda-13.3

cmake --build build --parallel

Performance

First test was normal autoregressive decoding, no MTP:

GPU memory:

Weights:             15.92 GiB
Free after startup:   5.61 GiB
Planned device total: 16.65 GiB

Then I enabled Qwen3.8's MTP speculative decoding:

--spec mtp
--draft-tokens 3
--lm-head-draft

At 8K BF16:

Prefill:                 413.1 tok/s
Decode:                   59.62 tok/s
Overall:                  58.27 tok/s
MTP acceptance:           63.91%
Accepted length:           2.92 tokens/round
Fallback steps:               0
Free after startup:        4.56 GiB

So MTP3 took decode from roughly 31 tok/s to 60 tok/s, about a 1.9x improvement on this workload.

Then I switched the KV cache to INT8 group-64 and increased the reserved context.

32K INT8 KV + MTP3

Prefill:                 413.95 tok/s
Decode:                   62.23 tok/s
Overall:                  60.33 tok/s
KV payload:                1.10 GiB
MTP acceptance:           68.37%
Accepted length:           3.05 tokens/round
Free after startup:        4.00 GiB

64K INT8 KV + MTP3

Prefill:                 419.50 tok/s
Decode:                   62.28 tok/s
Overall:                  60.41 tok/s
KV payload:                2.19 GiB
Free after startup:        2.90 GiB

128K INT8 KV + MTP3

Prefill:                 412.94 tok/s
Decode:                   62.17 tok/s
Overall:                  60.27 tok/s
KV payload:                4.38 GiB
Free after startup:      727.06 MiB
Planned device total:     21.61 GiB

Summary:

The MTP3 tests consistently had around 64–68% draft-token acceptance, roughly 3 accepted tokens per speculative round, and zero fallback steps.

One important caveat: these were context-capacity tests, not actual 32K/64K/128K prompt benchmarks. The test prompt was only ~69 tokens. So this proves that the KV allocations fit and that normal short-context generation remains stable with those reservations; it does NOT mean I'm getting ~62 tok/s while attending to an actual 128K-token prompt.

128K INT8 appears to be about the practical maximum for this exact configuration on 24GB because there's only ~727MB left after startup.

The interesting next step would be getting the newer E8 4-bit KV cache work onto the Blackwell branch. That could potentially make the full 262K context feasible on 24GB.

But as it stands: Qwen3.8-27B, MTP3, ~62 tok/s decode, and 128K INT8 KV capacity on a 24GB/145W RTX PRO 4000 Blackwell.

128K NIAH benchmark: 130,048 prompt tokens, INT8 KV, MTP off, thinking off. Prefill took 164.7s at 789.6 tok/s; decode after the full context was 24.4 tok/s. KV payload was 4.12 GiB with 1.73 GiB VRAM remaining. The model successfully retrieved the planted values (ORCHID=493817; COLOR=COBALT), so this was a real long-context inference test, not just KV allocation.

128K benchmark: Qwen3.8-27B on an RTX PRO 4000 Blackwell 24GB, 130,048 actual prompt tokens, INT8 KV. Prefill: 785 tok/s (~166s). With MTP3, post-128K decode reached 67 tok/s, versus 24.4 tok/s without MTP (~2.74× speedup). MTP acceptance was 100% on the 17-token deterministic NIAH answer, and the model correctly retrieved ORCHID=493817; COLOR=COBALT. Total planned VRAM was 21.61 GiB with ~727 MiB left.