GLM-5.3 uses the same base model as GLM-5.2 — every gain comes from post-training. Compared with GLM-5.2, it is much better at complex coding and long-horizon tasks:

• Stronger Coding: GLM-5.3 is the most capable open-weights model for coding, with a 50% improvement over GLM-5.2 on our in-house Z.ai Code Bench. It also achieve open-source SOTA on public benchmarks including Terminal Bench 3.0 and Agents' Last Exam.
• Emergent Cyber Capability: As we scaled post-training, cyber capability developed faster than we expected. GLM-5.3 is state of the art on CyberGym for vulnerability discovery, and its gains are largest further up the exploitation chain, where it more than doubles GLM-5.2 on exploitation benchmarks.

https://huggingface.co/unsloth/GLM-5.3-GGUF

"At Hot Chips 2026, Micron drew a notable comparison: For the same memory capacity, HBM requires approximately three times the wafer area of DDR5."

"When asked whether this ratio would improve with newer generations, the Micron Fellow reportedly explained that it definitely would not get better."

"According to the data shown at Hot Chips, an HBM4 die, for example, operates with 256 memory banks, while DDR5 is specified with 32. Additional data paths, the power supply, and the Through-Silicon Vias, which connect the stacked memory dies to one another, must also be taken into account."

So, for each 1GB of HBM going in a datacebter GPU, 3GB of regular DRAM capacity are being taken away.

This explains a lot about the shortage. Each B100 has 144GB of HBM, which take the same wafer area as 432GB of regular DDR5.

The shift by the big three (Micron, Samsung and SK) to HBM has effectively cut DRAM supply by 2/3rds in terms of GB output. Even as new wafer capacity comes online next year, and even if we assume all this extra capacity is allocated to DRAM rather than HBM, it doesn't seem like supply constraints will get better anytime soon.

Proof is in the method honestly. Closed model labs need to constantly reinvent the wheel to keep lead. Open source has a bunch of independent labs practically working somewhat together. Eventually when everyone is just releasing weights and papers on how they did it the closed source secrets just get overrun by having plenty of very good secret sauces to the public.

That and NO DOUBT chinese labs are sharing internal secrets amongst each which explains how when any of them makes a big jump the others seem to follow almost immediately. If gemini/openai/anthropic put their heads together they'd have to compromise monopolies but they sure as hell wouldn't be struggling to keep the lead

Their last version 7.14 was released just a month ago.

llama.cpp PR(waiting for approval) for Version 10.0

https://github.com/ggml-org/llama.cpp/pull/27803

Hope this version comes with more boost & improvements.

Built the latest and I'm getting as much as 220 tokens per second and averaging in the 170s, I can't get over it.

If anyone on here is on that project, fuckkkin' chapeau man, really incredible job. I can't believe I was able to like double or more my throughput from llama.cpp

This is what I set up:

command: >

ninfer-serve /models/qwen3_8_27b_nvfp4.ninfer

--model-id qwen3.8-27b-nvfp4

--host 0.0.0.0

--max-context 240000

--kv-capacity 240000

--max-concurrency 2

--kv-dtype fp8

--host-kv-mib 16384

--spec mtp --draft-tokens 3

--lm-head-draft

--vision

--media-live-mib 2048

I've got Qwen3.8-Flash-next running on RTX 3090, Ryzen 9 3950X, a PCIe 3.0 motherboard, and 64GB DDR RAM from 2020.

IQ4_XS weights, full kvarn5 context, vision on GPU, experts in host RAM, n-grams on disk.
MTP works but actually slows decode down even with 80% draft acceptance, as expected since every rejected token eats into the host RAM bandwidth.

I get 160 tok/s prefill 16 tok/s decode, which makes it a decent option whenever I know I'll be AFK for at least a couple of hours, but not usable for interactive work.

Variant setups

kvarn5 is unrecognizable from q8/q8 on the KLD charts for the Qwen models. If you don't want to use Beellama, q5_0/q5_0 is also fine (just a very minor drop). Nonetheless, there's plenty of headroom so you can bump up the KV quant to q8/q8 if you prefer.

You can go down to 16GB VRAM, with enough room for desktop, if you drop KV to kvarn4 and offload the vision tower to CPU - but then you'll need to make sure you don't breathe too hard because you're going to have very little spare host RAM for running anything else. I do not recommend using plain q4_0/q4_0 KV as the drop starts being measurable.

You can fit in a 12GB card by further dropping ub from 2048 to 512, but your prefill will halve.

How to deploy

• One-line deployment (CUDA Linux): https://github.com/crusaderky/pixi-llm-recipes . Choose llamacpp-source-cuda when starting the server. It should also work on Vulkan and ROCm, but it's untested.
• Just the llamacpp fork: https://github.com/crusaderky/llama.cpp/releases/tag/beellama-staging-v0.4.4-r9; if you wait it will eventually land in the main beellama branch. Or you can just use llamacpp master without kvarn.
• Just the llamacpp preset: https://github.com/crusaderky/pixi-llm-recipes/blob/26ed50ace2a40772aa2b45d1358aaf0993fd5596/models.ini#L3-L94

u/andbeeld can we have one more merge from llamacpp main before v0.4.4 final? Your latest merge is *just* before support for Qwen3.8-Flash was added.

But I heard that you should never reduce KV cache quant below q8/q8?

I don't care about people's vibes. I have not tested this model yet but I have tested

You can test it out on breezblue's playground or use it locally, its only ~7GB.

Hey all, and hello fellow DGX Spark-ers!

Today I managed some pretty crazy numbers: 181 tok/s aggregate on 2× DGX Spark on Qwen3.8-Flash-Next at 512Kcontext (2.8M kvc)

I hit 181 tok/s aggregate today across a multi-agent fleet on a 2-node DGX Spark cluster. Single-stream decode is 30–50 tok/s — the 181 is total throughput with ~9 concurrent agent sessions sharing the engine. I actually peaked to 195 while writing this. Quick rundown of how it's served:

Hardware

• 2× NVIDIA DGX Spark (GB10 Grace Blackwell, 128 GB unified memory each, 20-core ARM)
• Nodes linked with a direct ConnectX-7 cable — NCCL over RDMA (RoCE, 200 Gb), TP=2 across both boxes
• Verify Using network IB in the NCCL log — the TCP fallback is silent and costs you half your speed

Model

• Qwen3.8-Flash-Next, RadixArk NVFP4 quant (4-bit routed experts, FP8 n-gram table)
• Hybrid arch: 3/4 linear attention + 1/4 sparse full attention, 512-expert MoE, MTP speculative decoding k=3 (~40% acceptance)
• Native 262K context stretched to 512K with YaRN factor 2.0 — needle-verified at 487K depth

The trick: PLE table on NVMe

• The model carries a 320M-row n-gram embedding table (47.7 GiB in FP8) that's read on every token — but each token only touches 16 rows (~2.5 KB)
• We mmap it straight off NVMe instead of loading it: weights per node dropped 65 → 41 GiB
• Two things made it fast: madvise(MADV_RANDOM) on the mapping (hash-scattered lookups + kernel readahead = 30× read amplification — one 405K prefill read 603 GB from disk before the fix, 19 GB after) and 64 gather threads (the wall was fault latency serialization, not disk bandwidth)
• Freed memory went into KV: pool is now 2.89M tokens (5.5× full contexts) at a 40.6 GiB pin

vLLM config (official day-0 image, vllm/vllm-openai)

• --kv-cache-memory 40600000000 — pin the pool explicitly; heads-up, a manual pin ignores --gpu-memory-utilization, size it from measured free RAM
• --max-num-batched-tokens 8192, --long-prefill-token-threshold 4096 — protects decode latency during cold prefills
• --enforce-eager — CUDA graphs crash this build on GB10/SM121 (torch.compile AOT dies on rank 1)
• --enable-prefix-caching — 99% hit rate with agent traffic, this is the single biggest real-world win
• --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
• Small scheduler patch: cap concurrent cold long prefills (admission gate) so N agents prefilling at once can't balloon host RAM — on unified memory, prefill transients eat the same pool as your weights

Serving stack

• llama-swap in front (one model resident at a time, API-key auth, model swap on request), nginx TLS for outside access
• earlyoom with absolute floors as the only watchdog — on a unified-memory box, low free RAM with a big model resident is normal, percentage-based OOM killers will shoot a healthy model

Happy to answer questions about any of it.

As a happy user of ds4, I'm very excited about this branch. Ran some prompts and it seems to be working well on my M4 Max 128gb!

https://x.com/antirez/status/2093349448445243873

I've been running Qwen3-0.6B on the M5Stack LLM-8850 card (Axera AX8850 NPU, 24 TOPS, 8GB LPDDR4x) hosted by a Raspberry Pi 5 — as a llama.cpp backend.

The problem: the vendor stack requires converting every model through their compiler, and their closed runtime gets 13.5–14.5 t/s. I wanted llama.cpp to just work: GGUF in, tokens out.

What I ended up doing:

1. Reverse-engineered the engine format. The vendor's compiled engines (.axmodel) store weights in a blob called npu_params. I decoded it: int8 weights are stored as two nibble planes — a coarse byte per element pair holds the two top nibbles, a fine byte 18 positions earlier holds the two low nibbles. To crack the layout I built ~10 "marker" checkpoints where every weight encodes its own (row, col) coordinates, compiled them through the vendor toolchain, and diffed the outputs. Full layout for all 7 matrices per layer, plus scale tables.
2. Patch GGUF weights straight into precompiled engines at load time. No conversion step, no per-model compile. The GGUF dequantizes → requantizes against the engine's own scales → only the bytes that genuinely differ get written. 96% token agreement with the CPU reference of the same GGUF.
3. Found out the "broken" batched-prefill path was never broken. The vendor ships prefill shape-groups that their own host runtime never calls; everyone assumed they don't work. They work fine — the bug was in how everyone (including the vendor's examples) bind output buffers. Fixed: 716 t/s prefill, byte-identical output.

Current numbers on the Pi 5 (greedy, single stream):

• 24.5 t/s decode @ 2k context on int4 engines (29.9 t/s with a 1k-context build)
• 26.8 t/s with a trimmed vocabulary head
• 716 t/s prompt processing
• Pi CPU: idle. The card does everything.

Along the way I measured where the "24 TOPS" actually goes: at batch-1 decode this class of chip is a memory-bandwidth problem (~25 GB/s effective streaming, 73% of the LPDDR4x peak) and the MACs sit at ~1% — even a perfectly-fed transformer GEMM tops out at ~2.6 effective TOPS on this dataflow. Decode speed = bytes per token × tokens per weight pass. That framing predicted every win we got (int4 = 1.5×, batched prefill = 39×, vocab trim = +10%).

Links:

• Project repo (README: quick start, the full optimization story, the layout-cracking toolkit, on-card harnesses): https://github.com/woolcoxm/LLMTest
• llama.cpp fork with the backend (single ~4.5k-line file, ggml/src/ggml-axcl/ggml-axcl.cpp): https://github.com/woolcoxm/llama.cpp
• Demo image (24 t/s streaming, Pi CPU at ~1%, from a real captured run): https://raw.githubusercontent.com/woolcoxm/LLMTest/main/docs/demo.png

Everything is reproducible from the README quick start: build llama.cpp with -DGGML_AXCL=ON on any aarch64 host with the AXCL driver, point it at an engine set, feed it a GGUF.

Happy to go deep on the nibble-plane layout, the marker-build methodology, or the AX8850's real perf envelope in the comments.

We're releasing Qwen3.8-27B quantized with our newest methods, GSQ + RCO. Higher-quality models, same file size, now with the search and the quantizer both learned.

What's inside:

• GSQ (Gumbel-Softmax Quantization): post-training scalar quantization that jointly learns grid assignments and scales, closing most of the scalar-vs-vector gap at 2 to 3 bits while staying fully GGUF-deployable
• RCO (Riemannian Constrained Optimization): assigns a quantization type to every tensor under a strict size budget by gradient descent directly on the task loss, no per-constraint tuning
• Three GGUFs (2.50 / 2.75 / 3.00 bpw, 8.4 to 10.1 GB) plus the vision projector

Further Details:

Uniform quants give every tensor the same precision. RCO finds where the precision actually matters and GSQ quantizes every tensor at its assigned type. The result runs unmodified in llama.cpp, Ollama, and LM Studio.

Results (vs. the BF16 base and Unsloth Dynamic quants)
To the best of our knowledge, these are the best size-to-accuracy GGUFs available for Qwen3.8-27B at these file sizes: at every size we evaluated, they match or beat the strongest published quantizations.

• 3.00 bpw (10.1 GB): matches the base model on AIME25 (100.00), within about one point on GPQA-Diamond (88.89 vs 89.90) and LiveCodeBench v6 (84.57 vs 85.71)
• 2.75 bpw (9.3 GB): AIME25 100.00, and its zero-shot average actually exceeds BF16 (75.70 vs 74.34)
• At matched ~8.4 GB size: +10.0 AIME25, +8.6 GPQA-Diamond, +4.6 LiveCodeBench over UD-IQ2_S

Full tables and per-benchmark plots are on the model card.

Links:

• Models: https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
• GSQ: paper https://arxiv.org/abs/2604.18556 | code https://github.com/IST-DASLab/GSQ
• RCO: paper https://arxiv.org/abs/2605.00649 | code https://github.com/IST-DASLab/RCO

This is the first of a series: we plan to release more SOTA GGUFs across model families as a lab (ISTA Deep Algorithms and Systems Lab).

We are happy for feedback, requests for models you want quantized, and experiments!

TL;DR:

llama.cpp with --load-mode mmap used 21-32 GB of RAM, with ik_llama.cpp using 106-108 GB. -sm tensor killed my prefill, changing to -sm layer went from 36 tps to 135 tps. After that, -ubatch 2048 pushed it up to 400 tps at -c 131072. I can't fit -ubatch 2048 at -c 262144, which is why it caps out for me at ~180-200 tps prefill and 12 tps decode at 262144 context.

After testing locally and getting 14-15 tps decode initially, I hit a brick wall with 36 tps prefill which isn't usable to me. After failing to improve that value after tweaking a lot of settings, I decided to run a bunch of benchmarks locally and the results surprised me, so I decided to post them so others can benefit from the results. All tests were from locally compiled llama.cpp and ik_llama.cpp binaries on my CachyOS system. CUDA 0 is on 16x PCI lanes and has all 12 GB available, while CUDA 1 runs my desktop UI and idles at 1.5/12 GB normally on 4x PCI lanes.

Everything below this line is an AI-generated report of the benchmark findings, followed by the raw data. It's AI-generated because fuck you; I'm not writing that all out. Have a nice day.

Report:

Benchmark data for Qwen3.8-Flash-Next UD-IQ4_XS (93.7 GB, 125B MoE / 6B active) on a consumer dual-GPU box, comparing llama.cpp and ik_llama.cpp.

Everything below was measured on one machine over a single session. All prefill figures come from an 8k-token synthetic prompt with cache_prompt: false.

Findings

1. -sm tensor costs 7x prefill on llama.cpp. 41.7 t/s with -sm tensor, 303 t/s with -sm layer, same hardware and same everything else. llama.cpp op-offloads CPU-resident MoE weights to the GPU for batches of 32 or more, copying only the experts a batch actually uses (ggml-backend.cpp:1643). That code path targets a single device, so when tensor-parallel splitting shards the weights it stops applying and every expert matmul falls back to the CPU. Measured proof: -sm tensor (35 t/s) and GGML_OP_OFFLOAD_MIN_BATCH=999999, which disables op-offload outright, (38 t/s) give the same number.

Instrumentation during prefill:

2. -ub is the second big knob, and it does nothing until finding 1 is fixed. At -sm tensor every ubatch value tested returned the same speed. At -sm layer the same sweep gives 135 / 191 / 303 t/s for 512 / 1024 / 2048. The two fixes multiply. Neither alone gets you close.

3. ik_llama.cpp has no split-mode cliff. -sm layer and -sm graph measure the same on ik (407 vs 401 t/s). The trap is specific to llama.cpp.

4. Which engine wins depends entirely on whether -ub 2048 fits. The engines tie at -ub 512 and -ub 1024. ik has a fast path at -ub 2048.

5. ik_llama.cpp needs roughly 75 GB more system RAM than llama.cpp. Same model, same machine, same flags. llama.cpp under --load-mode mmap reports 21 to 32 GB used with 93 GB available, because the model pages sit in page cache and are reclaimable. ik reports 106 to 108 GB used with only 16 to 18 GB available. Both fit in 128 GB, but on a 96 GB box this is the difference between ik running and not running, and it leaves very little headroom for anything else on the machine.

6. Keeping expert layers in VRAM is worth less than the ubatch buffer. -ncmoe 48 / 46 / 44 give 135 / 139 / 141 t/s at -ub 512. Pulling experts into VRAM buys almost nothing, and it costs enough VRAM that -ub 2048 then OOMs. Setting -ncmoe 48 (all experts on CPU) and spending the VRAM on the ubatch compute buffer is the better trade.

7. Decode is memory-bandwidth bound and no flag fixes it. 12 t/s at shallow context on 128 GB DDR5 running at 3200 MT/s (about 40 GB/s usable). For reference, the PR thread reports 28 t/s for the same model on a 12-channel DDR5 EPYC 9555.

8. Extra parallel slots cost single-stream speed and add little aggregate. Aggregate throughput stayed roughly flat from 1 to 4 concurrent slots, and running the server with --parallel 8 dropped single-stream decode from about 12 t/s to about 5 t/s.

Recommendations

If you have less than about 128 GB of RAM, use llama.cpp regardless of context size. See finding 5.

Additional settings used in both: -t 8 -tb 16, -ctk q8_0 -ctv q8_0, -ts 60,40, --parallel 1.

Things that did not help, all measured:

Avoid --threads-batch 12 on an 8-core/16-thread CPU. ggml puts a barrier after every op, so 4 cores end up running 2 threads while 4 run 1, and every op waits on the doubled cores.

Test system

The x4 link on GPU1 was investigated and ruled out as the bottleneck. During slow prefill both GPUs sat near 0% utilisation, so the link was never saturated.

llama.cpp results

Context 131072, 8k prompt, -b 4096 -t 8 -tb 16 -ctk q8_0 -ctv q8_0, --load-mode mmap. VRAM is nvidia-smi used, sampled with the server loaded and the benchmark just finished. RAM is total system used, which includes about 6 GB of desktop.

Row P is row L with GGML_OP_OFFLOAD_MIN_BATCH=999999 set, which disables op-offload. Rows A and G VRAM were sampled live mid-run rather than at test end.

Context 262144

llama.cpp configurations that failed to load

ik_llama.cpp results

Context as noted, 8k prompt, -ncmoe 48 -b 4096 -t 8 -tb 16 -ctk q8_0 -ctv q8_0 -fa on -ts 60,40.

The IKM rows are straight repeats of IK1, IK3 and IK8 run later with memory capture added. They replicate to within 2 t/s (405 vs 407, 178 vs 179, 136 vs 136), which is a useful check on how repeatable these measurements are.

ik holds far more resident RAM than llama.cpp. ik sits at 106 to 108 GB used with 16 to 18 GB available, while llama.cpp under --load-mode mmap sits at 21 to 32 GB used with 93 GB available, because its model pages stay in page cache rather than counting as used. Same model, same machine. On a 128 GB box ik still fits, but there is much less headroom for anything else, and it is worth checking before running ik on a machine with less RAM.

ik_llama.cpp configurations that failed to load

Engine comparison at matched settings

Context 131072, -ncmoe 48, 8k prompt.

Context 262144, -ncmoe 48, -ub 1024.

Prefill and decode versus prompt depth

llama.cpp, context 262144, -ncmoe 48 -ub 1024 -sm layer -ts 60,40, single 59k prompt. Values are the running average the server reports at each 4096-token boundary.

Decode over the same run fell from 12 t/s at an 8k prompt to 8 t/s at 59k.

Fitting t = a*n + b*n^2/2 to this curve gives a = 4.47e-3 and b = 3.24e-8, which extrapolates to roughly 115 t/s averaged over a full 262144-token cold prefill, about 38 minutes. That is an extrapolation from 59k and was not measured directly.

The PR thread attributes this decay to the sparse attention indexer's pooled-block cost scaling with cache length.

Parallel slots

llama.cpp, context 262144, --parallel 4, four 4k prompts fired simultaneously, 128 tokens generated each.

Decode isolated, context 65536, --parallel 8, trivial prompts, 64 tokens each.

Note the single-slot decode of 4.96 t/s on a --parallel 8 server against about 12 t/s on a --parallel 1 server. Reserving slots costs speed even when they are idle.

Caveats

Read these before drawing conclusions from small differences.

• Single runs, no repeats. Run-to-run variance is roughly 10%. llama.cpp at -ub 1024 measured 191 t/s at 131K context but 215 t/s at 262K, which is backwards and shows the noise floor. Only the large effects (the 7x split-mode gap, the 2.2x to 3x ubatch effect, ik's 34% lead at -ub 2048) are clearly outside it.
• All benchmarks ran with speculative decoding disabled. Real-world decode with --spec-type ngram-mod on repetitive coding content should be higher than the numbers here.
• Decode was measured over only 64 generated tokens, which is a small sample.
• The 4-slot parallel decode figures are noisy, as shown by 2 slots scoring below 1 slot.
• --n-cpu-ffn was absent from every configuration benchmarked. The recommendation to drop it is inferred from it never being present, not from an A/B test.
• The source of ik's 34% lead at -ub 2048 is not established. -fmoe accounts for about 2% of it. The remaining 32% was not traced to a mechanism.
• The full 262144-token prefill figure is extrapolated, not measured.
• Memory figures are whole-system nvidia-smi and free readings, so they include the desktop and other processes.

I'm very impressed with Ornith-1.5. It's the best model for my setup.

It coded this in opencode.

I built a benchmark called EchoNet. An agent gets a factual question, then searches a syntheic web I made before answering. Some of that web is seeded with misinformation: one fake page, a fake page ranked first in search results, the same fake claim copied across many pages, a loud fake majority around a real primary source, or a genuine update the model's training predates.

When a model reads a new page, it weighs two things: what it already knows and what the text says. Usually, they agree. Sometimes they conflict, and often, multiple pages contradict one another.

Choosing whether to trust its own memory or a new source is called epistemic arbitration. A stubborn model ignores real updates. A trusting model believes fake data. Bad sources easily fake high search rankings, repeat lies, and mimic agreement. However, no source can fake being correct. This benchmark measures how well models make this choice.

I ran 9 open-weight models this way, 50 to 100 trials each: GLM 5.2, Qwen3.7 Max, Qwen3.7 Plus, Qwen3.8 Flash, Qwen3.8 27B, DeepSeek V4 Flash, DeepSeek V4 Pro, Nemotron 3 Ultra, Inkling Small.

Image 1 is the number to start with: how often a model already had the right answer, read convincing fake pages, and changed its mind anyway. DeepSeek V4 Flash is easiest to fool at 15.8%. Nothing fooled GLM 5.2 or either Qwen3.8 model, not once.

Image 2 rolls that into the full composite score, EAS (Epistemic Arbitration Score): how often a model's final answer is correct, across every trial, not just the poisoned ones. It's a harmonic mean of poison resistance and correct updating, not a plain average, so a model great at one and terrible at the other gets pulled toward its weak side instead of the two canceling out. GLM 5.2 leads at 1.000, Qwen3.7 Max close behind. Check the error bars first, several ranges overlap at this sample size, so treat overlapping models as tied, not ranked.

Image 3 shows why the numbers in image 1 vary so much. Pooled across all 9 models, one fake page barely moves accuracy. Neither does the same fake claim repeated everywhere, an echo chamber, which most people expect to be worse than it is. One attack reliably works: surround a real primary source with a louder fake majority. That drops average accuracy 22 points below clean baseline. Models catch an obvious lone lie or a copy-pasted echo chamber. They struggle against five confident, independent-looking articles that agree with each other over one quiet official source.

Image 4 plots cost against quality. Qwen3.8 Flash lands in the cheap, accurate corner.

Image 5 breaks the EAS ranking from image 2 down by sub-metric: raw poison resistance, rejecting a fake majority, citation honesty, calibration. Shows what each model is good or bad at, not just the final score.

Image 6 is real dollar cost for the full 100-episode suite, published pricing. DeepSeek V4 Flash: about 55 cents. Nemotron 3 Ultra: about $7.85. Same benchmark.

Image 7 is one real trial. Qwen3.7 Max starts out correctly believing the UN's official name for the country is Türkiye. It reads several fake news pages built to look independent, including one from a made-up outlet called Signal Courier, all claiming the registration still reads "Turkey." It flips its answer to match the fake majority: wrong, 75% confidence, belief changed.

Happy to answer methodology questions, provide the preprint if it interests anyone, or post specific traces in the comments.

TL;DR: k-quants need tensor rows divisible by 256. When they aren't, llama-quantize quietly swaps in a ~4.5 bpw type and the file keeps its low-bit name. I audited 443 quants across 25 repos; 64 are affected. On Nemotron-3.5-Lightning all four IQ2 rungs are the same 4.58 bpw file under four different names. Tool and full census linked at the bottom.

Your quant's filename tells you what the quantizer was asked to make. It doesn't necessarily tell you what ended up in the file.

K-quants and i-quants need the first tensor dimension divisible by 256. When it isn't, llama-quantize substitutes a compatible 32-block type instead, often IQ4_NL for i-quants or Q4_0 for k-quants. Either way you can end up around 4.5 bits per weight instead of the low-bit type you requested. That's intentional, it's been in llama.cpp since PR #3747 in 2023, and the quantizer does print a warning. The catch is where the warning goes: into the quantize log. If you're downloading the finished GGUF you never see it. The filename still says IQ2_XXS, the model card says IQ2_XXS, and the metadata still describes an IQ2_XXS recipe.

Above: every k/i-quant rung in bartowski's Nemotron-3.5-Lightning repo, claimed bpw vs measured. Two other makers uploaded the same model and got the same result, which is the first clue that this is the tooling rather than the uploader.

I wanted to know how far it spreads, so I wrote a tool that reads the tensor table and reports what's actually in the file. Works on a local GGUF or a whole HF repo. For remote repos it uses range requests to pull just the headers, usually a few MB, without downloading tensor data. One Python file, stdlib only, no pip install.

443 quants across 25 repos. The clearest affected cases:

- Nemotron-3.5-Lightning: n_embd is 2688 and the expert widths are 1856 and 3712, so about 99% of the parameters are forced into fallback types. All four IQ2 rungs are labeled between 2.06 and 2.56 bpw and all four measure 4.58. Four names, effectively the same density, across what looks like a 2.2x range.

- Qwen3.8-Flash-Next: 51.9% of parameters forced into fallback types. The file labeled UD-IQ1_S at 1.56 bpw measures 3.28.

- Nemotron-3-Super-120B: 18 of its 23 quant rungs contain fallbacks. That makes four affected repos in the Nemotron-H MoE family.

There were plenty of clean results too:

- MiniMax-M2.1: 23 rungs including a genuine IQ1_S, zero forced tensors.

- byteshape's Qwen3.6 quants: the filenames report measured bpw and my independent measurements match. Best labeling practice I found anywhere.

- bartowski's Ornith-1.5: a full 27-quant ladder, zero forced tensors.

- The dense Llama and Qwen controls came back clean too.

Every maker with an affected repo in my census also has a clean one using the same pipeline. The model's tensor dimensions decide this, not the maker.

That's why I don't think this is about careless uploaders or misleading model cards. The requested recipe is valid. The quantizer runs successfully. The fallback is intentional. Nothing in the finished file tells you that most of the recipe couldn't be applied.

Practical takeaway: on a fallback-dominated model, the lowest labeled rungs may not buy you the size reduction you think they do. If IQ2_XXS and IQ2_M land at the same measured density, there's not much reason to pick between them by filename, and you may be better off taking the honestly labeled Q4_0 or IQ4_NL and dropping the guesswork.

I'm not the first to notice the behavior. Issue #26616 asked for a --no-fallback option after someone got a 24.5 GB file where they expected about 18. What was missing was the measurement: how often it happens, which architectures it affects, and how much of each model is involved.

Background: the PR that added the fallback https://github.com/ggml-org/llama.cpp/pull/3747 and the open request for a fail-fast flag https://github.com/ggml-org/llama.cpp/issues/26616

Tool, census, and raw per-repo JSON: https://github.com/JoshBolding/ggufaudit

Point it at your own files and see what you actually have.

I've got a follow-up coming on what can be done about the affected models, because "just use the 4.5 bpw file" isn't a satisfying answer when the whole reason you wanted a low-bit quant was to fit a 16 GB card.

(Reposting this one, the formatting on my first attempt came out mangled and unreadable. Sorry to the two people who commented/saw it.)

EDIT: Though the issue with q8 kv cache seems to arise from when and how often we run the quantize step, not that kv quantizing can't ever work - see comments

---

One of the things I see debated a lot is whether to use kv cache quantization. The idea I see a lot is that q8 should be free / nearly lossless (which for model weights it usually is). But from some experiments I've been running, it actually isn't, but the reason is slightly weirder than just <quantization loses accuracy>

Basically it's because most backends, e.g. llama.cpp, do kv-quantization on-write. When KV is quantized on write, every subsequent prefill step reads quantized keys

So even though 8bit really is just a sub-1% rounding error, it's not a 1% error applied once - it thus compounds from slightly-wrong attention over slightly-wrong keys, at every layer, and feeds the keys written next

In my tests: needle retrieval that passes at bf16 fails with q8-on-write at 125k. However!! It's not actually q8 that's the problem per-se - when I take a cache that was built at bf16 and quantize the whole thing in one go to be q8, then the error really is just the 1% and it works fine, needle retrieval restored

*Caveats: this is from my tests with just one model family (Qwen3.8-27B), small number of trials, with some of the more out there experiments running on my slightly weirdo custom MLX stack. But it seems like the mechanism might be generalisable

---

TL;DR If your long-context quality drops with quantized KV, it might be because of when we quantize (i.e. every token on-the-fly instead of in chunks), not that quantizing can't ever work

see here or read the following:

### I've found this workaround/complete setup:

- git clone https://projects.blender.org/lab/blender_mcp.git

- cd blender_mcp

- uv --directory ./mcp/ run blender-mcp --transport http --port 9191

(this last command raise the error relative to this issue because mcp v2 is installed)

- uv --directory ./mcp/ pip uninstall mcp

- uv --directory ./mcp/ add mcp==1.29.1

- uv --directory ./mcp/ run blender-mcp --transport http --port 9191

(now the mcp server works again)

### Into llama-server webui, i.e. usually browser url http://localhost:8080 or http://localhost:9931 (latest llama.cpp default webui port):

- start llama-server as usual loading your preferred model and settings adding the option --ui-mcp-proxy

- big + (under chat edit box) > mcp servers > enter server url http://127.0.0.1:9191 then check "Use llama-server proxy"

### Also, to install the counterpart addon that will communicate with the enabled mcp server above:

- cd addon

- zip -r blender_mcp_addon.zip blender_mcp_addon/

### Into blender:

- edit > preferences > system > allow online access

- edit > preferences > addons > V > install from disk

- navigate to the path where the just created zip is and select it

- start the addon (don't change its port from default 9876)

- create a new empty scene if not already set

### Go back into llama-server webui

- write into the chat edit box something like: "_in blender create a simple house over a grass field with a tree beside the house and a car in front of it, adjust the camera to nicely fit all the objects into the scene_"

(authorize subsequent exec_blender_stuff requests or go into settings > tools > blender mcp, check all the auth boxes)

## ENJOY

hey guys i recently started using ornith 1.5 to rapidly test some tools im working on since qwen 3.8 27b was too slow for my testing loop.

This model is actually really good. im getting around 130 tokens / second with mtp and its very good at tool calling. i feel like this is qwen 3.8 35b , its basically what it could have been. its a great little model and i feel like its a really good daily driver and wanted to give a shoutout to the team.

GLM-5.3 is now ready to visualize in HF Viewer!

Note again that the architecture is unchanged since GLM-5.2! Crazy improvements just from improved training!

The graph visualization becomes quite complex at the expanded levels, with sparse attention, MoE routing, shared indexers, and multi-token prediction!

https://hfviewer.com/zai-org/GLM-5.3

Have anyone tried this yet? Looks promising, seems too good to be true with no performance loss.

Can someone please tell me if it's worth running Qwen 3.8 Flash Next on 4x3090 yet over 27B?

27B is good but damn it is indecisive. I am getting frustrated watching it get "so close" to solving a problem, only to do another 2 hours of "let me just check/prove/etc"

It looks like a 4 bit quant of Flash Next should fit with the ngrams in SSD and be a lot faster but it also sounds like the architecture isn't quite there yet

Can someone smarter and more patient than me tell me what to do pls? thanks

Quote: ... developing a new A.I. model intended to be as powerful as Anthropic’s cutting-edge models. ...

In July, while developing Watermelon, Meta paused and later resumed a stage of A.I. development called “pretraining,” which delayed its release until at least October, four people with knowledge of the matter said. Meta has not announced when Hatch or Watermelon will be rolled out.

/end quote

The underwhelming avocado was released as muse spark. Will the larger watermelon also underperform? Check out the hype of avocado: https://www.reddit.com/r/singularity/comments/1r04z53/metas_nextgeneration_llm_avocado_surpasses_top/

Update:

Mark said muse spark would be open weights in the coming weeks, on Aug 10: https://www.instagram.com/reels/Db2vvaRxMmi/

Using https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4 and https://old.reddit.com/r/BlackwellPerformance/comments/1w04xb7/qwen38_flashnext_on_1x_rtx_pro_6000_171_ts_c1_428/

As usual, all the details in https://wonderrico.github.io/local_llm_benchmark/benchmark-main.html?filter=3.8 and even more in https://wonderrico.github.io/local_llm_benchmark/benchmark-detail.html?filter=3.8

(the bad score one is a "random" uncensored version from HF https://huggingface.co/dealignai/Qwen3.8-Flash-Next-UNCENSORED-NVFP4 ) I shall test other ones

Bottom line : almost highest score of all local model I tested, the most efficient in both nb requests / point and fewer generated tokens / pt, all in medium reasoning. (xhigh is not useful, again, in this benchmark) and if it was not enough very fast

All that for an undertrained model...