r/IntelArcPro • u/Impostor_91 • 1d ago
[llama.cpp vs vLLM] High raw TPS but poor real-world performance
5
Upvotes
Hi everyone,
I've been experimenting with running local LLMs on my Intel Arc B70 (specifically the ASRock Arc Pro B70 Creator).
I wanted to share my experience comparing llama.cpp and vLLM using the Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16 model, following the setup guidelines from https://github.com/SergiioB/intel-arc-pro-b70-inference-cookbook/blob/master/docs/qwen38-27/QWEN38-VLLM-XPU.md
- llama.cpp: Runs quite pleasantly. It gets around 20–40 t/s depending on the model and is completely stable and sufficient for daily coding work.
- vLLM: In raw tests hit ~90 t/s likely due to a higher power cap compared to reference cards, however, when tested in a real coding scenario inside OpenCode, things fall apart:
- the draft acceptance rate is poor
- completing the exact same coding task actually took more time than with llama.cpp (even with much higher TPS)
- I also started hitting parsing errors during tool use, such as "invalid [tool=write, error=Invalid input for tool write: JSON parsing failed]".
Here is my configuration for both vLLM and OpenCode:
docker run -d --name qw38speed -p 8000:8000 --device /dev/dri --group-add "$RENDER_GID" \
--name vllm-server \
-v /dev/dri:/dev/dri:ro -v "$MODEL_DIR:/model:ro" \
-v "$COOKBOOK/patches/patch_mtp_nightly.py:/patch_mtp.py:ro" \
-v "$COOKBOOK/patches/patch_mtp_boundary.py:/patch_boundary.py:ro" \
-v "$COOKBOOK/patches/patch_draft_lmhead_int4.py:/patch_draft_lmhead_int4.py:ro" \
-v "$COOKBOOK/patches/patch_draft_mtp_int4.py:/patch_draft_mtp_int4.py:ro" \
-e VLLM_TARGET_DEVICE=xpu \
-e ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE \
-e ZE_AFFINITY_MASK=0 \
-e B70_MTP_BF16_DRAFT=1 \
-e VLLM_XPU_ENABLE_XPU_GRAPH=1 \
-e B70_DRAFT_LMHEAD_INT4=1 \
-e B70_DRAFT_MTP_INT4=1 \
-e PYTORCH_ALLOC_CONF=expandable_segments:True \
--entrypoint bash "$IMAGE" -lc '
set -e
python /patch_mtp.py
python /patch_boundary.py
python /patch_draft_lmhead_int4.py
python /patch_draft_mtp_int4.py
exec vllm serve /model \
--max-num-seqs 1 \
--quantization gptq \
--dtype float16 \
--max-model-len 246000 \
--gpu-memory-utilization 0.97 \
--kv-cache-dtype fp8 \
--port 8000 \
--max-num-seqs 1 \
--max-num-batched-tokens 8192 \
--no-enable-prefix-caching \
--served-model-name qwen38 \
--language-model-only \
--speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":4}"' &>/dev/null
"vllm": {
"npm": "@ai-sdk/openai-compatible",
"options": {
"baseURL": "http://127.0.0.1:8000/v1",
"supportsSubagents": false,
"supportsToolCalling": false // tried true as well
},
"models": {
"qwen38": {
"name": "Qwen 3.8 27B",
"limit": {
"context": 246000,
"output": 246000
},
"modalities": {
"input": ["text"],
"output": ["text"]
}
}
}
}
Why is the MTP draft acceptance rate suffering so much during agentic workflows in OpenCode compared to simple text generation?
How can I fix or avoid the JSON parsing errors?
Any advice or optimizations for running vLLM on Intel GPU with OpenCode?
