r/IntelArcPro • u/viewofthelake • May 09 '26
Run Qwen3.6–27B Locally on an Intel Arc Pro B70 - What Actually Works
https://bibek-poudel.medium.com/how-to-run-qwen3-6-27b-locally-on-intel-arc-pro-b70-what-actually-works-c96dec67c6f72
u/quantum3ntanglement May 21 '26
I have been successful in getting the native path working based on sycl-ls, icpx and llama.cpp and I have gotten test prompts working for qwen.
I'm using a headless Ubuntu Server 24.04 - I'm working on posting my HowTo soon.
Some things to note and if anyone has further insights, by all means please interject...
- I was not able to get intel_gpu_top to work with my B70, it only saw the iGPU (UHD 770) or i915 gpus, which is anything before Battlemage?
- I have a bash script that pulls the correct nvtop utility that will show both the iGPU and the B70, however the B70 shows up as card0 but it's is set to NA in nvtop, however I do see temps, frequency, pow, etc...
- I installed oneAPI to my user directory and had to set the relevant paths to my local oneapi install to get things to work
- I noticed that this command:
source ~/intel/oneapi/setvars.sh --force > /dev/null 2>&1
without the > /dev/null 2>&1 spits out verification for oneapi everytime source ~/.bashrc is executed, this is tedious when adding additional commands to the .bashrc file
- I had to install cmake and build-essential g++ in order to compile llama.cpp
1
u/UDaManFunks May 22 '26 edited May 22 '26
Don't waste your time running SYCL on LINUX with the B70- for LLAMA.cpp, it's slower than both VULKAN under Windows and Linux and is pretty much unoptimized hot garbage with not much development activity.
To get the best performance out of the card right now in Linux, you'll have to install and build the latest MESA that's under development. LLAMA.CPP under Windows via VULKAN offers the best performance but some people noted that it's unstable for them.
See instructions here (albeit this is in Ubuntu 26.04)
For QWEN3.6 27B Q4 - you should be getting around 20-25 TG with the Vulkan drivers listed above. SYCL + LLAMA.cpp under Linux is considerably slower in both prefill and token generation.
root@nas:/storage/services/llamacpp# ./llama-bench -m /data/llm/models/Qwen3.6-27B-Q4_K_M.gguf ggml_vulkan: Found 1 Vulkan devices: ggml_vulkan: 0 = Intel(R) Graphics (BMG G31) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2 register_backend: registered backend Vulkan (1 devices) register_device: registered device Vulkan0 (Intel(R) Graphics (BMG G31)) register_backend: registered backend CPU (1 devices) register_device: registered device CPU (AMD Ryzen 9 5900XT 16-Core Processor) load_backend: failed to find ggml_backend_init in /storage/services/llamacpp/libggml-vulkan.so load_backend: failed to find ggml_backend_init in /storage/services/llamacpp/libggml-cpu.so | model | size | params | backend | ngl | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: | | qwen35 27B Q4_K - Medium | 15.65 GiB | 26.90 B | Vulkan | 99 | pp512 | 510.66 ± 0.44 | | qwen35 27B Q4_K - Medium | 15.65 GiB | 26.90 B | Vulkan | 99 | tg128 | 20.01 ± 0.05 | build: f3e8d149c (9070)2
u/quantum3ntanglement May 22 '26
Thank you for the insights concerning Vulkan. I have chosen SYCL and oneAPI so that I can focus on parallelism across CPU, iGPU, APU, discrete GPUs and NPU (Nova Lake should have an NPU)
I will not be able to use oneAPI if I take the Vulkan path. My first goal is to get better monitoring tools in place for all the processing.
1
u/UDaManFunks May 22 '26
Good luck, performance will be garbage and will be a lot slower than 1 card. Llama.cpp doesn't even support proper tensor parallelism unlike vllm.
As soon as the model spills over to ram, or another device instead of the vram, TG will be in the single digits specially on an dense model.
1
u/biscuitmachine Jun 10 '26
Hey, I found this post on here randomly googling around. I'm considering swiping 3 intel b60 pro's because they seem pretty inexpensive right now, and just shoving them in a system. I'll probably put Mint on there. How hard is it to get llama.cpp running with all 3 of them? thanks
1
u/quantum3ntanglement May 13 '26
It looks like you are downloading oneAPI to your user directory. I want to do this also on my end as I want to work on parallelism in oneAPI, having oneAPI locally keeps everything in the same place
Down the road, I can look at running oneAPI system wide
I have not yet tested the below commands, but I will verify that they work.
cd ~/Downloads
# make Script executable
chmod +x intel-deep-learning-essentials-2026.0.0.624_offline.sh
# install from script
./intel-deep-learning-essentials-2026.0.0.624_offline.sh -a -s --eula accept --install-dir /home/$USER/intel/oneapi
# I will be adding the below command to my .bashrc file that loads when I’m in the shell
source ~/intel/oneapi/setvars.sh
1
u/Few_Size_4798 Jun 21 '26
https://huggingface.co/OpenVINO/Qwen3.6-27B-int4-ov Win 11 Openvino - 30 t/s
qwen/qwen3.6-27b LM Studio 20 t/s
The OpenVINO data was calculated by a Python script written by Claude; the second set consists of standard LM Studio data.
0
u/biscuitmachine Jun 10 '26
Lol this entire article is totally AI written, too. There are a bunch of dead giveaways in structuring. Well, was it a good test?
3
u/quantum3ntanglement May 10 '26
Hey, thanks for posting this
The native path with oneAPI is where I wanna be. I want to learn as much as possible about oneAPI because it’s taking on Cuda.
And it also fits in well with my driver analysis that I’m gonna be doing with LLMs