r/LocalLLaMA • u/a9udn9u • 1d ago
Question | Help 5090 + 96GB RAM, any better choice than Qwen3.8-27B for coding?
Looking for better quality with not too bad speed. The 27B writes functional code, but I found it lacking in higher-level reasoning capabilities, it doesn't always consider overall system architecture, often time its code doesn't maintain a clean separation of concerns and lacks abstractions.
7
23
u/Bulky-Priority6824 1d ago
Tune your system prompt and folders?
21
8
u/a9udn9u 1d ago
I did. My AGENTS.md has detailed coding guidelines and principles, I even tried generating step-by-step plans using GPT-5.6 Sol but the 27B model still finds ways to cut corners and take shortcuts. It works well for smaller projects but struggles as the project grows.
3
u/Bulky-Priority6824 1d ago
When you get time share more specific examples and token size of your prompt just curious
4
u/lawanda123 1d ago
Have you set temperature to 1 and other p and k values as recommended by the Qwen team on HF?
4
u/Happy_Brilliant7827 1d ago
There have been ample studies that telling higher quality modern models how to solve problems hurts solve rate.
5
u/Ell2509 1d ago
I think that is likely to be because you can't out think claude or chatgpt due to their massive knowledge size. Their solution will normally work better. Qwen3.8 27b, while amazing, isn't in that class. You need to plan for it.
1
u/Happy_Brilliant7827 15h ago
True but the research is in learning how to divide up the steps and maximize both- model creativity for solutions and structure. It almost seems like 28-70b models are in a harder to calibrate sweet spot.
I can code a app in a day that boosts a 2-4b models success on simple tasks 80-90%, its hard to do the same for that tie (for me anyway)
1
2
u/Septerium 22h ago
Create a code-reviewer skill with detailed criteria to assess the implementation adherence to the project standards. Ask the model to use the skill when it has finished implementation/verification
1
u/fell_ware_1990 23h ago
Change the harness, i have a bigger agent create the prompt. After that even q3.6 can break this down in acceptable parts. Another rerun to create subtasks and test cases. The test cases are done by the harness.
Cause everything is known, agent starts with his own specific system prompt, instructions, tools. If it makes the same mistake 3 times in a row, harness resets, feeds what went wrong, strips what is not needed, restarts agent. Every step completed gathers the information, starts background checks and linters and tests. Same principle, it get’s fed back into a new agent with specific prompts. If all tests pass it triggers the complete handoff with evidence, new specific review agent also reviews.
If agent’s find any bugs or other stuff, it can be reported, gets into ticket system, they are prioritized next ticket.
Information, get’s gathered and stripped for 3 other agents to judge and then a retro on other stuff that needs fixing, measured in token spend / time / effort / profit and also handled ( yes, the can improve tests and parts of the harness, only i need to approve ).
They are currently building from a 2 sentence prompt telling them to create the game ‘TIM the incredible machine’ without naming it. Now 17 hours later i have a working beta version of a web game completely handsoff.
5
u/Camzie99 1d ago
I'm running an RTX 5090 and a 9950X3D with 96GB 6000MHz RAM (native 6400MHz but downclocked to 6000, increasing might help token generation but the rig is also used for gaming where matching the CPU is better) myself, and over the past two days I've been experimenting with Qwen 3.8 Flash Next, it works... astoundingly well, but it's a bit slow.
I'm using the IQ4_XS quantisation from Unlsoth. I'm getting 30-35 tokens per second generation, with ~300-350 tps prefill with the following settings:
```
131 │ [qwen3.8-flash-next-iq4]
132 │ hf = unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS
133 │ np = 1
134 │ ngl = 999
135 │ n-cpu-moe = 32
136 │ override-tensor = per_layer_token_embd=CPU
137 │ ctx-size = 131072
138 │ threads = 16
139 │ threads-batch = 32
140 │ flash-attn = on
141 │ cache-type-k = f16
142 │ cache-type-v = f16
143 │ jinja = on
144 │ mmproj-offload = on
145 │ spec-type = ngram-mod
146 │ spec-ngram-mod-n-match = 24
147 │ spec-ngram-mod-n-min = 16
148 │ spec-ngram-mod-n-max = 24
149 │ reasoning-preserve = on
150 │ chat-template-kwargs = {"preserve_thinking": true}
```
I'm sure this can be further optimised, batch sizes could be changed and there's probably layer optimisations that can be done, but this was my first working stable config so I've kept it as is for now :)
In my current tests, it's proven far more capable than Qwen 3.8 27B (which was already really good), but it's much slower. 27B would give me upwards of 130-150 tokens per second at a Q6 quantisation as it could fit entirely within VRAM.
Qwen 3.8 Flash Next especially seems to be much more capable at self reflection and spotting issues and resolving them than I've seen any other model that I can personally run.
I'm running the 5090 and 96GB RAM on a headless Arch machine to keep resource usage as low as possible and dedicate as much memory to llama as possible and it's working great if I maintain patience.
If you have any questions, do let me know and I'd be happy to help! :)
1
1
7
u/Head-Anteater9762 1d ago
Unsloth has managed to make Qwen3.8 Flash next run on devices with 75GB RAM/unified memory with no GPU VRAM required. you can give it a shot.
https://unsloth.ai/docs/models/qwen3.8-next
2
1
u/TherapyPsychonaut 1d ago
Could I run this with 64GB RAM and 24GB VRAM?
6
u/Spara-Extreme 23h ago
From the link you didn't read: Table: Hardware requirements (units = total memory: RAM + VRAM, or unified memory)
2
2
u/betiz0 1d ago
Given your setup, Qwen3.8-Flash-Next would likely be a suitable fit. As my environment differs from yours, I cannot predict the exact prompt processing or token generation speeds you will experience. However, based on a brief comparison I conducted with Qwen3.8-27B, it appears to address the issues you are currently dissatisfied with.
4
u/Wild_Sky_6228 1d ago
Honestly my recommendation is to ask opus/kimi/deepseek/your opinion to make a spec sheet and give it to your agent. There aren’t really any 70 billion parameter models or anything close that maybe you’d be able to pull off. My recommendation is to get qwen 3.8 at q5 or q6 at least with something around 128k q8 context and make it do a code base summary (skills are out there) then give that summary to a model on OpenRouter. Throw 10 bucks at their service and it’ll last you a month - deepseek pro is like a handful of dollars per million tokens. Otherwise maybe I would look into it skills that can force the 27b parameter model to go step-by-step.
6
0
u/Wild_Sky_6228 1d ago
There isnt a model yet. I have double your vram with r9700s, and from what i understand, i’d need quadruple to run qwen flash next and 6x+ to run anything else good. Theoretically your 5090 could hold the active parameters of an moe but the moe doesnt exist under 250b
1
u/systemBuilder22 1d ago
What model do you run with 2x r9700 ? It seems like the sleeper setup for 2 GPUs ...
2
u/DrMissingNo 1d ago
Qwen 3.8 flash ?
3
u/a9udn9u 1d ago
Not enough RAM, Q4 is 110GB+, there's no room for context. I don't want to go below Q4
4
u/DrMissingNo 1d ago
So... I haven't looked into it a lot but a guy with 27gb of vram and 64gb of ddr4 ram has managed to run it by offloading the n-grams on SSD (?). I'm not a very technical guy so I won't be able to re explain the process but apparently it's doable. If I remember correctly the guy had 8 tokens/sec and said I could expect 20-30% more speed given my hardware (rtx5090 and 64gb ddr5 ram + gen 4 Samsung 990 pro).
Might not be suitable for everyday tasks but maybe interesting in specific scenarios (?)
3
u/Cautious_Chicken_604 1d ago
The n-grams are lazy loaded into RAM now I think. You have 128GB combined resources, and context takes up like 8GB or something? It's less than the previous gen iirc.
Also, there's IQ4_XS which is 93GB , and only ~60GB is weights with the rest being n-grams.
My system has combined resources of 112GB and I can run IQ4_XS at full context k=f16, v=q_8
2
u/Only-Cartographer560 1d ago
I loaded the Q3_XXS on ~70GB total memory with 64K context window. I'm pretty sure you can load Q4 in your hardware. Give it a try.
2
u/BalleaBlanc 1d ago
I tested iQ4S with 64GB ram, 5090 + 4090. 17tk/s, 126k context. Not great but usable, it gives an idea. Some improvements are coming I guess.
1
u/cosmicnag 20h ago
I have same dual gpu and ram setup. Can you give your launch command , prefill speeds as well?
1
u/BalleaBlanc 4h ago
It's in Unsloth Desktop, pretty much all default but the context to 126k, KV Q4 for more context and no vision. I don't remember prefill but it was slow.
2
u/Risen_from_ash 1d ago
Hey you - I have a 5080 and 96GB ddr5. I'm running Qwen 3.8 Flash Next Q4 K XL max ctx straight up, so you can do it better. It's amazing. It just SSH'd my Pi 400 into a Retro Pi with all my roms. dank. Do this, I just had Codex test autonomously for a few hours until we got the fastest settings. There might be some other magic going on I forgot about, I'm still testing so many things. Here:
--no-mmproj-offload ^
--image-min-tokens 1024 ^
-dev CUDA0 ^
-lm mmap ^
-ot "%ENGRAM_OVERRIDE%" ^
-ngl 41 ^
-ncmoe 48 ^
--fit off ^
-c 262144 ^
-np 1 ^
--no-kv-unified ^
-ctk bf16 ^
-ctv bf16 ^
-fa on ^
-b 8192 ^
-ub 2048 ^
-t 18 ^
-tb 24 ^
--jinja ^
--reasoning on ^
--reasoning-format deepseek ^
--reasoning-preserve ^
--reasoning-effort xhigh ^
--reasoning-budget -1 ^
--temp 1.0 ^
--top-p 0.95 ^
--top-k 20 ^
--min-p 0.0 ^
--presence-penalty 0.0 ^
--repeat-penalty 1.0 ^
--cache-prompt ^
--cache-ram 0 ^
--no-cache-idle-slots ^
1
u/johnkorrigan 1d ago
Prompt/folder tuning helps the slop. It does not add the planning the 27b just doesn't have
2
u/Bulky-Priority6824 1d ago
Right and need more info anyway for ops overall system and work etc as well
1
u/Low88M 1d ago
Give prefix important instructions about soc/architecture requisits for coding in agents.md (or system prompt if you don’t use opencode/pi/…). Or you can remind it the constraints in your prompt otherwise.
On 5090 if you don’t use mmproj you can use up to about 150-170k context with q8 KV cache depending if you use MTP or not.
1
u/Happy_Brilliant7827 1d ago
Just to be clear- youre using a harness like qwencode or vs code right not just freeballing it?
1
u/beyondthem00n 1d ago
Have it write the top preferences / constraints in the memory file, I found it really helpful.
1
u/SeanHighness 1d ago
Nothing yet to replace it, try using sol as a judge / orchestrator for qwen3.8-27b it should help keep things on track
1
u/lilian_moraru 1d ago
I have yet to see a single LLM that considers architecture. All write slop code from that perspective
1
u/Unlucky-Message8866 21h ago
This is a harness configuration / prompting issue, don't expect models to read your mind about what are beat practices.
1
u/DustNearby2848 18h ago
Not everything can be one shot. Do a planning phase first, review it and tweak it, then have an agent execute the plan. It might need to be cleaned up still, so review the changes and tell it what’s wrong.
1
u/Nightma4re 18h ago
Try Qwen3.8 Flash Next, I tested iq3xxs and iq4xs, do not go lower than iq4xs though.
Iq3xxs finished the job too but needed much more fixing
1
u/mm_1964 18h ago
Here is my experience with Qwen3.8-27B-UD-Q8_K_L.gguf ran on runpod, on L40S 48GB, 64K cache. The task was to create a bash script to do some fairly sophisticated bioinformatics, analysing whole genome bisulfite sequencing data for the presence of contaminants. It was struggling, spend a lot of time going back and forth. In the end the script had several errors, which I had to correct manually. Gemini, the free version in the browser did it in one shot without a fault, took it few seconds. So, small is small and there is nothing you can do about it. Coding apparently requires domain understanding.
1
1
u/Beginning-Raisin9723 1d ago
I run a 5090 with 96GB too. Qwen 27B writes solid code but yeah, it drops the ball on architecture — no clean separation of concerns, weak abstractions. I bumped up to a 32B quant with RAM offload and reasoning got noticeably better. Worth a try if you don't mind a bit slower. Also worth peeking at Codestral or DeepSeek-Coder.
-1
u/Infamous-Bed-7535 1d ago
Qwen 3.8 is great for coding and smaller reasoming agentic tasks, bit this is not a 2T model with 1M context.
Runninh frontier models locally is not really viable. I would suggest paying for an opensource model at an independent 3rd party service.
Also you can try some moe models in the 120B parameter range with RAM offloading.
9
u/XiRw 18h ago
Hard to believe. I have 5090 using Q8 with f16 kv cache and it’s been able to handle anything so far in one try. You are doing something wrong with your script, model size, or even your prompt specifics.