Run Qwen3.8 Locally: Setup and Hardware Guide

Setup guide to run Qwen3.8 locally on a consumer GPU

Developers who want to run Qwen3.8 locally have a clear path now. Qwen3.8-27B is the model local users most often credit with handling real coding work on their own hardware, whether that’s a single consumer GPU or a Mac with plenty of memory. A new setup walkthrough covers the install, the right model for each machine and the settings worth changing.

Start with a current llama.cpp build

The runtime is llama.cpp, which now builds with CMake, so older make instructions fail. Fresh Qwen releases often need a current build, which makes compiling from source the safer bet on Mac and Linux. Homebrew is the fast option on a Mac.

GPU support is a build flag. NVIDIA owners add the CUDA option. On Windows, CUDA runs inside WSL through the normal Windows driver, so there’s no need for a Linux GPU driver, but the CUDA toolkit must be installed inside WSL or CMake can’t find the compiler. That’s the most common stumbling block.

AMD owners have a twist. On RDNA3 and RDNA4 cards such as the 7900 XTX and 9070 XT, several users say Vulkan matches or beats ROCm and uses less VRAM. One user report puts Qwen3.8-27B at about 63 tokens per second on Vulkan against roughly 52 on ROCm.

Running the 27B model

One command downloads the Unsloth Q4_K_XL quant and opens a chat with a 32,768-token context and all layers on the GPU. The startup log reports how many layers were offloaded, and a count of zero means your build has no GPU support.

At Q4 the model wants roughly 24GB of VRAM, which means an RTX 3090, 4090 or 5090, or a 7900 XTX. A Mac needs 48GB or more. Those figures are estimates from file size plus context, not measurements. A 32GB Mac can load it, but a user report puts speed at only 2 to 4 tokens per second.

Picking a model for your hardware

Few people own a 24GB card, and the family has smaller and larger options.

  • Qwen3.5-4B: about 2.7GB at Q4, built for 4 to 8GB GPUs and 16GB laptops. On a Quadro T1000 with 4GB of VRAM under WSL, it hit 62 tokens per second for prompt processing and 35 for generation.
  • Qwen3.5-9B: a 5.7GB Q4_K_M file that fits an 8GB card with a moderate context.
  • Qwen3.6-35B-A3B: a mixture-of-experts model with about 3 billion active parameters per token. One user saw around 129 tokens per second on a 7900 XTX. It suits 32GB Macs, Strix Halo machines and 16GB GPUs.
  • Qwen3.8 Flash Next: a bigger mixture-of-experts model for 128GB machines such as Ryzen AI Max 395 systems and 128GB Macs. Users call it stronger on broad knowledge than the 27B.

The 4B suits quick questions, small functions, tagging and extraction. It isn’t meant for multi-file agent work.

Tuning that makes a difference

Only a handful of flags change results.

Even the 4B emits a full reasoning block in reply to a one-line prompt. A cap of –reasoning-budget 4096 stops the rambling. Quantizing the KV cache to q8_0 roughly halves context memory with little quality loss, and one user reports fitting about 190k tokens of context beside a Q4 27B on a 24GB card.

Current builds also support speculative decoding with MTP drafting, which needs a model that ships MTP heads. One user reports draft acceptance above 80% on code, though that speedup wasn’t measured. For mixture-of-experts models that overflow VRAM, –cpu-moe keeps expert weights in system RAM, and a larger micro-batch can double prompt processing speed, according to user reports.

Mac owners should know macOS caps GPU memory at about two thirds of RAM by default. A sysctl setting raises it, and the change resets on reboot.

Why this size changed local coding

Until recently, a local model that could really code was mostly wishful thinking. Small models in the 7B to 8B class finished lines but lost the thread once a job spanned several files. Users now describe Qwen3.8-27B working as an agent that opens files, runs commands and corrects its own errors, provided it gets a clear plan and the right context.

Treat that as community opinion rather than a benchmark result, and test it on your own work. The mixture-of-experts models widen access even further. The dense 27B reads every weight for every token, so it crawls on a 32GB laptop. The 35B-A3B reads only a small share each time and degrades more gently when part of it lands in system RAM.

Some friction is left. Unchecked, Qwen reasons far too long, search tools have to cover holes in its knowledge, and each new release tends to demand a newer llama.cpp build.

Serving it to your coding tools

To connect Qwen to a coding agent such as OpenCode or any OpenAI-compatible client, run llama-server instead of the CLI and point the client at port 8080 on localhost. That setup uses a 131,072-token context, q8_0 cache types and the 4,096 reasoning budget.

Mind the security default: llama-server ships without authentication. Leave it on 127.0.0.1 unless you need remote access. Binding to 0.0.0.0 opens it to everyone on your network, so put a firewall in front or set an API key.

Pick the model that fits your memory first, then tune the reasoning cap and cache. For agent work, the 27B or the 35B-A3B is the sensible place to begin.

Source: Igor's Techno Club

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.