MoE Expert Streaming Runs 180B Model on 32GB Mac

MoE expert streaming runs a 180B model on a 32GB Mac using Transformers

MoE expert streaming lets Transformers run a 180-billion-parameter model on a Mac with 32 GB of memory. A Hugging Face blog post by Tarek Ziade reports about 14 tokens per second, within roughly 6% of the Strata inference engine on short decoding.

The author wanted to run bigger models locally. His setup includes a PC with two RTX 4090s and a MacBook with an M5 chip and 32 GB of unified memory. The model is Qwen3.8-Flash-Next.

Why experts can live on disk

Experts make up more than 90% of a mixture-of-experts model’s weights, but only a few are active for each token. Strata keeps the hot experts in GPU memory and the rest in RAM or on disk. Ziade compared how other engines handle models that don’t fit:

  • llama.cpp keeps experts on the CPU and computes them there. It works well if they fit in RAM, but the page cache thrashes if they don’t.
  • vLLM offloads weights to pinned RAM and reads them over PCIe, with no cache and no disk tier.
  • Transformers, via Accelerate, copies a layer’s whole expert bank to the GPU for every token, and offloading to CPU frees nothing on a Mac with shared memory.

What he built

He says all the code was built with AI, including the Transformers pull request, a model-specific package and the Metal kernels, while he chose what to try, read the profiles and decided what to keep. The work is in a Hugging Face Space. The key idea is a slot pool: one preallocated tensor shared by all layers plus a per-layer table mapping expert ids to slots. The router’s expert ids are remapped to slots, so the pool behaves like a normal expert bank and the existing grouped_mm kernel runs unchanged. On the small OLMoE test model this reached 34.6 tokens per second against 38.1 for vanilla Transformers.

The feature is an option on from_pretrained, in Transformers pull request 50717. Missed experts load through eight I/O threads, with one contiguous read per expert, and the next two layers’ routers run early to guess what to prefetch. The API is still under discussion with maintainers.

Fitting 121 GB into 32 GB

Qwen3.8-Flash-Next has 48 layers of 512 experts, 10 active, and weighs 121 GB in FP8, plus a 51 GB n-gram table. The fit works like this: experts quantized to 3 bits take 49 GB on disk with 12 GiB cached on the device, dense layers use 8 bits (about 5.4 GB), and the n-gram table stays on disk. Conversion takes around five hours. A 6K prompt peaks at 25.4 GiB, so the browser has to be closed.

Closing the gap with Strata

The first version ran near 10 tokens per second. Gains came from committing GPU work earlier (+10%), custom Metal kernels, Strata’s smaller GGUF weights and faster prefill, which rose from 68 to about 228 tokens per second. No cache policy beat plain LRU by much, but reading one guessed expert per layer while the SSD sat idle helped.

What remains is roughly 10 ms per token of host round trips, because PyTorch doesn’t expose a way to queue a token’s work up front. Ziade says a real fix would be a native op on PyTorch’s MPS stream, with a risk of hangs.

What the author tried along the way

Ziade started with the small OLMoE-1B-7B model, which fits in memory, so he could compare every token against vanilla Transformers. His first version used experts on disk, an LRU cache on the device and a Python loop over one expert at a time. It reached 10.1 tokens per second against about 32 for vanilla, because the loop gave up the grouped kernel that runs all of a layer’s experts at once.

He also tried vllm-metal, vLLM on Apple GPUs through MLX, which streams routed experts through a bounded cache. It cannot run this model on 32 GB because the n-gram tables stay in memory. In his tests, the same model run with Strata was only slightly faster on short decoding.

The honest limit

At about 14 tokens per second it is still too slow for how the author uses coding agents, and his desktop keeps doing the real work. The value is in showing that MoE expert streaming in plain Python and Metal gets this close, with no C++ extension. Transformers users can try the pull request branch.

Source: huggingface.co

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.