RDMA over USB4 on Windows Links PCs for Local AI

RDMA over USB4 connecting Windows PCs to run a large local AI model

Microsoft is previewing RDMA over USB4 on Windows. Link two or more PCs with USB4 cables and one AI model can run across all their memory and GPU power. The aim is to handle open-weight models that won’t fit on a single machine, with no server networking card, no switch and no extra driver.

The developer preview is coming soon to Windows Insiders in the Experimental channel (Future Platforms). The code is already public.

Why RDMA, and why USB4

Remote Direct Memory Access lets one machine write straight into another’s memory, bypassing most of the network stack. Datacenters use it for storage and distributed computing because latency and CPU load stay low. Windows has long supported it through NetworkDirect, but only with Ethernet and InfiniBand adapters that consumer PCs rarely have.

A lot of AI PCs already ship with USB4 ports, often as Thunderbolt 4 or 5, and Windows 11 can already network two hosts over USB4. Microsoft built its RDMA path on that connection.

Latency matters most. Splitting a model with tensor parallelism makes the machines trade data many times per generated token. Between two PCs, Microsoft measured small-message round trips of roughly 28 to 50 microseconds, about four times faster than Ethernet over UDP across the same cable. That figure holds at 40 Gbps and at 80 Gbps, and the faster link adds bandwidth.

What the preview includes

  • Updated built-in USB4 drivers plus a NetworkDirect provider for USB4. Provider, samples and link-test tools are open source.
  • Peer-to-peer RDMA links on USB4 PCs that meet the preview requirements.
  • A llama.cpp fork that sends its RPC traffic over NetworkDirect rather than TCP and splits models across machines.
  • A Windows Insider build with RDMA support, still to come. The feature is off by default and needs a registry key and test signing.

llama.cpp is just the demo vehicle. Microsoft expects other frameworks to pick up the same NetworkDirect APIs.

How it works under the hood

Windows’ USB4 connection manager first opens a peer-to-peer tunnel between each pair of PCs. A new provider then builds RDMA in software on the USB4 host router’s DMA rings, translating sends, receives, queues and completions so data goes between registered memory regions without extra copies. Every link takes two pairs of rings, one carrying data and one carrying control.

Ring count and USB4 port count together cap cluster size. Connecting N PCs takes N-1 ports on each and a direct cable to every other machine. Daisy-chaining cuts performance sharply.

Early numbers from AMD and Intel

In Microsoft’s lab, three mini-PCs built on AMD Ryzen AI Max, each with Radeon 8060S graphics and 128 GB of unified memory, ran the 235-billion-parameter mixture-of-experts model Qwen3-235B-A22B at about 18.5 tokens per second with 4-bit quantization. The machines sat in a 40 Gbps USB4 full mesh. AMD’s own three-PC rig clocked 21.98 tokens per second decoding and 117.98 on prompt processing.

Intel ran GPT-OSS-120B, a model of about 120 billion parameters that can’t fit in one 64 GB PC. On Core Ultra Series 3 machines with Thunderbolt 4 and 64 GB each, a three-PC cluster managed about 49 tokens per second at a 1K-token context and four PCs reached about 54.

Microsoft says the transport is no longer the bottleneck for these workloads. Results still depend on the model, the framework build and each machine’s memory and GPU. Tested clusters so far are up to three PCs on Ryzen AI Max and up to four on Core Ultra Series 3, though Microsoft says those aren’t the only platforms that work.

What you need and what comes next

You’ll need two or more USB4 PCs with ports managed by Windows, one cable per pair of machines, and enough unified memory, memory bandwidth and GPU power for your model. Ports at 40 Gbps work; 80 Gbps ports such as Thunderbolt 5 help with bandwidth-heavy jobs like long-context prompts. Every PC must run the same Insider build, preview drivers and provider version.

Microsoft will publish the Windows build and setup steps once the preview is available, and has shown a demonstration at Microsoft Ignite. Developers can start preparing now with the open-source provider and the llama.cpp sample.

For anyone running local models, the practical appeal is simple: a small stack of existing laptops or mini-PCs becomes a cluster. The catch is that port counts limit how many machines you can link.

Source: TECHCOMMUNITY.MICROSOFT.COM

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.