RoCE vs InfiniBand: AI Inference Test on B200 Nodes

RoCE vs InfiniBand interconnect test for AI inference

The network between GPU nodes can matter as much as the GPUs themselves. In a test of RoCE vs InfiniBand on two-node NVIDIA HGX B200 clusters, Lambda found that RoCE delivered roughly 70% lower throughput than NVIDIA Quantum InfiniBand at around 250 concurrent users, with a time to first token 2.5 times slower.

The result comes from a serving experiment on a large mixture-of-experts model, and it puts numbers on a choice that cloud customers often make without much data.

Two ways to connect GPUs

Backend interconnects for GPU clusters generally come in two forms: RDMA over Converged Ethernet, known as RoCE, which includes RoCEv1 and RoCEv2, and NVIDIA Quantum InfiniBand.

RoCE works at layer 2 of the network stack. It uses Ethernet and MAC addresses, so nodes can cross several switches but must stay in the same base network. RoCEv2 wraps traffic in UDP/IP packets, so nodes can talk across subnets, buildings or data centers. The cost is latency, since those nodes may not share a rack or even a site.

InfiniBand uses a dedicated fabric instead of Ethernet and IP and can span multiple switches and locations. It needs InfiniBand-enabled switches and fabric in place first to deliver its best performance. Lambda says its 1-Click-Clusters guarantee that setup.

RoCE versus NVIDIA Quantum InfiniBand performance comparison
Image: lambda.com

Latency on paper

Both options can reach very low latency, often just a few microseconds. InfiniBand is usually lower and steadier, typically around 0.5 to 2 microseconds. RoCE on a well-tuned local fabric reaches roughly 1 to 3 microseconds.

RoCE is more sensitive to configuration, congestion and topology. With competing traffic, misconfigured switches or untuned flows, latency can rise several times over and reach tens of milliseconds under congestion. A gap of 1 to 3 microseconds sounds small, but it applies to every transfer and ignores packet sizes and tensor allocation.

How Lambda ran the test

Lambda compared two setups. One used two 1-Click-Cluster HGX B200 nodes linked by Quantum InfiniBand switches and adapters. The other used two On-Demand HGX B200 nodes connected over RoCE, placed in the same cloud region but not necessarily the same data center.

  • Model: nvidia/GLM-5.2-NVFP4, a Blackwell-optimized variant of Z.ai’s GLM 5.2
  • Serving framework: SGLang, with TP16, DP16 and EP16 parallelism
  • Benchmark tool: NVIDIA’s AIPerf
  • Workload: 8,192 input tokens and 1,024 output tokens per request
  • Concurrency: 10 to 300 simulated requests

Both servers used the same core SGLang flags, including data-parallel attention and DeepEP in low-latency mode across two nodes. Only the network-specific environment settings differed.

What the numbers showed

As concurrency approached about 250 users, where the hardware saturates, RoCE produced roughly 70% less throughput than InfiniBand. Time to first token was 2.5 times slower. At lower concurrency the fabric differences were also visible, with RoCE showing the slowest inter-token latency and the lowest tokens per second overall.

Lambda draws a plain conclusion: the interconnect is not a background detail. At scale it shapes GPU utilization, responsiveness and the return on infrastructure spending. When a model spans several nodes, each matrix multiplication moves data across the fabric, so communication can become the bottleneck.

Reading the results with care

The comparison has limits. It covers one model, one workload shape and two nodes per side, and the RoCE nodes were not guaranteed to share a data center. Lambda itself notes that real-world performance varies substantially. The company also sells the InfiniBand product it recommends, so buyers should weigh that.

The direction still agrees with the theory. RoCE is sensitive to topology and congestion, and the test placed it in the less controlled position.

What this means for buyers

For teams serving models at scale, Lambda’s advice is to choose dedicated InfiniBand clusters. For those who can’t, two On-Demand or Reserved Instance nodes will work, with lower expectations. Anyone running RoCE can narrow the gap by tuning switches and keeping nodes close together. The practical next step is to benchmark your own model and traffic mix over both fabrics before signing a long contract.

Source: lambda.com

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.