Firezone has pushed WireGuard on macOS from about 1 Gbit/s to more than 5 Gbit/s. On a pair of M1 Macs linked over 40 Gbit/s Thunderbolt, its client now reaches 5.87 Gbit/s upload and 5.76 Gbit/s download.
The slow uploads came from an odd corner of Apple’s kernel, and the fix involves how the VPN reacts when the network is busy.
The symptom
The company’s Linux and Windows builds already ran fast. On macOS and iOS, downloads topped out around 1 Gbit/s, and uploads often fell to a few hundred Mbit/s. Things got worse as latency rose, a hint that packets were disappearing somewhere.
That matters because of how TCP works. Its congestion controller, CUBIC by default on current Windows, Linux and macOS, reads packet loss as a sign the path is full. When it sees a loss, it cuts the congestion window by 30%. A VPN is just another network device to the controller, so any packet the VPN drops tells the sender to slow down.
Reordering hurts too. A packet that arrives late looks like a lost one to the sender, which retransmits it and may reduce its rate for nothing.
Backpressure instead of loss
Firezone’s design connects its processing stages, which run on separate threads, with bounded channels. When one stage fills up, the stage before it pauses. The aim is for pressure to flow back to the application’s socket, so the kernel’s own buffers throttle the sender before any packet is dropped. Local backpressure is instant, while loss-based feedback arrives a round trip late.
This relies on non-blocking I/O behaving as documented: a write to a full buffer fails with a “would block” error, and the program is told when space returns. On Linux that is EWOULDBLOCK and epoll.
What XNU does differently
XNU, the kernel under macOS and iOS, shares roots with FreeBSD, and its socket API goes back to 4.2BSD in 1983. Apple publishes the source on GitHub, a few months after each release. Firezone read it to find out why uploads were slow.
The key finding: XNU allocates a send buffer for every UDP socket, but never uses it. Packets go straight to the network interface’s transmit queue, so the buffer’s fill level stays at zero. Setting SO_SNDBUF only changes the maximum datagram size. The space check always passes, whatever the queue holds.
The result is that UDP backpressure is broken on macOS in practically every async runtime. Writes never fail. When the interface queue is full, packets are silently dropped, and the program thinks they were sent.
The workaround
Connected UDP sockets behave differently. After a connect call, XNU returns ENOBUFS once the interface queue is full. Runtimes normally treat that as a plain error, but registering for write-readiness afterward still wakes the program when the queue has room.
Firezone switched to connected sockets and added a dedicated code path for ENOBUFS that pauses its UDP transmit thread. The change is in its pull request 13725. A brief stall now propagates through the bounded channels back to the TUN device, instead of turning into packet loss that shrinks the congestion window.
Cutting per-call costs
With loss fixed, the team went after speed. It splits costs into per-packet work, such as encryption, NAT, checksums and flow logs, and per-call work, such as syscalls, thread wakeups and allocations. Batching spreads the second kind across many packets. Firezone had already used the batched sendmsg_x and recvmsg_x calls on its UDP sockets before it found the backpressure problem, so historical benchmarks are hard to compare.
What comes next
At 5+ Gbit/s the client handles about 600,000 packets per second at its 1280-byte MTU. Profiling shows the UDP thread is now the busiest, spending most of its time inside the kernel, where XNU still treats each datagram individually at the IP layer.
Linux and Windows avoid that with segmentation offload, which hands the kernel one large buffer plus a segment size. Firezone thinks that if Apple added a similar option to its BSD socket API, throughput on WireGuard on macOS could be two to three times higher. That would also help any UDP-heavy workload, including userspace QUIC stacks.
Source: Firezone