A100 Inference Tuning Cuts 33B Model Cost by Half

A100 inference tuning results for a 33B reasoning model on Azure

Tuning a 33B reasoning model on a single Azure A100 cut the cost of sustained chat serving from $0.90 to $0.41 per million output tokens. The result comes from 337 automated trials run over 41 hours, and it shows how much speed sits unused in default inference settings.

A100 inference tuning of this kind matters most where newer GPUs are scarce. The setup was modest. Hopper and Blackwell GPUs are hard to reserve in many cloud regions right now, while Ampere-generation A100s are easier to get. The engineer behind the experiment reserved a couple of single A100 80GB PCIe nodes in Azure’s Central US region, billed at $4.15 an hour on demand.

The model, the engines and the method

The test model was Nanbeige 4.3-33B, a 33B reasoning model from a family that has been climbing small-model leaderboards. It ran on two open-source serving engines: vLLM 0.34.0 and SGLang 0.5.23.

Rather than tweak flags by hand, the engineer built a hill-climbing benchmark rig. GitHub Copilot, backed by Claude Opus 5.5, drove the loop: change a setting, measure, keep what helps, repeat. The baseline was out-of-the-box vLLM with BF16 defaults on one A100.

The results apply to this model and to four synthetic traffic shapes. They won’t transfer unchanged to other models or workloads, but the size of the gaps is hard to ignore.

Single-user decode speed

A single user talking to the model saw token generation jump from 23.4 tokens per second to 94.8 on tuned vLLM. That’s a 4.1x gain. SGLang reached 74.4 tokens per second, a 3.1x improvement over the vLLM baseline.

At 23 tokens per second, a long reasoning answer drags. At nearly 95, it reads close to real time.

Sustained chat throughput

The chat test ran in 9-minute windows with two limits: time to first token (TTFT) of 2 seconds or less, and time per output token (TPOT) of 100 milliseconds or less. Under those limits, the default vLLM setup handled 1.47 requests per second. Tuned vLLM reached 3.21, a 2.2x gain, and SGLang landed at 3.24.

The two engines finished almost level here. Once each was tuned, the choice of engine mattered far less than the tuning itself.

Multi-turn agent latency

The sharpest change came in agent-style sessions. The test used 6-turn sessions and tracked p90 time to first token, meaning the delay that 90 percent of requests beat. On default vLLM that figure was 114 seconds. After tuning it fell to 1.05 seconds on vLLM and 0.94 seconds on SGLang.

A wait of nearly two minutes makes an agent unusable. Around one second is fine.

What it costs per token

At $4.15 an hour, the sustained chat result translates into serving cost. The untuned setup came to $0.90 per million output tokens. The tuned one came to $0.41, which is less than half. The hardware bill stays the same, so every dollar saved comes from squeezing more work out of the same GPU.

  • Single-user decode: 23.4 to 94.8 tokens per second on vLLM, 74.4 on SGLang
  • Chat throughput: 1.47 to 3.21 requests per second on vLLM, 3.24 on SGLang
  • Agent p90 first-token latency: 114 seconds to 1.05 seconds on vLLM, 0.94 on SGLang
  • Cost: $0.90 to $0.41 per million output tokens

What A100 inference tuning means for teams running their own models

For teams without access to the newest GPUs, the numbers make a practical argument. Older hardware that is easy to reserve can serve a 33B reasoning model at a usable speed, provided the serving stack is tuned for the traffic it will actually see. Defaults are written to work everywhere, so they rarely suit any one workload well.

Automated search is also becoming a realistic way to do that tuning. Hundreds of trials in under two days is more than most engineers would run by hand, and an AI coding assistant handled the repetitive loop here.

What to check before copying the results

Treat the figures as a reference point, not a promise. Your model size, prompt lengths, concurrency and latency targets will change which settings win. The sensible next step for A100 inference tuning is to define your own traffic shapes and limits, such as a TTFT ceiling, then run a similar hill-climb on a single node before buying more hardware.

Anyone planning that exercise should log the baseline first. Without a default measurement like the 1.47 requests per second here, there’s no way to tell how much of a later gain is real.

Source: X (formerly Twitter)

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.