Inference now makes up about two-thirds of AI workloads, and that shift is changing where AI inference chips get installed. Cameron McCaskill, who runs sales and business development for Positron AI, says the split was roughly 85% training and 15% inference back in 2023.
The change matters for buyers because the two jobs stress hardware differently. Training wants huge clusters. Inference, the work of generating answers from a model that already exists, rewards sites that can be built fast and run cheaply.
How the workload mix flipped
McCaskill puts today’s mix at about two-thirds inference and one-third training. He argues that this is why inference now drives the economics of how companies build out infrastructure. When most of the traffic is serving models rather than creating them, spending follows the serving side.
That means smaller, quickly built sites are taking a bigger share of new capacity, next to the giant training clusters. Power, cooling and memory supply now decide where tokens get generated about as much as raw chip speed does.
It is a reversal worth spelling out. Three years ago, a buyer planning a data center could assume almost all of the capacity would go to training runs. Now the majority of the compute is spent answering requests, which is steady, recurring work tied directly to revenue. That is the reason, in McCaskill’s telling, that inference has become the factor behind purchasing decisions.
Positron’s bet on commodity memory
In September the startup closed a funding round of $875 million that valued it at $5 billion. Its appliances use ordinary memory chips rather than high-bandwidth memory, known as HBM.
HBM is what most of the industry uses. McCaskill describes it as very limited in supply and very expensive, and says Positron chose a different path. Avoiding HBM sidesteps the scarcest part of the supply chain, though it also means the company has to make its own design work with ordinary memory.
The company’s first-generation Atlas server runs on a programmable chip. Custom silicon for the follow-up Titan server is due to reach tape-out in the first months of next year.
The trade-off is plain. HBM gives chips very high memory bandwidth, but the supply is tight and the price reflects that. A design built on commodity memory can be sourced more easily, which matters to anyone trying to fill a rack on a fixed timeline.
Air-cooled racks and a 40-kilowatt budget
Memory choice also holds down power draw. McCaskill says each Positron server pulls roughly 4,500 watts, while Titan racks stay under a 40-kilowatt ceiling. That is low enough for air-cooled rooms, so operators don’t need liquid cooling to run them.
That footprint suits locations with a handful of megawatts of inexpensive electricity and light permitting. For inference providers selling new capacity, speed to launch is a large part of the pitch.
Modular data centers want these chips
McCaskill says one group of customers is growing fast: operators of modular data centers. These come in sizes such as half a megawatt, 1 megawatt or 2.5 megawatts, and they can be deployed very quickly.
Many inference providers, he says, set a target of 90 to 120 days between securing land and power and selling tokens. Low-power hardware that needs no exotic cooling fits that schedule.
The speed target is aggressive. A conventional large data center can take years to plan, permit and build, while a modular unit arrives as a prefabricated block. Pair that with AI inference chips that stay inside an air-cooled power budget, and the time from site to revenue gets much shorter. That is the combination McCaskill says customers are asking for.
Who has signed on so far
Oracle is putting around 340 Atlas machines into service in its cloud during the current quarter, McCaskill said. A neocloud operator has ordered Atlas and pledged to take Titan too. Both claims come from the company, so treat them as vendor statements until the deployments are publicly confirmed.
What to watch after the Titan tape-out
The first test is whether Positron’s own silicon reaches tape-out early next year as planned. After that, the question is whether Titan racks hold the 40-kilowatt figure in real deployments.
For buyers, the practical lesson is to check power and cooling limits before comparing chip benchmarks, since those limits decide what a site can host at all.
The larger question is whether commodity memory can keep up as inference demand grows. If the two-thirds figure holds, the market for compact, air-cooled inference hardware will keep growing, and more chipmakers will chase it with their own AI inference chips.
Source: SiliconANGLE