Local AI Models Are Ready, Says Pete Warden

Pete Warden discussing local AI models running on laptops

Local AI models are good enough for most everyday work today, according to Pete Warden, the co-creator of Moonshine Voice and a veteran deep learning engineer at Useful Sensors. He says the obstacle isn’t model quality. It’s that few people have heard these models exist.

Warden made the case in a wide-ranging interview with host Ben Lorica, covering open-weight models, laptop hardware, voice interfaces and the economics of running AI without a cloud subscription.

Why most people haven’t tried local models

Warden compares the situation to open-source software in general: usability trails the commercial product, and marketing trails even further. Local models are hard to monetize. A company that sells a subscription knows who uses its service and how often. A model that runs on your own machine gives its maker none of that, so Anthropic, OpenAI and the large tech companies have little reason to promote it.

The result is a lot of advertising for what he calls a text box on a website, and almost none for alternatives. Tools like Ollama, probably the best known way to run models locally, are simpler than most people expect, he says.

He also points to a pattern from the dot-com years. Companies once paid thousands of dollars for SPARC servers or Microsoft IIS, until Apache and Nginx improved and overtook them. He doesn’t claim local models will repeat that outcome exactly, but it’s the trend he sees.

What hardware you actually need

Warden says the hardware for local AI models bar is lower than most assume. Capable models with about 9 billion parameters are available now, and the Qwen family does particularly well. At eight-bit precision, a 9-billion-parameter model takes roughly nine gigabytes, which many midrange laptops can hold in graphics memory. With 4-bit quantization it shrinks to fit a four-gigabyte graphics card, and plenty of older laptops have that much VRAM.

At the high end, Apple’s new M5 Studio can be configured with 256 gigabytes of system memory that its built-in accelerators can use. That is enough to run models with hundreds of billions of parameters at a usable speed. At that size, Warden says, it can be hard to tell whether you’re talking to Claude or an open-weights model.

Memory prices are a drag on hardware makers at the moment, but he says every manufacturer is working on this.

Benchmarks, guardrails and privacy

Local AI models run only a few months behind the top commercial models on benchmarks. Warden says they’ve reached the point where they can handle what people want from them. He cites a report that Anthropic is concerned about users shifting to specialized models instead of the largest and most expensive one.

Some users prefer local models for another reason: fewer restrictions. Security professionals who ask about malicious techniques as part of defensive work often hit guardrails on commercial models that were built with good intentions but get in the way of benign tasks.

Privacy is the other draw. Warden thinks local AI should be an operating system feature that apps can call, and he sees Apple heading that way. Local also doesn’t have to mean a laptop. Enterprises can host models on-premises or in a cloud instance they control, and he notes that users are now paying closer to the true cost of running these models through Anthropic and OpenAI. Hosting yourself can cost less. For inference, companies may already own enough capacity without buying high-end Nvidia GPUs, which are still needed for training.

Voice as an interface

Warden’s own work centers on speech. Many people dismiss voice because of Siri and Alexa, but he asks a simple question: on the couch, do you text the person next to you or talk to them? Almost everyone, apart from a few Gen Z respondents, says talk. In his view, people don’t dislike voice interfaces. They dislike bad ones.

Latency is the biggest problem. Cloud voice systems usually take at least a couple of seconds to respond, long enough for a speaker to wonder whether they were heard. User experience research puts the target at 200 milliseconds or less. Typing a key and waiting two seconds for it to appear would feel broken, he says.

Cost matters too. If every app developer paid a few cents each time someone tapped an icon, apps would look very different, and many wouldn’t exist. Moving from a cent to zero per interaction, Warden argues, widens what developers can build.

Compound models versus end-to-end

Warden sits in what Berkeley researchers call the compound AI camp. Instead of one end-to-end model that takes in speech and outputs speech, he prefers chaining a speech-to-text model, a language model and a text-to-speech model. Converting audio to text loses intonation and other cues, and text is the only interface every language model shares. But each part can be trained separately, so a new accent or voice doesn’t force retraining of everything.

He considers “end-to-end” largely a marketing term, since only the best-funded companies can train such systems. He also sees a role for local cameras and microphones, because on-device processing can be audited in a way cloud uploads can’t.

What to watch next

Warden expects local AI models to ride on operating system support and cheaper hardware rather than on a single breakthrough. The practical test for readers is easy: pick a 9-billion-parameter model, load it through Ollama on a current laptop and compare it with a cloud assistant on your own tasks.

Source: The Data Exchange

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.