AI Agents at Scale: Lessons From Five Teams

Speakers discussing AI agents at scale at the Crazy Agent Scale event

Five founders recently showed what running AI agents at scale looks like when a company pushes the idea as far as it will go. The Crazy Agent Scale event at Sutter Hill Ventures put Clement Pang, David Cramer, Shawn Wang, Quinn Slack and John Hayes on stage, and the numbers they shared were specific.

The event opened with an Alan Kay line: you can live in the future if you’re willing to pay for it. Xerox PARC did that in the 1970s with its expensive Alto computers. The speakers are doing a version of it with agents.

Intent: treat agents like a company

Clement Pang, co-founder of Wavefront, spent six months building Intent, an open source system for coordinating many agents at once. He warned that AI tips age badly. His first suggestion: put the spec somewhere the agents can read, search and edit, not inside a single context window. With that shared space, agents start storing logs, research and UI mockups there on their own.

Intent tracks work as a graph in which some tasks block or conflict with others. Models tend to quit early, so Intent tells them when a completed task opens up new ones. Pang also pairs agents against each other: one product-manager agent questions the user to build a spec, and another only critiques it.

Missing context can help. A code reviewer that never saw the implementation discussion often spots an option the author overlooked. Agents that run for a long time can step aside to avoid compaction: a new agent starts, the pair exchange notes, and the old one says goodbye.

Waiting on CI burns tokens when an agent sleeps in a loop. Intent uses small monitor scripts instead, which wake the agent only when something needs action. Once tokens stop being the constraint, compute is next, so the Rust backend joins machines into a mesh and sends an Xcode task to a Mac.

Sentry’s Junior: one agent for the whole company

Sentry co-founder David Cramer built Junior, an open source agent that handles almost every job at the company and has made more commits than any person there. Opus 5.5 does 99% of the lifting. Building it took thousands of hours and a team of five, so he suggests most companies begin with an existing framework such as Mastra.

His rules are practical:

  • Use cloud sandboxes. They avoid local disk, port and Docker problems, and compute is only about 10 to 20% of Junior’s bill.
  • Drop the role play. A single agent with a single system prompt covers every task that isn’t highly confidential.
  • Give the agent a corporate identity. Junior files pull requests as itself and credits the requester as co-author on each commit.
  • Subscribe to events instead of polling. Intercom reportedly spent about 60% of its tokens on checking pull requests.

Cramer enforces key workflows in code, not in the prompt. The system rejects tool calls that skip a step, and an egress firewall replies to a bare gh pr create with directions to the proper tool. Secrets stay hidden from the agent. Its best feature, he says, is answering questions: anyone at Sentry, including sales, can ask how something works, and Junior clones the repos and reads the code.

Tiny teams and internal tools

Shawn Wang, who is behind the Latent Space podcast and AI Engineer, talked about tiny teams that earn more than $1M in revenue per employee. A mid-sized SaaS tool was costing his team $40,000 a year, so he offered a $10,000 prize for the best replacement. Contestants each spent a few hundred dollars on clones, and Wang hired the top two. His team now owns the result.

Amp: $100M ARR with 21 people

Quinn Slack, who runs Amp and co-founded Sourcegraph, reported that Amp crossed $100M in annual recurring revenue with the 21 people it had a year ago. Every employee holds at least 1% of the company, and one person paired with an agent owns a product from the build decision through support and marketing.

In three months Amp received roughly 9,300 bug reports, mostly submitted by customers’ own agents. Another agent groups them into themes, adds telemetry and logs, begins fixes and writes to everyone affected, including users who never reported anything. Those emails often carry free credits: about $500,000 over the period, below 0.5% of revenue. A person approves each message before it goes out.

Slack thinks selling software to a broad audience is dead. Amp has no support, finance or marketing teams, and it earns money through joint ventures with firms such as banks.

Reve: a software factory in production

John Hayes of Reve and Pure Storage described a production software factory behind Motif, which turns a whole marketing department into code. Marketers don’t review code, so agents build, check and repair it on the live system. One runbook skill runs a script first and prompts an agent only if it fails, after which the agent patches the script.

It hasn’t been smooth. At around 130 components the system deadlocked for 48 hours tracking findings. The fix was to alternate agents and scripts, since agents forget steps however firmly the prompt insists. Storage in Git, about 600,000 files for one larger customer, made careless reads expensive. Hayes changed the API so batching was the only option and stopped letting agents read Git history.

What teams can take from this

The common thread is that human attention, not tokens, becomes the limit. Cramer pushes questions and review-ready pull requests into one inbox. Slack keeps a human on outgoing customer email. Wang adds that memory and skills for agents remain largely unsolved.

Teams trying this should start small: move one polling loop to event watches, enforce one workflow in code instead of the prompt, and measure what the agents cost before scaling them.

Source: lukew.com

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.