Fast microVMs for Coding Agents on GKE: Lessons

Diagram-style view of microVMs for coding agents running on GKE nodes

Turning on microVMs for coding agents takes one setting, but making them feel quick is the hard part. Kosem gives every agent environment its own microVM. It says extracting a Linux 6.6 source archive fell from roughly two minutes to under nine seconds, file-metadata lookups across 20,000 files ran 45 times faster, and a new environment is ready in 20 seconds.

The stack is Kata Containers with the Cloud Hypervisor virtual machine monitor, hosted on Google Kubernetes Engine nodes that allow nested virtualization. Almost none of the speedup came from the hypervisor itself.

Why agents need a real kernel

Agents on these platforms act like a developer at a terminal. They add OS packages, execute test suites and bring up databases and build services with docker compose. Doing that needs a genuine Linux kernel. Because the code belongs to many different customers and can’t be trusted, the hardware itself has to separate one environment from the next.

So each environment starts a private Linux kernel, version 6.6, with a full Docker daemon inside. Docker wants a privileged container, which in a Kata microVM gives root only within the guest, while the hypervisor stays the wall against the host. An admission policy backs this up: Kubernetes refuses any Pod in the namespace that skips the Kata runtime class, or asks for host networking, host ports or host-path mounts. Pods carry no service account token, and outbound traffic can reach the public internet only.

Keep hot file operations off virtiofs

Kata normally passes a Pod’s storage to the guest through virtiofs. Each file call, such as stat, open, unlink or readdir, has to exit the guest, pass the VM boundary and execute on the host before returning. Running git status in a big repository triggers tens of thousands of such round trips.

Kosem’s answer was to hand the guest the raw disk. It requests each Hyperdisk volume in Block mode, Cloud Hypervisor presents it through virtio-blk, and the guest’s init formats it as ext4 on first start. File work then never leaves the guest. Extracting the Linux tree, around 80,000 files, took 126 to 129 seconds on virtiofs and 7.6 to 8.8 seconds on the block disk, about 15 times faster.

Skip the discard on a fresh disk

The first boot on the new setup still took more than two minutes. Of that, 101 seconds went to mke2fs discarding every block of a fresh Hyperdisk that Google had already zeroed.

Three mke2fs options fixed it: nodiscard, plus lazy_itable_init and lazy_journal_init so the guest zeroes inode tables and the journal in the background. Formatting finishes in under a second whatever the size. Setting the root reserve to zero, instead of ext4’s default 5%, frees 1 GiB on a 20 GiB disk. A first boot lands at 20 seconds and a restart onto an existing disk at 4.7 seconds. The VM vanishes when its init process exits, taking unwritten cached data with it, so init flushes and unmounts the disk first.

Move the root filesystem too

Large repository operations were fast after that, but short test loops still dragged, since binaries, libraries and Python imports were served from the container’s root filesystem, which is also shared over virtiofs. One iteration of an agent’s COBOL test loop in the Terminal-Bench cobol-modernization task took 106 ms in the microVM against 9 ms on a laptop.

Kosem now packs each image once, at registration, into a read-only EROFS file. At boot, init mounts it as the lower layer of an overlayfs, with the upper layer on the ext4 disk. The same test case now takes 31 ms. Anything installed with apt, global toolchains and the home directory persist across a stop. The /tmp and /run directories begin empty each time.

Memory traps

An early design copied the image onto the disk at boot. With Microsoft’s 3.8 GiB devcontainers/universal image in an 8 GiB environment, the host kernel killed cloud-hypervisor within a minute. Kata adds 2 GiB of guest memory beyond the Pod limit by default, and the guest’s page cache counts as real host memory. The hypervisor crossed its cgroup limit while the guest still thought it had room.

The fix was to set the guest’s extra memory to 256 MiB with a Pod annotation and to use a custom RuntimeClass with 1024 MiB of overhead. Kosem tested its largest size, 52 GiB, under the new settings. The EROFS change also eliminated the multi-gigabyte copy that caused the crash.

Docker exec needs pivot_root

One bug survived every docker run check yet broke docker exec, compose healthchecks and nsenter. The cause was a chroot: runc joins a running container’s namespace and resolves paths against its real root. Kosem switched to pivot_root in a private mount namespace, as runc does, and now refuses to start an environment if the pivot fails.

Smaller lessons, and what is next

Kosem also keeps a warm pool of low-priority placeholder Pods, because a microVM that lands on a brand-new node waits two to four minutes. It names exact machine types, since a request for 8 vCPUs and 60 GB minimum led GKE to pick a pricier c4-standard-16, and it reviews Hyperdisk’s included performance of 3,000 IOPS and 140 MiB/s. The company says a follow-up covers the details, including YAML.

Source: kosem

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.