Fly.io has replaced HashiCorp Vault with Petsem, a home-built secrets store that cut the 99th-percentile time to load an app’s secrets from 10 seconds to 300 milliseconds. The Petsem secrets store also narrows what an attacker gets after breaking into a single worker server.
Short for Petsematary, Petsem became the source for customer secret reads on the Fly.io platform in July 2025. The company says it picked the name after the other crypt, coffin and vault names were taken.
Why Fly.io outgrew Vault
Fly.io was built on three HashiCorp tools: Consul for service discovery, Nomad for fleet orchestration and Vault for secrets. As the platform scaled, it replaced them one by one. Consul went in 2022, Nomad in 2023 and Vault in 2025. The company still calls the stack good, battle-hardened software.
The trouble with Vault was geography. Vault is, roughly, a disk-backed key-value store with an authenticated API, spread over a cluster that stays consistent through the Raft consensus protocol. Raft works well only when nodes sit close together, so the cluster lived in one region.
That produced a p99 of 10 seconds for apps loading secrets at boot. App startup also depended on the single region, IAD, being available, and apps on the other side of the world paid extra latency. A secrets read of 10 seconds or more on the boot path is a reliability problem as much as a speed one.
Trading consistency for speed
Fly.io decided that strict consistency was not worth the cost for this job. In its words, it isn’t running a bank. A read that returns a slightly stale value is acceptable.
A multi-region Vault under an enterprise agreement, backed by enterprise Consul, was possible. Fly.io estimated it would cost about as much each year as a fully loaded engineer’s salary. Petsem keeps a single write master in IAD but spreads read replicas across regions. That fixed both availability and latency, and the p99 fell to 300 ms.
Macaroons limit who can read what
Petsem is also shaped around how Fly.io runs apps. A fleet has thousands of worker servers, and only a few run any given app. Roughly 80 percent of apps on Fly.io live on a single worker. Only the workers hosting an app should be able to ask for its secrets.
Vault supports client policies, but an orchestrator would have had to update them each time an app was placed or moved. Timing gaps made that fragile, since a policy update could land after a Machine was already on a worker.
Fly.io used macaroons instead. A worker asking for a secret can’t just show its own credentials. It also needs a third-party discharge token from a separate service confirming that this caller has been granted access to the specific app. The worker gets the discharge by presenting the app’s service token, which the orchestrator hands over at placement. A read goes through only if both the worker’s credential and the discharge are valid.
How the migration ran
The migration began in March 2024, when the platform started writing new secrets to both stores. Existing secrets were backfilled soon after. By May, Petsem and Vault ran side by side, with Vault as the authority and Petsem as the fallback. After a long soak period, the company flipped the final switch in July 2025.
Vault isn’t gone. Fly.io still uses it in other parts of its infrastructure. It is just no longer load-bearing for customer secret reads.
What Petsem does not stop
Fly.io calls Petsem a speed bump, not a roadblock. Someone who compromises a worker host can reach guest memory and take whatever is there. What Petsem prevents is grabbing every secret at once. An attacker on one worker sees only secrets for apps deployed on that worker. To get more, they have to move sideways to other workers or attract new deployments to the one they hold. Denied reads are logged and set off alerts in the company’s observability pipeline.
The threat has a modern twist. Attackers used to hunt for keys to payment systems. Keys for AI inference providers are now a prime target, and Fly.io offers them through Connectors on Sprites and through Tokenizer and SSOkenizer for Machines.
Fly KMS and what comes next
The best secret is one an app never holds. For cases where an app needs only the result of an operation, such as a signature over a hash, Fly.io built Fly KMS on top of Petsem. A user writes a secret into Petsem, which then exposes encrypt, decrypt, sign and verify operations using it. Reads are not supported, so the secret never leaves Petsem or sits in app memory where it could leak.
The company says it is a little-known feature that more customers should try. The broader lesson is the one Fly.io draws itself: when a general-purpose tool sits on your boot path, check whether its guarantees match the ones you need.
Source: Fly