AI Prompt Leaks: Two Million Canary Prompts Tested

AI prompt leaks traced through inference routers and providers

A security research team sent nearly two million prompts to AI services, each carrying a unique fake secret, and found that AI prompt leaks are happening at several layers of the inference chain. Between July and November 2026, the seeded credentials and links showed up in places they should never have reached, including attacker logins and an internal chat in China.

The test was simple. Every prompt held a trackable URL and canary credentials, such as AWS keys tied to empty, fully logged accounts. If anyone touched one, that person had seen the original prompt.

Three providers were breached

The most serious result: attackers were stealing data from three AI hosting providers. The earliest incident dates to mid-July, and none of the three knew about the breach until the researchers told them.

The largest case involved a couple of hundred AWS keys, each sent once, all within a few windows in late October. Every request sent in those windows was copied. Some stolen keys came from requests the provider rejected before any model ran, which suggests the attacker was copying traffic at the provider’s front door. In two of the breaches, stolen requests were even flagged for Zero Data Retention (ZDR), the strictest privacy setting. That flag does nothing once an attacker is inside.

The attackers then followed a predictable routine. They checked whether each key was live, sometimes within hours. They listed S3 buckets, searched Secrets Manager and Parameter Store, tried to enable hosted models in AWS Bedrock, created Bedrock API keys and added their own IAM users. In total the bait keys drove tens of thousands of AWS API calls.

One adversary appears to have compromised two providers at once, using secrets from both from a single rented server. Over three weeks its tooling shifted from a fake Chrome script to Boto3 and then the AWS CLI on Kali Linux. On November 1 it added ARTEX, an LLM-driven penetration testing framework, though in this case ARTEX only confirmed that keys were still valid. Two of its user agents, artex-agent and artex-judge-r121, appear in no public indicator list the researchers could find, so defenders may want to add them to detections.

Figure 1 in Blog for stolen aws key
Image: Dragos

Marketplace sellers read customer prompts

The second finding needed no exploit. Sellers on an AI inference marketplace could list a popular model at the lowest price, switch on request logging and wait for customer prompts to arrive. The people behind it looked like individuals on home internet connections.

In one case, two prompts sent through a single seller were opened about a day and a half later from a home broadband connection, and someone began guessing usernames on the login page. Prompts for the same model through a different seller were never touched, so the leak followed the seller rather than the marketplace. In another, a user on a Mac in France typed passwords from two prompts into root SSH logins by hand. One router was quietly forwarding part of its traffic to that marketplace.

Blog Figure #2 Adversary Tools
Image: Dragos

Privacy settings were ignored

The third group of AI prompt leaks involved no stolen credentials. Companies that were never meant to see the prompts fetched the secret URLs anyway, and tried to clone the git repositories they pointed to.

A Chinese AI lab’s systems fetched the URLs from more than half the prompts sent to its models between July and October. That included roughly two-thirds of the requests marked “no data collection” or ZDR. The scraper usually appeared two or three days after the request. Several U.S. hosts resell the lab’s models, and prompts sent to them triggered the same scraper, so a customer who chose a U.S. provider to keep data at home did not get that.

Large U.S. providers had problems too. One kept crawling a secret URL more than two months after a single call to its search endpoint. Another ran a scraper that came back days after each request, whichever router was used.

The oddest case: 24 days after a prompt went to one Chinese provider, its secret link appeared in a Feishu chat channel and was opened within two minutes on three desktops and three phones in Beijing and Shanghai. Someone was reading a stored copy of the full prompt, weeks after it was sent with the ZDR flag to a U.S. provider.

Figure 3 Blog SSH Root Attempt
Image: Dragos

Hidden hops in the route

The team also built a traceroute for inference, using an image URL and deliberately broken requests whose nested errors reveal each hop. Of about two thousand routes traced, several hundred included an unidentified system, and one marketplace seller quietly adds about a thousand hidden tokens to every prompt.

What teams should do about AI prompt leaks

The researchers do not say to stop using AI services. They say the route your prompts take is a choice, and the lowest price will make it if you don’t. A privacy flag only binds the first hop, and agents will send anything they can read, credentials included.

  • Host models yourself if you can afford it. A Mac Studio with 512 GB of memory starts at $22,299, and an eight-GPU B300 server lists for $570,000 or more.
  • Go directly to a first-tier provider, which gives you one contract and one privacy policy.
  • Choose upstream providers yourself when using routers, and avoid reseller marketplaces, some of which run on stolen keys.
  • Seed your own canaries by planting a fake credential in the context your agents already send. A free Thinkst Canarytoken is enough to start.
  • Turn on secret filtering where offered, though it won’t catch sensitive data that doesn’t look like an API key.

Where disclosures stand

The team is now working with about a dozen providers. The router that forwarded traffic to the marketplace says it stopped, and one breached provider has published a preliminary security notice. Another has still not disclosed its incident publicly, and stolen keys and secret URLs were still being used months after the first leak.

Source: Dragos

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.