Aggressive crawlers are knocking over websites and open APIs worldwide, and nobody has a clear answer on who runs them. One analysis of the AI scrapers problem argues the worst traffic isn’t coming from the big labs at all, but from small data brokers selling bulk text to AI start-ups. It’s a theory, not a proven finding, and the author says so.
Why this wave of bots is different
Scraping isn’t new. Sites have dealt with bots for years, and the biggest crawlers, Google’s among them, tend to behave. A common argument is that search engines have a reason to be gentle: they send readers back to the sites they crawl, so a crashed site hurts them too.
AI bots, the argument goes, only want the data and don’t care whether the site survives. The analysis was skeptical of that at first. Hammering a site is inefficient, and taking it down cuts off your own access. Polite crawlers go unnoticed, while rude ones get blocked.
The explanation also seems aimed at the large labs: OpenAI, Anthropic, ByteDance and Meta. Many people at those companies came from earlier crawling teams, so it’s hard to see why their habits would suddenly get worse.
The traffic nobody can name
Site maintainers do see GPTBot, ClaudeBot and Bytespider in their logs. But the load they call unmanageable doesn’t identify itself. It arrives from tens of thousands of residential addresses, with bots posing as ordinary browsers. In that setting, even bots that carry a label can’t be trusted to be who they claim.
Cloudflare’s figures point mostly at bots collecting training data. That figure only counts bots Cloudflare can identify. It also leaves a puzzle: why isn’t more of the load coming from ordinary users whose AI agents browse on their behalf?
A gold-rush theory
The author’s working explanation follows the money. Start-ups, mostly in the US and China, have raised large sums to build proprietary models. They then face a shortage of people who know how to build a working model, and the one resource they know they need is data. Industry orthodoxy says quantity beats quality, so they pay well for raw data in bulk and don’t ask where it came from.
That creates a market for sellers. These are firms paid more or less by the byte, with presumably some uniqueness test, since nobody pays for a petabyte of zeros. Their goal is as much novel data as possible, as fast as possible. They sit far down the chain from the venture funds and big AI companies, and they have no reason to be polite.
The buyers aren’t excused, the analysis adds, since they pay without asking how the data was gathered. But the model explains how the incentives produce the behavior.
It leaves loose ends. Why don’t those start-ups just use existing open models? Why haven’t the data sellers consolidated or coordinated to cut costs? The author suspects the answer is the short life of this market and the usual chaos of a gold rush.
The bills aimed at bots
Bad bot behavior is already being turned into a political weapon. New York’s Bill S9934A passed in June and awaits the governor’s signature. It would let the state attorney general sue operators who crawl news sites without declaring who they are and what they plan to do with the data. In Congress, a similar bill has been pending since July, and its Senate version, the Stealth Bot Prohibition Act, was introduced this week.
Senator Wicker, introducing it, framed the issue around Chinese intellectual property theft and malicious bots aimed at online publishers. Critics see a different effect. Stealth bots will ignore the law, while law-abiding crawlers will have to announce themselves. That would let a publisher such as the New York Times block OpenAI and Anthropic until they buy a licence for non-paywalled content.
NetChoice and the EFF oppose the bills. Much of the tech industry has stayed quiet, partly because it dislikes the bots and partly because it distrusts publishers legislating a business model into existence.
What site operators can expect next
The analysis ends without a fix. Millions of site operators are still working out how to cope, and the author expects the proposed laws to leave the real problem untouched while making the internet less free for everyone else.
The author also notes a different worry: a future where every user has several always-on agents hunting for deals and news. For now, though, the current training-data rush looks like the bigger threat to site stability. If the gold-rush theory is right, the traffic will fade when the buying slows, and until then operators are left to absorb the load.
Source: Oblomovka