AI Crawlers Take 11,000 Pages Per Visit. The Pushback Just Started.

AI crawlers fetch 11,000 pages from your site for every visit they send back. That is not a search relationship.

Cloudflare's Radar data for May 2026, analyzed by TechnologyChecker, measures something most SEO dashboards don't show: the ratio between pages a crawler fetches and human visits it returns. Anthropic's ClaudeBot: 11,122 pages per referral. OpenAI's GPTBot: 857. PerplexityBot: 190, up from 95 in April. Google: 5.

DuckDuckGo, for reference, sits at 1.5 pages per visit. The most reciprocal ratio in the dataset.

Mehmet Suleyman, who compiled the Cloudflare Radar analysis and previously worked on Microsoft's Bing and AI team, put it plainly: "For every 1 human visit Claude.ai sends back to a publisher, Anthropic's ClaudeBot has already fetched 11,122 pages from that publisher. It isn't even a bad search relationship. It's a data extraction pipeline with almost no compensating traffic flow."

All AI chatbots combined account for 0.29% of search referral traffic. Google sends 87.6%.

Someone shipped an off switch

The same week those numbers circulated, Microsoft's President of Search Jordi Ribas announced a browser extension called Bing AI Search Choice. One click turns off AI-generated answers in Bing search results. Users can also type "-ai" after any Bing query to suppress Copilot summaries.

"Not everyone wants to use AI for everything all the time," Ribas wrote on X.

The extension is available for Chrome and Edge. It also sets Bing as the default search engine and opens Bing on every new tab, which makes it a distribution play as much as a user-choice gesture. But the demand behind it appears genuine. DuckDuckGo weekly installs in the US surged 30% after Google's AI search overhaul, per Windows Central.

When a search company that spent years building AI into its product ships a toggle to turn that AI off, the internal usage data must be saying something uncomfortable.

More crawlers, fewer visits

Humans are now less than 45% of web traffic, according to the same Cloudflare dataset. Bots generate 46.97%. AI bots specifically account for 5.54%, up sharply from a year ago. Bytespider, ByteDance's crawler, doubled its share of AI bot traffic from 5.73% in April to 10.25% in May.

More crawlers are showing up. The referral visits they return are not.

Ahrefs' AI Benchmark from May 2026 found that 71% of cited sources appear on only one AI platform. If ChatGPT cites your page, there is roughly an 89% chance Perplexity will not. Different platforms, different source pools, and neither one sends traffic proportional to what it extracts.

Over half of all AI crawler traffic (51.8%) is classified as training data collection, according to Cloudflare's breakdown. Only 9.33% is tied to search products, the category that actually has a referral mechanism. The rest is extraction for model training, with no traffic flow back to publishers.

What changes

Google's 5:1 crawl-to-refer ratio still looks like search. Crawl pages, send visitors. The old bargain. ClaudeBot's 11,122:1 does not. If your discoverability strategy assumes anything reading your content might become a customer, the Cloudflare data says that assumption broke sometime in the last year.

The Bing toggle is a small signal, but it is the first time a major search company has acknowledged that users might want less AI in their search results, not more. Ribas could have framed the extension as a power-user feature. He framed it as a choice. That wording was deliberate.

Sources: TechnologyChecker / Cloudflare Radar, Windows Central, PPC Land, Ahrefs AI Benchmark