Cloudflare just changed the AI crawler defaults, and there's a trap in it

Cloudflare sits in front of more than 20% of the web, so when it changes which bots get in by default, the whole AI-visibility baseline moves with it. On 1 July it changed a lot.

Start with the headline. From 15 September, new Cloudflare domains will block Agent and Training bots by default on any page that shows ads.

Search stays allowed. “Agent” is the word doing the work there. It covers ChatGPT-User and the browser-use agents driving Chrome, the ones that fetch your page live to answer someone’s question right now.

So on the pages you actually monetise, real-time AI answers are off unless you opt back in.

Then the trap, and this is the bit nobody’s leading with. Cloudflare now judges a multi-purpose crawler by its strictest rule. Googlebot, Bingbot and Applebot all crawl for both search and training.

So the moment you tick “block AI training”, you also block Googlebot. A setting that looks like it only touches AI scrapers can quietly take you out of Google. If you go near these controls, read that twice.

Please.

A couple more things.

“Verified” no longer means “allowed”, a verified bot is now only let in inside a category you’ve chosen to allow, so being a known-good bot isn’t a free pass any more.

There’s a new robots.txt signal, use=immediate|reference|full, for what a bot can keep and reshare after reading you, with reference (index and link, don’t reproduce) as the default Cloudflare writes into its managed robots.txt.

I haven’t flipped these settings in anger yet, this is me checking the announcement. But the direction is unambiguous.

The default web is getting less open to AI agents, and the people most likely to get caught out are the ones who tick “block AI” without realising Google rides the same crawler.

If you want to know what your own edge is actually doing to AI crawlers right now, I built a free AI Crawler Access Checker that probes your site as GPTBot, ClaudeBot, PerplexityBot and the live fetchers and tells you who’s getting through.