AI Search Can't See Your Site? The 5-Minute robots.txt Check

Something odd is happening on the web right now. AI assistants answer more shopping and research questions every month, but websites are blocking the bots that feed those answers at a record pace.

New numbers tell the story. Sites on Cloudflare’s network denied 13.47% of AI bot requests in the third quarter of 2026, up from 5.98% a year earlier, according to Cloudflare Radar data analyzed by TechnologyChecker.io and reported October 9. A separate snapshot of 4,242 robots.txt files found sites block GPTBot 1.6 times for every site that allows it. Meanwhile the crawlers keep knocking: search-triggered AI crawl requests nearly doubled from 6.51% of AI crawl activity in Q2 to 11.92% in Q3.

Read that twice. AI systems want to read your site more than ever. Site owners are saying no more than ever. Somewhere in the middle, businesses are becoming invisible to AI search without realizing it.

Most blocks are accidents, not strategy

Here’s the uncomfortable part: in our experience, most sites blocking AI crawlers never made that decision on purpose.

It usually goes like this. Someone enables Cloudflare’s one-click “block AI bots” toggle during setup and forgets it exists. A WordPress security plugin ships with aggressive bot rules and the owner never reviews them. A developer copies a robots.txt template from a forum post in 2024 that disallows every bot it can name. A store gets hammered by scrapers one weekend, blocks everything in a panic, and nobody reverts it on Monday.

We’ve seen the panic version more than once. A site under a scraping attack turns on every protection available, the attack stops, and six months later the owner wonders why ChatGPT never mentions their brand while competitors show up in every answer. The block did its job. Then it kept doing its job long after the threat left.

That’s the pattern to watch for: a temporary defense that quietly becomes a permanent handicap.

The check takes five minutes

You don’t need an audit or a consultant for this. Open a browser and go to yoursite.com/robots.txt. Read it like a bouncer’s guest list.

Look for lines naming these crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Bytespider. If you see Disallow: / next to any of them, that crawler is turned away at the door. A blanket User-agent: * followed by Disallow: / blocks everyone, AI or otherwise — and yes, we still see live business sites running exactly that.

Second, check your CDN or firewall. This is the step most guides skip, and it matters: a firewall or bot-protection rule can return a 403 denial even when robots.txt allows the crawler. In Cloudflare, that’s Security → Bots. In other setups, it’s whatever managed bot protection your host enabled. If server logs are available, search them for GPTBot or PerplexityBot entries and look at the response codes. A wall of 403s next to crawler user-agents is your answer.

Third, don’t trust a single tool’s verdict blindly. Free robots.txt testers sometimes cache old versions of the file. Fetch the live file yourself. It takes thirty seconds and it’s the ground truth.

Which bots to let in (our take)

Not every crawler deserves access, and anyone telling you to allow everything is selling something. Here’s how we’d draw the line.

Allow the crawlers behind assistants that cite sources and send traffic: GPTBot and OAI-SearchBot (ChatGPT), ClaudeBot (Claude), PerplexityBot (Perplexity). These are the bots that read your pages so an assistant can quote you. When someone asks “what’s the best budget espresso grinder,” the assistant can only recommend a product page it was allowed to read.

Be more cautious with pure training crawlers like Google-Extended and Bytespider if bandwidth costs worry you — they feed model training, not direct citations. That’s a legitimate business call either way. What isn’t legitimate is blocking everything by default out of vague unease. That’s the digital equivalent of locking your shop because window shoppers annoy you. You’re not stopping theft; you’re stopping customers.

One more thing people get wrong: blocking AI crawlers doesn’t meaningfully protect your content from being used in training anyway. Your pages are already in Common Crawl and a dozen other datasets. The block mostly costs you visibility in answers, not privacy.

What invisibility actually costs

This isn’t abstract. We recently wrote about getting your store ready for AI shopping agents — structured product data, clear pricing, fast pages. All of that preparation is worthless if the agent’s crawler bounces off your robots.txt. The agent can’t recommend what it can’t read, and it will recommend a competitor’s product instead. That’s not a theory; it’s how retrieval works.

The direction is one-way. Rank Math predicts AI visibility becomes a standard SEO metric in 2027. Rezolve AI’s October announcements cited a McKinsey estimate of $3 trillion to $5 trillion in agent-mediated commerce by 2030. Every quarter you spend invisible is a quarter where competitors’ pages become the default answers and yours don’t exist.

And once your crawler access is fixed, what the bots find matters too. Product descriptions written to convert pull double duty now — they sell to humans and they’re the exact text AI assistants quote back. Thin, duplicated manufacturer descriptions give assistants nothing to cite.

The honest caveat

Allowing crawlers doesn’t guarantee AI assistants will cite you. They won’t. Assistants cite a handful of sources per answer, they favor pages that directly answer the question, and nobody outside the labs knows the full ranking recipe. Fixing robots.txt moves you from “impossible” to “possible,” which is a big move, but it’s the start of the work, not the end.

Also, keep the data in perspective: the 13.47% figure covers Cloudflare’s network only, and TechnologyChecker.io labels the Q3 numbers preliminary. Your site’s reality could be better or worse. That’s exactly why you check your own logs instead of trusting an industry average.

Different assistants also run different crawlers, so “allow AI bots” isn’t one switch. ChatGPT, Gemini, and Claude each handle crawling differently, which means a robots.txt that welcomes one assistant can still shut out another. Check the names, not just the concept.

Do it today

Open your robots.txt. It takes less time than reading this article did. If you find AI crawlers on the block list and you didn’t put them there deliberately, that’s a five-minute fix for a problem that’s been compounding for months.

The web is splitting into sites AI can read and sites it can’t. Make sure you’re in the first group on purpose, not the second one by accident.