Should You Block AI Crawlers? A Straight Answer

Every few months a client forwards us an article about AI companies scraping websites, with one line attached: should we block them? The honest answer is that it depends on what your website is for, and most of the advice floating around answers for a newspaper when you run a dental clinic.

So let us do this properly. What blocking actually does, what it costs you, and a decision you can make in ten minutes.

What blocking actually does

There are two mechanisms people mean when they say "block AI crawlers", and they are not the same thing.

robots.txt. This is a plain text file at yoursite.com/robots.txt that lists which automated visitors are welcome on which parts of your site. It is a request, not a lock. A well-behaved crawler reads it and obeys. A badly behaved one reads it and carries on, or does not read it at all. The major AI companies publish the names of their crawlers and state that they respect robots.txt; whether every crawler on the internet claiming to be them actually is them is a different question.

Server or firewall blocking. This is enforcement — your server or your CDN (content delivery network, the layer that sits in front of your site and serves it faster) returns a 403 Forbidden to requests from a known bot, so nothing is served at all. Cloudflare and similar providers now ship with a one-click toggle for known AI crawlers. This genuinely works against bots that identify themselves honestly, and does nothing against a scraper pretending to be Chrome on a laptop in Noida.

Neither removes anything already collected. If your pages were crawled in 2023 and sit inside a model's training data, blocking today changes nothing about that. You are only affecting future crawls.

The two kinds of AI bot, and why the difference decides everything

This is the bit most articles skip, and it is the whole argument.

Training crawlers collect text to train future models. OpenAI's GPTBot, Common Crawl's CCBot, Google-Extended, Applebot-Extended, ByteDance's Bytespider and Meta's crawler broadly sit here. Blocking them reduces the chance your words end up inside a model's weights. It does not reduce your visibility in any product today, because these crawls feed models that may ship in a year.

Answer crawlers and live fetchers are how an AI assistant finds and cites a page right now. OpenAI's OAI-SearchBot indexes pages for search inside ChatGPT. ChatGPT-User fetches a page because a human asked a question that needed it. PerplexityBot and Anthropic's search and user-triggered agents work the same way. Block these and you remove yourself from the citation list. When somebody asks ChatGPT "good commercial interior contractors in East Delhi", your site is simply not among the pages it can read.

Google draws the same line deliberately. Google-Extended is documented as controlling whether your content is used for Gemini and related AI products — and Google states it has no effect on how you rank in Google Search, because it is a separate control from Googlebot. You can opt out of the AI training use and keep your search listing. That distinction is real and publicly documented. Treat anyone who tells you blocking Google-Extended will hurt your rankings with suspicion.

The decision

Ask one question: is your content the product, or is it the advertisement for the product?

If your content is the advertisement — allow everything

You run a clinic, a manufacturing unit, a law practice, a restaurant, a B2B services firm. Your website exists so that a human being decides to call you. Every page is, in effect, a sales argument you paid to publish.

Blocking AI crawlers here is self-harm. You gain nothing you can bank — a model knowing your service list costs you nothing — and you lose the one channel where visibility is still cheap to win. People are already asking assistants for recommendations, and the pages those assistants can read are the pages that get named. That is the whole premise behind our work on AI visibility.

For this group the correct robots.txt entry for AI crawlers is: nothing. Leave them alone.

If your content is the product — block training, allow answers

You run a publication, a research or data business, a stock photography outfit, a paid course, a recipe or template library. Someone pays for access to what you write. A model that absorbs your archive and answers the question directly has substituted for the thing you sell.

Here the split matters. Block the training crawlers. Keep the answer crawlers, so you still get cited and still get the click. You are trading away the bulk ingestion and keeping the referral.

Be realistic about what this buys you. You are not going to be compensated. Cloudflare has introduced a pay-per-crawl mechanism for site owners, which is an interesting development, but it is early and it is not a revenue line for a mid-sized Indian business today. Block because you do not want the substitution, not because you expect a cheque.

If you are somewhere in between

Most people are. A manufacturer with a genuinely useful technical knowledge base. A consultant whose blog archive took six years. The practical answer: allow everything, and change what you publish instead. Put the generic explanation out in the open where it gets you cited, and keep the specific, costly, firm-identifying material — pricing logic, your actual methodology, client-facing documents — behind a login or off the public web entirely. A crawler cannot take what you did not publish. That is a far more reliable control than a text file.

What to write, if you are blocking

A training-only block looks like this:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Three things people get wrong:

• Do not block Googlebot or Bingbot. Obvious, but it happens — somebody copies a list off a forum and takes the site out of search. Check /robots.txt after any change, on the live site, not the staging copy.
• robots.txt is not noindex. Disallowing a page stops it being crawled; it can still appear as a bare URL in results. If you want a page out of search, use a noindex meta tag on a page that is crawlable.
• Crawler names change. New ones appear every few months. A robots.txt written in 2023 is already out of date. If you block, review it twice a year — or accept that your list is partial.

The honest caveat on all of this: nobody outside these companies can verify what is actually ingested. Treat every confident claim about AI crawling, including the cautious ones here, as the best available reading rather than settled fact.

What we would not buy

An "AI content protection" retainer. Adding four lines to robots.txt and switching on a toggle in Cloudflare is a twenty-minute job, once. There is no ongoing service hiding inside it. If someone is quoting a monthly fee to keep your content out of AI models, ask exactly what they do in month two.

A promise that your content will be removed from existing models. No agency can deliver that.

Blanket bot-blocking bought as a security product. Aggressive rules routinely catch legitimate crawlers, uptime monitors, payment gateway callbacks and link previews. If you want fewer junk requests, that is a sensible server hygiene conversation, but it is a different problem from AI training and should not be sold as the same thing.

A "GEO package" that is mostly a robots.txt edit. Being readable by AI is largely the same work as being readable by a human in a hurry: clear claims, real detail, fast pages, pages that say who you are and where you are. That is ordinary good SEO with the emphasis moved.

What to do next

Open yoursite.com/robots.txt right now and read it. Most small business sites have something a developer pasted in years ago, and roughly one in a handful contains a rule nobody intended.

Then answer the single question: product or advertisement. If your pages exist to win you a phone call, do nothing — and spend the effort on making those pages worth quoting instead. If you genuinely sell access to what you publish, block the four training crawlers above, leave the answer crawlers alone, and diary a review for six months' time.

If you are unsure which side you fall on, or your robots.txt says something you cannot explain, send us the URL and we will tell you what it is currently doing.

Questions we get asked