How to block AI crawlers from your website

You block AI crawlers the same way you block any crawler: with rules in your robots.txt file naming each bot and disallowing it. The important thing to understand before you start is what those rules are — a request that well-behaved crawlers honor, not a technical barrier that stops a determined one. If you need a wall rather than a sign, you have to block at the server, and this guide covers both.

Decide what you are actually blocking, and why

Blocking AI crawlers is a real trade-off, not a free safety measure. Allowing them means your content can be used to train models and can be quoted in AI answers, with a citation back to you. Blocking them protects the content from both — and removes you from those answers entirely. If your traffic increasingly comes from people asking assistants for recommendations, that is a cost worth naming out loud.

Most sites that want to block also want to keep normal search traffic. Those are separate decisions controlled by separate mechanisms, and it is entirely possible to stay fully indexed by Google while refusing every AI crawler. The rest of this guide assumes that is the goal.

The AI crawlers and their user agents

Each AI company crawls with named user agents you can target individually. The ones worth knowing:

• GPTBot and OAI-SearchBot — OpenAI, for training and for its search feature. ChatGPT-User is a third: it fetches a page when a user pastes your link into ChatGPT and asks about it.
• ClaudeBot — Anthropic’s crawler for Claude.
• PerplexityBot — Perplexity’s crawler.
• Google-Extended — Google’s control for Gemini and Vertex AI, covered separately below because it does not do what most people assume.
• CCBot (Common Crawl, whose archive trains many models), Bytespider (ByteDance), Amazonbot, Meta-ExternalAgent and Applebot-Extended round out the list most sites care about.

The robots.txt rules

Each bot needs its own block: a User-agent line naming it, then a Disallow: / to refuse the whole site. A block for one agent does not cover the others, so you list them all:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Add these to the robots.txt already at your site root rather than creating a second file, and leave any existing rules for Googlebot and Bingbot untouched — those are your search crawlers, and blocking them would take you out of ordinary results, which is not what you are trying to do.

Google-Extended does not remove you from Google’s AI

This is the point nearly every guide gets wrong. Google-Extended controls whether your content is used to develop Gemini and Vertex AI. Google has stated it does not affect your inclusion or ranking in Google Search. That much is straightforward.

The part that surprises people: AI Overviews, the AI summaries at the top of Google’s results, are part of Search and draw on the normal Googlebot index — not on Google-Extended. So blocking Google-Extended does not take you out of AI Overviews. Reducing how your content appears there is a Search-snippet decision instead, made with nosnippet or max-snippet directives — and those limit your ordinary snippet too, so they are a real trade-off rather than a clean opt-out.

robots.txt is a request, not a wall

Everything above depends on the crawler choosing to obey. robots.txt has no enforcement of its own; it is a convention, and honoring it is voluntary. The major, named AI crawlers generally do — but in 2024 reporters documented cases of AI services fetching pages from sites that had disallowed their crawler, sometimes through undeclared user agents or user-triggered requests that sidestep the rule. There is also no universal “noindex for AI”: the meta tags that work for search engines are not a standard AI crawlers are bound to respect.

So a robots.txt block is the right first step and stops the compliant majority, but treat it as a clearly posted sign rather than a locked door.

If you need enforcement, block at the server

The only thing that actually prevents a fetch is refusing it at the server or CDN, where you are not asking politely. You can deny requests by user agent or by IP range in your web server config, and most CDNs — Cloudflare among them — now offer a one-switch option to block known AI bots, which they maintain as the bot list changes so you do not have to.

Server-level blocking is more work and can occasionally catch a legitimate visitor if a rule is too broad, so it is worth it when the content genuinely needs protecting and overkill when a robots.txt sign would do. Match the effort to what you are actually guarding. If the underlying worry is that a block might also be hurting your normal search visibility, the guide on what “blocked by robots.txt” means walks through how to tell the two apart.

Check what you are allowing right now

Before you change anything, see where you stand. Our robots.txt checker reads your file and reports which of the named AI crawlers you currently allow and which you refuse, so you are editing from facts rather than guessing. Plenty of sites discover they are already blocking bots they meant to allow, or allowing ones they meant to block, because a plugin default or a copied file made the choice for them.

If your aim is the opposite — you want assistants to read and cite you — the guide to GEO optimization and the GEO checklist cover how to make that happen. And to see everything an AI crawler encounters on your site in one pass, run the free website check.