How to block AI training bots but stay in AI search

Many businesses want to stop AI companies copying their content for training – without disappearing from the AI assistants their customers now use to find them. With the right robots.txt rules, you can do both.

Last updated · By the XKey team

Training crawlers and search crawlers are different

Most AI companies now use separate crawlers for separate jobs. Training crawlers collect pages to build future AI models. Search crawlers fetch pages so an assistant can find, quote and link to you when someone asks a question. A third kind fetches a page only when a user asks the assistant to look at it. Blocking a training crawler doesn't stop the search crawler from the same company.

The main crawlers to know

  • OpenAI – GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (pages a user asks about).
  • Anthropic – ClaudeBot (training), Claude-SearchBot (search), Claude-User (pages a user asks about).
  • Perplexity – PerplexityBot (search), Perplexity-User (pages a user asks about).
  • Google – Googlebot crawls for Search, including AI Overviews and AI Mode. Google-Extended isn't a separate crawler but a robots.txt token that controls whether your content is used for Gemini models; blocking it doesn't affect Google Search.
  • Apple – Applebot (search features such as Siri and Spotlight); Applebot-Extended controls use for Apple's AI training.
  • Common Crawl – CCBot builds a public web archive that many AI models have been trained on.

A robots.txt that blocks training and allows search

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.co.uk/sitemap.xml

OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot aren't named, so they follow the User-agent: * group and stay allowed. Create a version tailored to your site with the robots.txt generator, then test individual pages with the robots.txt tester.

Things robots.txt can't do

  • It's a request, not a lock. Reputable companies follow it; badly behaved scrapers ignore it. For those, firewall rules are the answer – Cloudflare, for example, offers a setting to block known AI crawlers at the network level.
  • It isn't retrospective. Content already collected for past training isn't removed.
  • Crawler names change. New bots appear every year, so review your rules every few months.

Watch out for blocking everything by accident

Some security plugins, hosting firewalls and "block AI" settings stop search crawlers as well as training crawlers. If your business never appears in ChatGPT or Perplexity answers, that's one of the first things to check. The AI SEO checker fetches your robots.txt and shows which AI crawlers are allowed in, one by one.

For more on getting recommended by AI assistants, read our AI search optimisation guide.

Frequently asked questions

Will blocking GPTBot remove me from ChatGPT?

No. ChatGPT's search feature uses OAI-SearchBot, which is controlled separately. Block GPTBot and allow OAI-SearchBot to stay findable without contributing to training.

Does blocking Google-Extended affect my Google rankings?

No. Google says Google-Extended doesn't affect inclusion or ranking in Google Search, including AI Overviews.

How often should I update my AI crawler rules?

Every few months, or whenever a major AI company announces a new crawler. The robots.txt generator is kept up to date with the main ones.

Related guides

Robots.txt explained

What robots.txt does, what a safe file looks like for a small business, the mistakes that hide whole sites from Google, and how to test yours free.

What is llms.txt?

What llms.txt is, who uses it, what Google says about it, and whether it's worth adding to your website – an honest, plain-English answer.

How to appear in ChatGPT

How ChatGPT finds businesses to recommend, the crawlers your site must allow, and practical steps to make a small business show up in ChatGPT answers.

Technical SEO checklist

A plain-English technical SEO checklist: indexing, sitemaps, redirects, canonicals, speed, mobile and security, with a free tool for each item.

Run a free SEO check now

About 50 checks covering Google, AI search, speed, security and local SEO. No sign-up.

Want the fixes done for you? XKey is made by IceWork, a web studio in Blackpool. IceWork builds fast, search-ready websites – a 3-page site including domain and a year's hosting is £350 all in.