robots.txt generator with AI crawler controls
Choose what to block, including AI training crawlers, and download a correct robots.txt file. Then test it with the robots.txt tester.
- No cost
- No sign-up
- No email
- No subscription
- Unlimited – runs in your browser
Should I block AI crawlers?
There are two kinds. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot and others) collect pages to train AI models; blocking them doesn't affect your search visibility. AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch pages so the assistant can show and link to them; blocking those removes you from those answers. Most businesses that want customers block training if they prefer and always allow search. Google's AI Overviews use normal Googlebot, so they can't be blocked separately without leaving Google.
What should I block in robots.txt?
Usually very little: admin areas, internal search results, baskets and checkout pages, and endless filter combinations on shops. Never block your CSS or JavaScript – Google needs them to render your pages.
Does robots.txt hide a page from Google?
No. It stops crawling, not indexing – a blocked page can still appear in results if other sites link to it. To keep a page out of Google, allow it to be crawled and add a noindex meta tag. Test your finished file with the robots.txt tester.
Frequently asked questions
Do all AI crawlers obey robots.txt?
The major providers (OpenAI, Anthropic, Google, Perplexity, Apple and Common Crawl) say their crawlers follow it. Badly behaved scrapers may not – for those you need a firewall or your CDN's bot protection.
More free SEO tools
Free SEO check
Full single-page audit: Google readiness, AI search, speed, security and local SEO.
AI SEO checker
Can ChatGPT, Claude, Perplexity and Google AI Overviews read and cite your site?
Website audit
Crawl up to 250 pages for broken links, duplicate titles and orphan pages.
Competitor comparison
Your site against up to two competitors, side by side.
SEO monitoring
A free weekly re-check with a score history graph.
Title tag checker
See your title and description in pixels, exactly as Google cuts them.
robots.txt tester
Paste your rules and test any URL against Googlebot or AI crawlers.
llms.txt generator
Create an llms.txt file that gives AI assistants a map of your site.
Mobile-friendly test
Check your page works on phones – Google's own test closed in 2023.
Broken link checker
Find links on your site that lead to errors or redirects.
Schema checker
Validate your structured data and see Google's required properties.
Page speed checker
Core Web Vitals from real Chrome users plus a Lighthouse test.
Sitemap checker
Check your XML sitemap is found, valid and listed in robots.txt.
Security headers checker
HTTPS, HSTS and the headers browsers use to protect visitors.
Local SEO checker
LocalBusiness markup, contact details and map signals for UK businesses.
Meta tag generator
Title, description, canonical, robots and social tags, ready to paste.
Open Graph generator
Control how your link looks on Facebook, LinkedIn, WhatsApp and X.
XML sitemap generator
Find your pages and download a valid sitemap.xml.
Local business schema generator
LocalBusiness JSON-LD for Google and AI assistants.
Long-tail keyword generator
Turn one topic into dozens of winnable search phrases.
Blog title generator
Headline ideas measured to fit Google's results.
Word counter
Words, characters, reading time, readability and keyword density.
Keyword difficulty checker
How hard is page one? We grade the pages that rank now.
Plagiarism checker
Find copies of your text across the web – no word limit.
Anchor text checker
Grade every link on a page and generate better anchors.
Run a free SEO check now
About 50 checks covering Google, AI search, speed, security and local SEO. No sign-up.
Want the fixes done for you? XKey is made by IceWork, a web studio in Blackpool. IceWork builds fast, search-ready websites – a 3-page site including domain and a year's hosting is £350 all in.