Instant tool

robots.txt tester

Paste a robots.txt file, choose a crawler and test a URL. We apply Google's documented rules: the most specific matching group, longest matching rule, and Allow winning ties.

How does robots.txt matching work?

  • Groups: a crawler obeys the group that names it most specifically; if none does, it uses the User-agent: * group.
  • Longest match wins: when both an Allow and a Disallow rule match, the longer (more specific) rule applies.
  • Ties go to Allow.
  • * matches any characters and $ marks the end of the URL.

What are the most common robots.txt mistakes?

  • Disallow: / left over from a development site, blocking everything.
  • Blocking CSS and JavaScript, which stops Google rendering the page properly.
  • Using robots.txt to hide pages from search – it stops crawling, not indexing. Use noindex instead.
  • Blocking AI search crawlers by accident when you only meant to block training crawlers.

Frequently asked questions

Does robots.txt stop a page appearing in Google?

No – it stops crawling, not indexing. A blocked page can still be indexed from links pointing to it, just without a description. To keep a page out of Google, allow crawling and add a noindex tag.

Where does robots.txt go?

At the root of your domain, e.g. yoursite.co.uk/robots.txt. Each subdomain needs its own file.

More free SEO tools

Free SEO check

Full single-page audit: Google readiness, AI search, speed, security and local SEO.

AI SEO checker

Can ChatGPT, Claude, Perplexity and Google AI Overviews read and cite your site?

Website audit

Crawl up to 250 pages for broken links, duplicate titles and orphan pages.

Competitor comparison

Your site against up to two competitors, side by side.

SEO monitoring

A free weekly re-check with a score history graph.

Title tag checker

See your title and description in pixels, exactly as Google cuts them.

llms.txt generator

Create an llms.txt file that gives AI assistants a map of your site.

Mobile-friendly test

Check your page works on phones – Google's own test closed in 2023.

Broken link checker

Find links on your site that lead to errors or redirects.

Schema checker

Validate your structured data and see Google's required properties.

Page speed checker

Core Web Vitals from real Chrome users plus a Lighthouse test.

Sitemap checker

Check your XML sitemap is found, valid and listed in robots.txt.

Security headers checker

HTTPS, HSTS and the headers browsers use to protect visitors.

Local SEO checker

LocalBusiness markup, contact details and map signals for UK businesses.

Meta tag generator

Title, description, canonical, robots and social tags, ready to paste.

Open Graph generator

Control how your link looks on Facebook, LinkedIn, WhatsApp and X.

XML sitemap generator

Find your pages and download a valid sitemap.xml.

robots.txt generator

Block AI training bots while staying visible in AI search.

Local business schema generator

LocalBusiness JSON-LD for Google and AI assistants.

Long-tail keyword generator

Turn one topic into dozens of winnable search phrases.

Blog title generator

Headline ideas measured to fit Google's results.

Word counter

Words, characters, reading time, readability and keyword density.

Keyword difficulty checker

How hard is page one? We grade the pages that rank now.

Plagiarism checker

Find copies of your text across the web – no word limit.

Anchor text checker

Grade every link on a page and generate better anchors.

Run a free SEO check now

About 50 checks covering Google, AI search, speed, security and local SEO. No sign-up.

Want the fixes done for you? XKey is made by IceWork, a web studio in Blackpool. IceWork builds fast, search-ready websites – a 3-page site including domain and a year's hosting is £350 all in.