Robots.txt explained for small business websites

Last updated 13 October 2026 · By the XKey team

What does robots.txt actually do?

Before crawling a site, well-behaved bots read /robots.txt and follow its rules. Each group starts with User-agent: (which bot the rules are for, or * for all) followed by Disallow: and Allow: lines naming folders or pages. It controls crawling, not indexing: a blocked page can still appear in Google with no description if other sites link to it. To keep a page out of Google, use a noindex tag and let Google crawl it.

What should a small business robots.txt look like?

For most small sites, a short file is best:

User-agent: *
Disallow: /basket/
Disallow: /my-account/

Sitemap: https://www.example.co.uk/sitemap.xml

Block only private or useless areas such as baskets, checkouts, account pages and internal search results. Everything you want to rank must stay open, including the CSS, JavaScript and images Google needs to see your pages properly. Build one with the robots.txt generator.

Which robots.txt mistakes hurt SEO?

• Disallow: / under User-agent: * – blocks the whole site. It's often left behind when a site moves from a test address to live.
• WordPress "Discourage search engines" ticked under Settings → Reading.
• Blocking folders that hold CSS, scripts or images, so Google sees a broken page.
• Using robots.txt to hide duplicate pages instead of canonical tags. See duplicate content.
• A robots.txt that returns a server error: Google may pause crawling the whole site until it's fixed.

How do crawlers read the rules?

A few details catch people out, even on simple files:

• Each bot follows one group only. If there's a group for Googlebot, Googlebot follows that group and ignores the User-agent: * group completely. Repeat any shared rules in every named group.
• The most specific rule wins. Google applies the rule with the longest matching path, so Allow: /blog/ beats Disallow: / for blog pages. When an Allow and a Disallow match equally, Google uses Allow.
• Paths are case-sensitive. Disallow: /Admin/ doesn't block /admin/.
• Wildcards work in Google and Bing: * matches anything and $ marks the end of an address. Disallow: /*?sort= blocks sorted versions of category pages; Disallow: /*.pdf$ blocks PDF files.
• One file per host. The file at www.example.co.uk/robots.txt doesn't cover shop.example.co.uk – every subdomain needs its own.
• Some lines are ignored by Google. Google doesn't follow Crawl-delay (Bing does) and stopped supporting noindex in robots.txt in 2019.

Editing robots.txt on WordPress, Shopify, Wix and Squarespace

• WordPress makes a virtual robots.txt for you. Yoast and Rank Math both have a file editor in their tools settings, or you can upload your own file to the site's root folder. Check Settings → Reading isn't set to discourage search engines.
• Shopify creates a sensible file automatically. You can change it by adding a robots.txt.liquid template to your theme, but most shops never need to.
• Wix has a robots.txt editor in its SEO settings.
• Squarespace creates the file for you and doesn't let you edit it. You can still keep individual pages out of Google with each page's SEO settings.

What about test and staging sites?

Don't rely on robots.txt to hide a test site – Google can still list its addresses if anything links to them. Put it behind a password instead. If you do use Disallow: / or a noindex tag on a staging copy, add "remove the block" to your launch checklist: it's the most common reason a brand-new website never appears in Google. Our website redesign guide covers the rest of a safe launch.

Can robots.txt block AI bots?

Yes. You can disallow AI training crawlers such as GPTBot or Google-Extended while still allowing Googlebot and the bots that power AI search answers. Our guide to blocking AI training bots explains which are which, so you don't vanish from AI search by accident.

How do I test my robots.txt?

Paste any page address into the robots.txt tester to see whether Googlebot and other crawlers may visit it and which rule decides. Our free SEO check also flags a missing, broken or blocking robots.txt on the page you check.

Frequently asked questions

Does every website need a robots.txt file?

No – without one, bots assume they can crawl everything, which is fine for many small sites. Adding one is still useful to point bots to your sitemap.

Is robots.txt a security measure?

No. It's public and only polite bots follow it. Protect private pages with a login, not robots.txt.

How quickly does Google notice robots.txt changes?

Google usually caches robots.txt for up to a day, so changes take effect within about 24 hours.

Will blocking a page in robots.txt remove it from Google?

Not reliably. Google can still list a blocked address if other pages link to it. Use a noindex tag, and leave the page crawlable so Google can see the tag.

Should I block my images or CSS files?

No. Google needs your CSS, JavaScript and images to see the page as visitors do, and blocking images also keeps them out of Google Images.