Free website crawler
Crawl a whole website and get an SEO issues report, a score out of 100 and a spreadsheet of every page – titles, descriptions, headings, word counts, links, contact details and product prices.
What the website crawler gives you
A row for every page
Status code, title, description, H1, headings, word count, response time, canonical, robots tag and the page that first linked to it.
SEO issues and a score
Broken pages and where they're linked from, missing or duplicate titles and descriptions, missing H1s, thin and slow pages, images without alt text and noindex pages – ranked High, Medium and Low.
Excel, CSV and Google Sheets
Download an Excel workbook with Summary, Issues and Pages sheets, a CSV of every page, an issues-only CSV, or a formula that pulls the data straight into Google Sheets.
Every product in a shop
The fast product list reads a shop's sitemaps and lists every product address, with names and product codes, in a minute or two – even for shops with tens of thousands of products. Then visit any section for prices, SKU, brand, stock and images.
Competitor comparison
Crawl your site and a competitor's with the same settings and see them side by side, with the better result highlighted.
Contact details
Email addresses and phone numbers published on each page, so you can check your details are the same everywhere.
Website crawler or website audit?
The website audit checks up to 250 pages in a few minutes and gives you a ranked report with indexing checks such as orphan pages and sitemap conflicts. The crawler goes much further – up to 10,000 pages – and is built for getting the data out: a spreadsheet you can sort, filter and share. Use the audit for a quick health check and the crawler when you need every page in a spreadsheet.
How long does a crawl take?
The crawler is deliberately polite. It follows robots.txt, including any crawl delay, waits between 2.5 and 5.5 seconds between pages and backs off if a site says it's busy. That works out at about four seconds a page: 100 pages take around seven minutes and 1,000 pages about an hour. Keep the crawl page open while it runs; once it finishes, the results and downloads stay available for 48 hours.
What it can't do
- It stays on the site you enter and doesn't check links to other websites.
- It reads the HTML each page sends, so content that only appears after JavaScript runs may be missed.
- It runs on a free server that sleeps when nobody is using it, so the first visit can take up to a minute to wake it, and very long crawls may be interrupted.
- Only crawl sites you own or have permission to crawl.
Not sure where to start? Run the free SEO check on your home page first, then crawl the whole site. Our guide to broken links and redirects explains how to fix what the crawler finds.
Frequently asked questions
Is the website crawler free?
Yes. No sign-up, no email and no subscription. There's a limit of a few crawls at once so the free server stays available for everyone.
Can it list every product a shop sells?
Usually, yes. Open Product directory options and tick Full product list, fast: it reads the shop's sitemaps, which list every product page, and gives you each product's address, name and code in a minute or two. For prices and stock, start a second crawl with Visit product pages for prices and a section of the shop – that visits each page, so allow about 4 seconds a product.
Is my crawl data kept?
Finished crawl files are deleted automatically after 48 hours. Nothing is shared with anyone else.
Related guides
Technical SEO checklist
A plain-English technical SEO checklist: indexing, sitemaps, redirects, canonicals, speed, mobile and security, with a free tool for each item.
Website redesign SEO
Redesigning or moving your website? A checklist to keep your Google rankings: list every page, redirect old addresses and check it all after launch.
Ecommerce SEO
Ecommerce SEO for small UK online shops: category pages that rank, filters that don't create duplicates, out-of-stock items, and product schema.
Duplicate content
Duplicate content in plain English: is there a Google penalty, what causes it on small sites and shops, and how to fix it with canonicals and redirects.
More free SEO tools
Free SEO check
Full single-page audit: Google readiness, AI search, speed, security and local SEO.
AI SEO checker
Can ChatGPT, Claude, Perplexity and Google AI Overviews read and cite your site?
Website audit
Crawl up to 250 pages for broken links, duplicate titles and orphan pages.
Backlink checker
See which websites link to any site – .co.uk, .com or anything else – and which link to your competitor but not you.
Domain authority checker
Authority scores for up to 100 websites at once, with links and global rank.
Redirect checker
Follow any address hop by hop: 301s, 302s, chains, loops and meta refreshes.
HTTP header checker
Every response header for any address, with what each one means for SEO.
Website down checker
Is a website down for everyone or just you? Live status and response time.
Open Graph checker
See how any page looks when shared on Facebook, LinkedIn, WhatsApp and X.
Hreflang checker
Check language and country tags, return links and x-default on any page.
Heading checker
See any page's H1 to H6 outline and fix missing, double or skipped headings.
Alt text checker
Find images with missing, empty or weak alt text, and linked images with no words.
Canonical checker
Find a page's canonical URL, test it loads, and spot noindex tags and headers.
Website hosting checker
Who hosts a website? The hosting company, IP address, CDN and server location.
SSL checker
Is a site secure? Certificate expiry, https redirects, HSTS and mixed content.
UTM link builder
Tracking links for Google Analytics: source, medium and campaign, ready to copy.
URL slug generator
Turn titles into short, clean, SEO-friendly page addresses.
Case converter
Sentence case, Title Case, UPPER, lower, camelCase and more in one click.
Favicon generator
favicon.ico, Apple and Android icons from letters or your logo, with the HTML.
Hreflang generator
Correct hreflang tags for every language and country version, as HTML or a sitemap.
Email header analyzer
Paste email headers to check SPF, DKIM and DMARC, trace the route and spot fakes.
Website technology checker
What is a site built with? CMS, theme, plugins, analytics, hosting and email.
DNS lookup
A, MX, TXT, NS and other records, plus SPF and DMARC email checks.
DNS propagation checker
Has your DNS change spread? Ten public DNS services and your own name server.
Reverse DNS lookup
The hostname (PTR record) of any IP address, and whether it points back.
Meta tag checker
Every meta tag on a live page, with a Google preview and plain fixes.
MX lookup
Which mail servers receive a domain's email, in priority order, and who provides them.
SPF checker
Check an SPF record, every include and the 10-DNS-lookup limit.
DMARC checker
Look up a DMARC policy tag by tag and see if it stops fake email.
DKIM checker
Find DKIM keys under common selectors or yours, with key size and type.
Domain age checker
How old is a website? Registration and expiry dates for up to 100 domains.
WHOIS lookup
Who a domain is registered with, when it expires and its name servers.
Competitor comparison
Your site against up to two competitors, side by side.
SEO monitoring
A free weekly re-check with a score history graph.
Title tag checker
See your title and description in pixels, exactly as Google cuts them.
robots.txt tester
Paste your rules and test any URL against Googlebot or AI crawlers.
llms.txt generator
Create an llms.txt file that gives AI assistants a map of your site.
Mobile-friendly test
Check your page works on phones – Google's own test closed in 2023.
Broken link checker
Crawl your whole site and find every dead link and broken image.
Schema checker
Validate the structured data on any page or pasted code against Google's rich result rules.
Page speed checker
Core Web Vitals from real Chrome users plus a Lighthouse test.
Sitemap checker
Check your XML sitemap is found, valid and listed in robots.txt.
Security headers checker
HTTPS, HSTS and the headers browsers use to protect visitors.
Local SEO checker
LocalBusiness markup, contact details and map signals for UK businesses.
Meta tag generator
Title, description, canonical, robots and social tags, ready to paste.
Open Graph generator
Control how your link looks on Facebook, LinkedIn, WhatsApp and X.
XML sitemap generator
Find your pages and download a valid sitemap.xml.
robots.txt generator
Block AI training bots while staying visible in AI search.
Local business schema generator
LocalBusiness JSON-LD for Google and AI assistants.
Long-tail keyword generator
Turn one topic into dozens of winnable search phrases.
Blog title generator
Headline ideas measured to fit Google's results.
Word counter
Words, characters, reading time, readability and keyword density.
Keyword difficulty checker
How hard is page one? We grade the pages that rank now.
Plagiarism checker
Find copies of your text across the web – no word limit.
Anchor text checker
Grade every link on a page and generate better anchors.
Local search checker
See Google results as they appear in any UK town.
AI title & description writer
AI reads your page and writes titles and descriptions that fit Google.
Bulk SEO checker
Score up to 20 websites at once and download a CSV.
Redirect map generator
301 rules for .htaccess, Nginx, Cloudflare and more.
SEO ROI calculator
What a better Google position could earn you.
Image compressor
Shrink photos and convert to WebP in your browser – no upload.
QR code generator
QR codes that never expire, as PNG or SVG.
Google review link generator
A one-tap review link, QR code and printable card.
Privacy policy generator
A UK GDPR privacy policy and cookie wording for your site.
Website report
Authority, rank, linking sites, age and hosting for any domain.
SEO score badge
Show your score on your site – it updates itself weekly.
Rank tracker
Track your Google positions for every search, by country, from your own Search Console data.
SEO content editor
Write with a live score, related terms and questions taken from the pages that rank.
Toxic backlink checker
Spot spam links, check your anchor text mix and build a disavow file.
Competitor ad checker
See which ad platforms a competitor uses and open every ad they run.
Uptime monitor
Checks your site every 15 minutes and alerts you the moment it goes down or slows.
Website analytics
Cookie-free visitor stats: live visitors, pages, sources and conversions.
Brand monitoring
Find mentions of your business, unlinked mentions and copied content.
WordPress SEO fixer
A free plugin that fixes missing alt text, descriptions, extra H1s and broken links.
Accessibility checker
Test any page for WCAG problems: alt text, labels, link names, zoom and headings.
Run a free SEO check now
About 50 checks covering Google, AI search, speed, security and local SEO. No sign-up.
Want the fixes done for you? XKey is made by IceWork, a web studio in Blackpool. IceWork builds fast, search-ready websites – a 3-page site including domain and a year's hosting is £350 all in.