Free robots.txt checker
Read any site's robots.txt and test a specific path against it. The verdicts use the real longest-match rule that crawlers use, and cover the eight AI crawlers alongside Googlebot and Bingbot.
Longest match wins, not first match
The rule that catches people is precedence. Crawlers do not read robots.txt top to bottom and stop at the first match — they find every rule whose path prefix matches the URL and apply the longest one. Given Disallow: /blog/ and Allow: /blog/public/, the URL /blog/public/post is allowed, because the Allow rule is longer and so more specific. Reordering the file changes nothing.
This tool implements that resolution rather than a simple prefix scan, which is why the verdict for a specific path can differ from what the file looks like it says at a glance.
Blocking is not de-indexing
Disallow prevents crawling. It does not prevent indexing. Google can and does list a URL it was never allowed to fetch, when enough other pages link to it — you get the bare URL in results with no title or description, which is worse than either outcome you wanted.
Worse still, blocking a page in robots.txt makes a noindex tag on that page unreadable. The crawler is not permitted to fetch the page, so it never sees the instruction to drop it. To remove a page from the index you must allow crawling and use noindex. The indexability checker tests both together and reports which signal is actually deciding.
The AI crawler question
Alongside Googlebot and Bingbot, this tool tests eight AI crawlers individually: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, Google-Extended and Bytespider. Each has a different job — some collect training data, some fetch pages live to answer a question and cite the source.
Blocking them is a legitimate decision, and it is a different decision from SEO: none of it affects Google Search rankings. What it does affect is whether your pages can be quoted and linked in AI answers. Most sites that block these crawlers never chose to — the lines arrived with a plugin default or a copied template. Our GEO checklist covers what to do once you have decided, and the AI visibility check tests the rest of the picture.
What else to check
robots.txt is one of several files and tags that together decide whether a page is reachable. If crawl access is clean and pages are still missing, look at the sitemap, then at indexability. The free website audit runs all of it in one pass and scores the result.
Robots.txt questions
- What is robots.txt?
- robots.txt is a plain text file at the root of a domain that tells crawlers which paths they may request. It is a set of instructions, not an access control — well-behaved crawlers obey it, and anything that ignores it will not be stopped by it.
- Does robots.txt stop a page being indexed?
- No, and this is the most consequential misunderstanding about the file. Disallow stops a page being crawled, not indexed. Google can still list a URL it has never fetched if other pages link to it — showing the URL with no description. To keep a page out of the index you need a noindex tag, which requires the page to be crawlable so the tag can be read.
- How do robots.txt rules match?
- The longest matching rule wins, not the first one. If Disallow: /blog/ and Allow: /blog/public/ both exist, /blog/public/post is allowed because the Allow rule is longer and therefore more specific. Order in the file does not matter, which surprises most people reading one for the first time.
- Should I block AI crawlers in robots.txt?
- Blocking AI crawlers is a choice about how your content is used, not a mistake either way. Blocking GPTBot, ClaudeBot or PerplexityBot keeps your content out of training data and out of AI answers — including the answers that cite and link back to you. It has no effect on Google Search rankings. The point is to make it a decision rather than a default nobody read.
- What does User-agent: * mean?
- User-agent: * in robots.txt is the fallback group, applying to any crawler that does not have a group naming it specifically. A crawler uses the most specific group that matches its name and ignores the rest entirely — so a bot with its own group never reads the * rules, even for directives its own group omits.
- Does Google respect Crawl-delay?
- Google ignores Crawl-delay entirely; crawl rate is managed in Search Console instead. Bing does honor it. So a Crawl-delay line is not wrong, it is just doing less than the person who added it usually expects.
- How do I fix 'Blocked by robots.txt' in Google Search Console?
- This warning means a rule in your robots.txt is telling crawlers not to fetch the URL, so the fix is to remove or narrow the Disallow line that matches it. Edit robots.txt at the root of your domain, confirm the path is no longer blocked with the tester above, then request indexing for the affected page.
- Why is my page indexed though blocked by robots.txt?
- Robots.txt only stops crawling, not indexing, so Google can still list a blocked URL it found through links, usually with no description. To keep a page out of results, allow it to be crawled and add a noindex tag instead — blocking it in robots.txt actually prevents Google from seeing that noindex.
- Where should the robots.txt file be located?
- Robots.txt must sit at the root of the domain, at exactly /robots.txt, because crawlers only look there. A file in a subfolder is ignored, and each subdomain needs its own robots.txt since one does not cover another.
- What is the difference between Disallow and noindex?
- Disallow in robots.txt tells crawlers not to fetch a URL, while noindex tells them not to include a page they have fetched in the index. They are not interchangeable — to remove a page from results you need noindex on a crawlable page, whereas Disallow only hides the content from the crawler.
- Can I have more than one robots.txt file?
- Each host serves a single robots.txt at its root, so you cannot split rules across multiple files on one domain. Separate subdomains and separate protocols are treated as different hosts, so each of those can — and should — have its own file.