Blocked by robots.txt
The crawl-versus-index distinction everyone gets wrong, how to find the offending Disallow rule, and why blocking a page never removes it from Google.
“Blocked by robots.txt” means your site’s robots.txt file contains a rule telling Google not to crawl this URL, and Google obeyed. The page is not indexed because Google was never allowed to fetch it. Whether that is a problem comes down to one question: did you mean to block it?
What robots.txt actually controls
This is the distinction that causes most robots.txt mistakes.Disallow tells crawlers not to fetch a URL. It does not tell them not to index it. If other pages link to a blocked URL, Google can still list it in search results — usually with no description, because it was never allowed to read the page. So blocking is perfect for keeping crawlers out of low-value areas, but it is the wrong way to remove a page from Google. For that you need a noindex tag, which requires the page to stay crawlable.
Find the rule that is blocking the URL
Open your robots.txt (at /robots.txt) and look for a Disallow pattern that matches the path. The robots.txt checker tests a specific URL against your file and shows which rule catches it, which is faster than matching wildcard patterns by eye. Search Console’s URL Inspection does the same for a single URL. The usual culprits are over-broad rules:
A leftover Disallow: / from staging that blocks the entire site. A parameter rule like Disallow: /*? that unintentionally catches legitimate pages. Or a directory block that is wider than intended and sweeps up real content along with the section you meant to hide.
Is the block correct?
Plenty of blocks are right and should stay. Admin areas, internal search results, cart and checkout flows, API endpoints, and infinite faceted-filter combinations all belong behind a Disallow to keep Google from wasting crawl budget on them. The block is wrong when it catches pages you want to rank, blocks the whole site by accident, or hides the CSS and JavaScript Google needs to render your pages.
Never block the resources needed to render
A common self-inflicted wound is blocking /assets/ or script and style folders. Google renders pages the way a browser does, so blocking the CSS and JavaScript it needs means it sees a broken layout, which hurts mobile-friendliness and can hurt rankings. Leave rendering resources crawlable — see JavaScript SEO and rendering for why this matters.
How to unblock a page
Remove or narrow the offending Disallow so it no longer matches the URL, then re-test with the robots.txt checker to confirm the path is now allowed. Check the page is otherwise indexable — no leftover noindex — and request indexing in Search Console so Google re-crawls it. Crawling resumes within days once the block is gone.
The opposite problem
If a URL is blocked in robots.txt but appears in search results anyway, that is a different status: Indexed, though blocked by robots.txt. It is the direct consequence of the crawl-versus-index distinction above, and it is fixed differently — by allowing the crawl so Google can see a noindex, not by tightening the block. For the full list of report statuses, see the Page indexing report reference, or run a full site check.
Frequently asked questions
- What does 'Blocked by robots.txt' mean?
- It means your robots.txt file has a Disallow rule matching the URL, so Google did not crawl it and therefore did not index it. It is Google obeying an instruction on your site. Whether it is a problem depends entirely on whether you intended to block that URL — many blocks are correct and deliberate.
- Does robots.txt stop a page being indexed?
- Not reliably. robots.txt stops Google crawling a page, but a blocked page can still be indexed if other pages link to it — Google lists the URL without a description because it could not read the content. To actually keep a page out of the index you need a noindex tag, and the page has to stay crawlable for Google to see that tag.
- How do I find which robots.txt rule is blocking my page?
- Open your robots.txt at /robots.txt and look for a Disallow pattern that matches the URL path, or use a robots.txt checker or Search Console's URL Inspection to test the specific URL and see exactly which rule catches it. Common over-broad culprits are a leftover 'Disallow: /' from staging, a parameter rule like 'Disallow: /*?', or a directory block wider than intended.
- How do I fix 'Blocked by robots.txt' when I want the page indexed?
- Remove or narrow the Disallow rule so it no longer matches the URL, then re-test that the path is allowed. Confirm the page has no leftover noindex tag and is otherwise indexable, then request indexing in Search Console. Google resumes crawling within days once the block is removed.
- Should I block a page in robots.txt to keep it out of Google?
- No. Because blocking stops the crawl, Google can never see a noindex tag on that page, and the URL can still be indexed from external links. To remove a page from Google, allow crawling and add a noindex tag. Only after the page has dropped out of the index is it safe to also block it in robots.txt to save crawl budget.
- Is it bad to block CSS and JavaScript in robots.txt?
- Yes. Google renders pages like a browser, so blocking the CSS and JavaScript needed to render a page means Google sees a broken version of it, which harms mobile-friendliness and can hurt rankings. Keep rendering resources crawlable and only block genuinely low-value areas like admin, cart, and internal search.