What is robots.txt?
What the file does, the crawling-versus-indexing distinction that trips everyone up, and the leftover staging block that quietly deindexes a whole site.
robots.txt is a plain text file at the root of your site that tells search engine crawlers which parts of it they may and may not fetch. It is the first file a well-behaved crawler requests before it looks at anything else, and it is the oldest and bluntest of the tools you have for steering how bots move through a site.
Where it lives and what it looks like
It must sit at the very root of the domain — https://example.com/robots.txt — because that is the only place crawlers look for it. A rule in a subfolder is ignored. The syntax is a list of groups, each naming a crawler and the paths it may not take:
User-agent: * — the group applies to all crawlers. Disallow: /admin/ — do not fetch anything under /admin/. Allow: /admin/help — an exception that carves one path back out of a broader disallow. Sitemap: https://example.com/sitemap.xml — where to find the sitemap. An empty Disallow: means everything is allowed; Disallow: / blocks the entire site.
The distinction that matters most
Almost every serious robots.txt mistake comes from the belief that blocking a URL removes it from Google. It does not. robots.txt stops the crawler fetching the page’s content. If other pages — yours or anyone’s — link to that URL, Google can still index the address itself, showing it in results with no description because it was never allowed to read the page. That is the status indexed, though blocked by robots.txt, and it is the direct consequence of using the wrong tool for the job.
The rule of thumb: use robots.txt to save crawl effort on parts of a site that produce no useful search results — internal search, faceted filters, admin areas. Use a noindex tag to keep a specific page out of the index. And never use both on the same page, because a blocked crawler can never read the noindex that would have removed it.
What robots.txt cannot do
It is not a security control. The file is public — anyone can read yoursite.com/robots.txt — so disallowing /secret-admin/ just publishes a list of the paths you would rather people did not find. Protect private areas with authentication, not a Disallow line.
It cannot remove an already-indexed page. Blocking a URL that is already in the index freezes it there — Google can no longer crawl it to see a noindex or a removal, so it lingers. To remove a page, allow the crawl and add noindex, then block it later if you wish once it has dropped out.
It does not guarantee obedience. Reputable search crawlers respect it. Many scrapers and spam bots ignore it entirely. It is a convention for the crawlers that choose to honor it, not a fence.
The mistakes that quietly cost traffic
The leftover staging block. A site launches with User-agent: * / Disallow: / carried over from staging, and every page silently drops out of Google. It is the single most common catastrophic SEO error, and the fix is one line — but only if someone notices.
Blocking CSS and JavaScript. Older setups block /assets/ or /scripts/ to save crawl budget. Google renders pages like a browser, so blocking the resources it needs to render means it sees a broken layout and may misjudge the page. Let it fetch what it needs to render.
Trying to deindex with Disallow. Covered above, but it is worth repeating because it is so common: if a page is already indexed and you want it gone, blocking it in robots.txt does the opposite of what you intend.
How to check it
Our robots.txt checker fetches your file, shows the rules as a crawler reads them, and tells you whether a specific URL is allowed or blocked — which is how you catch an over-broad Disallow before it costs you pages. The indexability checker then confirms the fuller picture for a URL: whether it is blocked, carries noindex, or canonicalizes elsewhere. If Search Console is already reporting pages as blocked by robots.txt when they should be ranking, that guide walks through finding the offending rule.
When you want the whole site checked at once — including whether your robots.txt is accidentally hiding pages you meant to rank — run a full site check.
Frequently asked questions
- What is robots.txt?
- It is a plain text file at the root of a website — such as example.com/robots.txt — that tells search engine crawlers which paths they may and may not fetch. Crawlers request it before anything else. It uses simple User-agent, Disallow and Allow rules, and can point to the sitemap, but it only steers crawlers that choose to honor it.
- Does robots.txt stop a page being indexed?
- No, and this is the most common misunderstanding. robots.txt controls crawling, not indexing. A disallowed page can still be indexed if other pages link to it — Google shows the URL with no description because it was never allowed to read the content. To keep a page out of the index, use a noindex tag on a crawlable page, not a robots.txt block.
- Where does the robots.txt file go?
- At the root of the domain, reachable at example.com/robots.txt. That is the only location crawlers check, so a robots.txt placed in a subfolder is ignored. Each subdomain needs its own file, since blog.example.com and example.com are treated as separate hosts.
- Can I use robots.txt to hide private pages?
- No. The file is publicly readable, so listing a path under Disallow effectively advertises it. And because blocking only prevents crawling, a private URL that gets linked can still surface in results. Protect anything genuinely private with authentication or server-side access controls, not a robots.txt rule.
- Why did my whole site disappear from Google after launch?
- The most likely cause is a Disallow: / rule carried over from the staging site, which blocks every URL. Staging environments are usually configured to block all crawlers, and if that robots.txt ships to production it silently removes the site from search. Check example.com/robots.txt for a site-wide Disallow and remove it if the site should be public.
- Should I block CSS and JavaScript in robots.txt?
- No. Google renders pages the way a browser does, so it needs to fetch your CSS and JavaScript to see the page correctly. Blocking those resources means Google may render a broken layout and misjudge the page's content or mobile-friendliness. Allow crawlers to fetch the assets required to render your pages.