What does a DNS error in Search Console mean?

Googlebot could not resolve your hostname to an IP address, so the request never reached your server and your server logs show nothing. Common causes are a slow or unreachable nameserver, a recently changed record still propagating, an expired domain, DNSSEC misconfiguration, or a missing record for one hostname while the other works — typically www existing without the apex, or the reverse.

Should I worry about a few crawl errors?

No. A handful of scattered connectivity errors with no pattern is normal — servers have brief failures and Googlebot retries. What warrants action is a spike with an identifiable start date, sustained errors across a whole template, errors coinciding with a traffic drop, or any error affecting robots.txt.

Why is a robots.txt error more serious than a page error?

Because Google's handling depends on the status code. A 404 on robots.txt means no restrictions and Google crawls freely, but a 5xx or timeout is treated as a temporary instruction to stop crawling the entire site — Google cannot know whether the unreachable file forbids what it is about to do, so it does nothing. Sustained unavailability can lead to pages dropping out of the index.

Why does Search Console show server errors when my site seems fine?

Because some 5xx responses are specific to Googlebot and invisible to user-based monitoring. Rate limiting or DDoS protection can treat a crawl burst as an attack, a WAF rule can match the crawler's user agent, and the crawl itself can exhaust resources. If Search Console shows sustained 5xx while your uptime monitoring is green, this is almost always why.

How do I verify a crawler is really Googlebot?

Do a reverse DNS lookup on the requesting IP — it must resolve to a googlebot.com or google.com hostname — then a forward lookup on that hostname, which must resolve back to the same IP. Both directions are required, because either one alone can be spoofed. Google also publishes its crawler IP ranges as JSON. Never allowlist based on the user agent string, which is trivially forged.

Do crawl errors reduce how much Google crawls my site?

Yes. Googlebot adapts its crawl rate to how your server responds, so consistent errors or slow responses reduce crawling — and the suppression persists for a while after you fix the underlying problem. That is why a period of instability can leave new pages undiscovered for weeks after the site itself is healthy again.

DNS and server connectivity errors

Transport failures, where nothing about your content is even being considered — including the one that quietly tells Google to stop crawling.

By Max8 min read

DNS errors, timeouts and 5xx responses are a different species of problem from the rest of SEO. Every other issue means Google fetched your page and disagreed with something about it. These mean Google never got the page — so nothing about your content, markup or links is being considered at all.

The `robots.txt` case, first, because it is the worst

Google’s handling of robots.txt depends on the status code it receives, and the three cases behave very differently:

200 with rules — the rules are followed. 404 — Google assumes there are no restrictions and crawls freely. This is the safe absence.

5xx or a timeout — Google treats the site as temporarily disallowed in its entirety. The reasoning is conservative and correct from Google’s side: it cannot know whether the unreachable file contains a rule forbidding what it is about to do, so it does nothing. Extended unavailability can lead to pages dropping out of the index.

Which produces a genuinely counter-intuitive priority. If your robots.txt is intermittently 5xx-ing — often because it is generated dynamically by the application rather than served as a static file — that is more urgent than any individual page error, and it is invisible unless you look for it. Serve it statically if you can.

DNS errors

A DNS error means Googlebot could not resolve your hostname to an IP address. The request never reached your server, so your server logs show nothing, which is why these are so often dismissed as false alarms.

Common causes: a nameserver that is slow or unreachable from Google’s resolvers; a recently changed record still propagating; a domain that expired or auto-renewed late; DNSSEC misconfiguration, which causes validating resolvers to refuse the answer while non-validating ones succeed; and a missing record for one hostname — typically the www variant existing while the apex does not, or the reverse.

To diagnose: resolve the name from several networks and locations, not just yours, since your ISP’s resolver may have a cached answer that hides the problem. Check both the apex and www. Verify your registration expiry date. And use Search Console’s URL Inspection live test, which resolves from Google’s own infrastructure and is therefore the only test that answers the actual question.

Server errors (5xx)

Your server was reached and failed to produce a response. In Search Console these appear as “Server error (5xx)” in the Page indexing report and in the response-code breakdown of Crawl stats.

The distinction worth making is between errors that affect everyone and errors that only affect Googlebot. Application crashes, database connection exhaustion and out-of-memory failures affect all visitors, and you will hear about them. But some 5xx patterns are Googlebot-specific: crawl-rate limiting or DDoS protection that treats a burst of requests as an attack, WAF rules matching the user agent, and resource exhaustion caused by the crawl itself.

The Googlebot-specific ones are far more damaging, because nobody reports them and monitoring based on user traffic will never see them. If Search Console shows sustained 5xx while your uptime monitoring is green, this is the explanation.

Note also that Googlebot adapts: consistent 5xx responses reduce your crawl rate, so a period of instability suppresses crawling for a while after the underlying problem is fixed. See crawl budget and faceted navigation.

Timeouts and refused connections

Timeout means the connection opened but no response arrived in time. Almost always genuine slowness rather than a fault: a page whose queries take twenty seconds, a blocking third-party API call, or a server so loaded that requests queue. Reducing time to first byte addresses the same underlying causes at the point before they become errors.

Connection refused or connection reset means nothing accepted the connection, or something closed it deliberately. Usually the web server is down, a firewall is dropping the traffic, or bot protection is blocking Googlebot’s address range outright.

Blocked due to unauthorised request (401) or access forbidden (403) are adjacent: your server answered, but refused. Frequently a security plugin, an IP allowlist, or basic authentication left on a section after testing.

Verifying it is really Googlebot

Before whitelisting anything, this matters — the user agent string is trivially forged, and a lot of unwanted crawling claims to be Googlebot.

The correct check is a reverse DNS lookup on the requesting IP, which must resolve to a googlebot.com or google.com hostname, followed by a forward lookup on that hostname resolving back to the same IP. Both directions are required; one alone is spoofable. Google also publishes its crawler IP ranges as JSON if you prefer matching against those.

Never whitelist by user agent string alone. That is an open door with a sign on it.

Deciding whether to act

Search Console reports errors from a rolling window and does not distinguish a one-second blip from an outage, so read the shape rather than the count.

Ignore a handful of scattered errors with no pattern. Googlebot retries; this is normal.

Investigate a spike with a start date — that points at a change you made, and the date is the most useful diagnostic you will get.

Act immediately on any error affecting robots.txt, on sustained errors across a whole template, and on errors that coincide with a traffic drop.

Check the time of day. Errors clustered at specific hours suggest a scheduled job, a backup window, or crawl bursts colliding with peak traffic. Server logs answer this in minutes; Search Console cannot.

Ruling it out

Because these errors are intermittent by nature, a single successful test proves nothing — but a single failing test proves a great deal. Our free site check fetches your pages live and reports the status and response time it actually received, which is a quick way to confirm the site is reachable right now and how close to a timeout it is running. The robots.txt checker is the more important one here: it confirms the file is served with a 200 rather than erroring, which is the highest-consequence check on this page. If everything responds cleanly and pages still are not indexed, the cause is elsewhere — Search Console page indexing errors maps the alternatives.

Frequently asked questions

What does a DNS error in Search Console mean?: Googlebot could not resolve your hostname to an IP address, so the request never reached your server and your server logs show nothing. Common causes are a slow or unreachable nameserver, a recently changed record still propagating, an expired domain, DNSSEC misconfiguration, or a missing record for one hostname while the other works — typically www existing without the apex, or the reverse.
Should I worry about a few crawl errors?: No. A handful of scattered connectivity errors with no pattern is normal — servers have brief failures and Googlebot retries. What warrants action is a spike with an identifiable start date, sustained errors across a whole template, errors coinciding with a traffic drop, or any error affecting robots.txt.
Why is a robots.txt error more serious than a page error?: Because Google's handling depends on the status code. A 404 on robots.txt means no restrictions and Google crawls freely, but a 5xx or timeout is treated as a temporary instruction to stop crawling the entire site — Google cannot know whether the unreachable file forbids what it is about to do, so it does nothing. Sustained unavailability can lead to pages dropping out of the index.
Why does Search Console show server errors when my site seems fine?: Because some 5xx responses are specific to Googlebot and invisible to user-based monitoring. Rate limiting or DDoS protection can treat a crawl burst as an attack, a WAF rule can match the crawler's user agent, and the crawl itself can exhaust resources. If Search Console shows sustained 5xx while your uptime monitoring is green, this is almost always why.
How do I verify a crawler is really Googlebot?: Do a reverse DNS lookup on the requesting IP — it must resolve to a googlebot.com or google.com hostname — then a forward lookup on that hostname, which must resolve back to the same IP. Both directions are required, because either one alone can be spoofed. Google also publishes its crawler IP ranges as JSON. Never allowlist based on the user agent string, which is trivially forged.
Do crawl errors reduce how much Google crawls my site?: Yes. Googlebot adapts its crawl rate to how your server responds, so consistent errors or slow responses reduce crawling — and the suppression persists for a while after you fix the underlying problem. That is why a period of instability can leave new pages undiscovered for weeks after the site itself is healthy again.