What is an HTTP header?
Request versus response headers, the status code, and the SEO-critical ones — X-Robots-Tag, Link, Content-Type, Vary — plus why a header can deindex a page invisibly.
An HTTP header is a line of metadata a browser and a server exchange before the page content itself. Every request your browser sends and every response a server returns carries headers describing what is being sent, how it should be handled, and what to do with it. Most are invisible plumbing — but a few response headers carry instructions that decide whether Google indexes your page at all, which makes them some of the most important, and most easily overlooked, technical SEO signals.
Request headers versus response headers
Headers travel in both directions. Request headers are what your browser sends to the server — things like User-Agent (what browser or crawler you are) and Accept (what formats you can handle). Response headers are what the server sends back alongside the page, and these are the ones that matter for SEO, because they can carry crawling and indexing instructions.
The status code comes first
Before anything else, the response includes a status code, and it is the first thing a crawler reads: 200 (OK), 301 and 302 (redirects), 404 (not found), 410 (gone), and 503 (temporarily unavailable). The status code tells Google whether the page is worth indexing, whether it moved, or whether to come back later — so a wrong status (a broken page returning 200, say) quietly misleads the crawler.
The SEO-critical response headers
Beyond the status code, a handful of response headers directly affect search. X-Robots-Tag carries the same directives as the robots meta tag — noindex, nofollow — but at the header level, which means it can control indexing for files that have no HTML to put a meta tag in, like PDFs and images. The Link header can declare a canonical URL or hreflang for those same non-HTML files. Content-Type tells Google what kind of file it is fetching. Cache-Control governs how long responses are cached, and Vary (especially Vary: User-Agent) signals that you serve different content to different clients, which matters if you render a separate mobile version.
Why headers are a stealthy source of bugs
The danger with header-level directives is that they are invisible where you normally look. A noindex in the HTML shows up in the page source; a noindex in the HTTP header does not. A server misconfiguration, a CDN rule, or a leftover setting from a staging environment can attach X-Robots-Tag: noindex to whole sections of a site, and the pages vanish from Google with nothing wrong on the page itself. The same goes for a wrong Content-Type, or a canonical that is set in a header and contradicts the one in the HTML. If pages are dropping out of the index and the HTML looks fine, the headers are the first place to look.
How to check yours
Our HTTP header checker shows the raw response headers a server returns, so you can catch a stray X-Robots-Tag, a wrong status code, or a missing canonical Link at a glance. To see how those headers combine with your meta tags and robots.txt into a single indexing verdict, run the indexability checker, which tells you whether a page can actually be indexed once every signal is accounted for.
Frequently asked questions
- What is an HTTP header?
- An HTTP header is a line of metadata that a browser and server exchange before the page content itself. Request headers describe what the browser is asking for; response headers describe what the server is sending back. A few response headers carry crawling and indexing directives that directly affect SEO.
- How do HTTP headers affect SEO?
- Some response headers are directives, not just information. The X-Robots-Tag header can noindex or nofollow a page — including non-HTML files like PDFs — the Link header can set a canonical or hreflang, and the status code tells crawlers whether a page is OK, redirected, or gone. A wrong header can deindex or misrepresent a page even when the HTML is fine.
- What is the X-Robots-Tag header?
- X-Robots-Tag is a response header that carries the same directives as the robots meta tag — such as noindex and nofollow — but at the HTTP level. Because it works on the response rather than the HTML, it can control indexing for files that have no HTML to hold a meta tag, like PDFs and images. It's also invisible in page source, making an accidental noindex here hard to spot.
- Why did my pages get deindexed when the HTML looks fine?
- A common cause is an X-Robots-Tag: noindex header applied by a server misconfiguration, a CDN rule, or a leftover staging setting. Because the directive lives in the HTTP header rather than the HTML, it never shows up in 'view source', so pages vanish from Google with nothing visibly wrong. Check the raw response headers to find it.
- Which HTTP response headers matter most for SEO?
- The status code (200, 301, 404, 410, 503), the X-Robots-Tag (indexing directives), the Link header (canonical and hreflang for non-HTML files), Content-Type (the file type), Cache-Control (caching), and Vary — especially Vary: User-Agent if you serve different content to mobile clients.
- How do I check a page's HTTP headers?
- Use an HTTP header checker to see the raw response headers a server returns for a URL, which reveals the status code and any X-Robots-Tag, canonical Link, or Content-Type issues. To understand how those headers combine with meta tags and robots.txt into a single indexing decision, follow up with an indexability checker.