A law firm in Manchester had an old landing page for a service it no longer offers. The brief to the agency: get the page out of Google. The agency put a Disallow in the robots.txt, reported "done" and left. Three weeks later the URL was still in the results, now without a description, plus a fresh message in Search Console: "Indexed, though blocked by robots.txt".
That is the normal case, not the exception. robots.txt controls crawling, noindex controls indexing. robots.txt is a doorman: it decides whether a bot may fetch a URL, and says nothing about whether that URL may appear in the results. noindex is a sign inside the page: come in, but do not put this in the directory. If the doorman lets nobody in, nobody reads the sign.
This piece gives you a decision table for the five most common goals, both ways to set a noindex, and every trap with the symptom that gives it away.
Which tool achieves which goal
| What you want | The right tool | The usual wrong grab |
|---|---|---|
| A page should not appear in Google | noindex, crawling allowed | Disallow in robots.txt |
| Save crawl capacity on thousands of filter URLs | Disallow in robots.txt | noindex on every single URL |
| Hide content from humans | Password or login | Disallow in robots.txt |
| Pull a page out of the results right now | Removals tool, then noindex | Disallow alone, then waiting |
| Take a PDF or image out of the index | X-Robots-Tag: noindex header | a meta tag that cannot exist there |
Row one is the case above. Row two is the only one where robots.txt really fits: a shop with 400 products can easily produce 80,000 URLs through filter combinations. You do not want noindex there, because Google would have to fetch every one to see it. Here you block the pattern.
Row three is the most embarrassing mix-up, because it works in the wrong direction (more below). Row four needs both, because the fast tool only lasts a while. Row five: a PDF has no <head> for a meta tag.
robots.txt: where it lives, what it does, what Google ignores
The file has to sit at exactly /robots.txt in the root, and it applies only to the host, protocol and port it is served from. That rule trips up subdomains: https://example.com/robots.txt does not apply to https://shop.example.com/. The shop needs its own file.
Google evaluates exactly four fields:
User-agent: *
Disallow: /cart
Disallow: /*?filter=
Disallow: /search
Allow: /search/popular-queries
Sitemap: https://example.com/sitemap.xml
User-agent names the bot, * covers everyone without their own block. Disallow blocks a path, Allow cuts an exception out of it, Sitemap points to your sitemap file. Google understands two wildcards: * for any number of characters and $ for the end of the URL, which is why Disallow: /*.pdf$ blocks all PDFs. A trailing star does nothing.
Where rules conflict, the more specific one wins, measured by the length of the rule path. Above, /search/popular-queries is longer than /search, so that one page is allowed and the rest is blocked. At equal length the less restrictive rule wins, so Allow. Two details: the file is publicly readable, and Google honours only the first 500 KiB.
And now the list this section is really about, because plenty of outdated half-knowledge circulates. Google does not evaluate these directives:
Crawl-delay: Bing and Yandex honour it, Googlebot does not. Google switched off the crawl rate tool in Search Console in early 2024. If Googlebot overloads your server, the documented emergency brake is a temporary503or429.Noindex: /pathandNofollow: /path: Google retired the code for these on 1 September 2019. They worked unofficially before that, which is why the lines still show up in tutorials. They do nothing.Host: a Yandex invention for picking the main domain. Use canonical tags instead.
Found one of these lines in your file? Delete it. It does nothing.
Setting noindex: as a meta tag and as an HTTP header
Route one belongs in the <head> of the page:
<meta name="robots" content="noindex">
That addresses every search engine that supports the rule. To exclude Google while staying in Bing, use <meta name="googlebot" content="noindex">. The tag has to be in the <head> and the page fetchable. A noindex that only arrives via JavaScript is a gamble.
Route two is the HTTP header X-Robots-Tag: noindex. It works on any file, including ones with no <head>, and hardly anyone knows it. For Apache in .htaccess, here for all PDFs:
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex"
</FilesMatch>
And for nginx:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex";
}
If you want the old price list PDF out of Google, this is the route. A Disallow: /*.pdf$ only makes the file unreadable, not invisible. Check the header with curl -I https://example.com/price-list.pdf.
"Indexed, though blocked by robots.txt" and the way out
This message is not a Google bug, it is the logical consequence of a block. Google learns the URL from a link, from a sitemap or from its own history, tries to fetch it, and robots.txt says no. Google has an address but no content. And because a linked URL might matter to somebody, Google keeps it in the index. Without a snippet.
The way out has an order you cannot shortcut:
- Remove the block from robots.txt. Yes, really. It feels wrong; it is the only way.
- Set
noindexon the page, via meta tag or header. - Fetch the URL in URL inspection and request indexing. That speeds up the visit, it does not guarantee it.
- Wait until the status flips to
Excluded by 'noindex' tag. Google names no deadline; in practice it is days to weeks. - Only then put the block back, if you genuinely need it. Usually you do not.
If the report holds other status messages you cannot read, Page not indexed sorts them out. Only some are actually a problem.
The traps the usual guides miss
Blocking CSS and JavaScript. Ten years ago Disallow: /wp-content/ was common advice. Today Google renders pages like a browser, and a page without its stylesheet is a wall of text to it. Symptom: in URL inspection the rendered screenshot shows an unstyled page, or the body copy is missing entirely.
Using robots.txt as a hiding place. Anyone can open your file. Whoever writes Disallow: /backup-2024/ or Disallow: /internal-pricing/ in there has published a map of their secrets, and nothing is blocked: if you know the URL, you fetch it. Symptom: none, which is the dangerous part. What helps is a password. Behind one, Google can neither crawl nor index, and no text file announces it.
The noindex that escapes from staging. A noindex belongs on the staging domain. It goes wrong when the configuration travels with the deploy. Symptom: traffic collapses one to three weeks after the relaunch, and the count of pages marked Excluded by 'noindex' tag climbs. Make it a fixed deploy check: one curl -I on the homepage, one look at the source.
A robots.txt that returns a server error. A missing file with status 404 is harmless: Google assumes there are no rules and crawls everything. A 5xx error is the opposite: for the first twelve hours Google stops crawling the site, then falls back to the last cached copy for up to 30 days, and only after that treats the file as missing. A server throwing a 500 on /robots.txt is therefore worse than no robots.txt at all. Useful in reverse: a deliberate 503 is the documented emergency brake for a maintenance window.
Confusing nofollow with noindex. nofollow says: do not follow the links on this page. It says nothing about indexing it. A <meta name="robots" content="nofollow"> on its own leaves the page happily in the results. If you want both, write content="noindex, nofollow", though the nofollow is usually unnecessary.
When something wrong has to go now, and how you check what applies
If something public should not be public, a wrong price, a name, an internal draft, noindex is too slow: it only acts at the next crawl. For that, Search Console has the Removals report, tab "Temporary removals", button "New request". The URL leaves the results within hours.
The important part is in the name: temporary. The removal lasts about six months. After that the URL returns if the page is reachable and carries no noindex. The right sequence: remove it first so it is gone now, then in the same hour set noindex, put a password in front of it, or delete the page with a 404 or 410. Do only the first step and the problem is back in six months, with nobody left who remembers why.
What actually applies is visible in two places. Under Settings → robots.txt sits the robots.txt report: the hosts of your property, the status of the last fetch ("Fetched" or "Not fetched"), the date, and the lines Google could not parse. It covers the last 30 days, and you can request a recrawl there. The second is URL inspection in the search bar at the top: for a single URL it tells you whether crawling and indexing are allowed and which rule applies. How to set the property up is in Set up Search Console.
AI crawlers: same file, different calculation
For GPTBot, ClaudeBot, PerplexityBot and relatives the same robots.txt applies with the same four fields. The calculation differs, because this is not about search results but about training data and citations in answers. A noindex does not help there: it is an instruction to search engines, not an opt-out from AI systems. Which bot does which job, and which block costs visibility, is in Control AI crawlers. To show those systems where your best content sits, an llms.txt is the half hour of work.
Where to start
- Open
https://your-domain.com/robots.txtin a browser and read it line by line. At everyDisallowask: should this page merely not be crawled, or not be in Google at all? In the second case the line is wrong. - Delete everything Google does not evaluate:
Crawl-delay,Noindex,Nofollow,Host. - Check for paths nobody should know about. If so: out of the file, behind a password.
- Open the page indexing report and look for "Indexed, though blocked by robots.txt". Every URL there gets the five steps from above.
- If you run subdomains or a separate shop, open their robots.txt files individually.
An afternoon of work, and it clears more visibility problems than most optimisation projects. If you have no systematic way to hunt for things like this, the work starts one step earlier: at Find SEO problems.
Frequently asked questions
What is the difference between robots.txt and noindex?
robots.txt controls crawling: it tells a bot which URLs it may fetch. noindex controls indexing: it tells the search engine this page should not appear in the results. A page blocked by robots.txt can still end up in the index, a page with noindex cannot. The two tools exclude each other rather than complement each other, which is what most guides get wrong.
How do I exclude a single page from Google?
You set noindex on the page, as a meta tag in the head or as the X-Robots-Tag HTTP header, and make sure robots.txt does not block that same page. Google has to be able to fetch it to see the rule. Expect days to weeks before the page drops out; Google publishes no binding timeframe.
Why is my page in Google despite robots.txt?
Because robots.txt only prevents fetching, not listing. Google knows the URL from links, from your sitemap or from earlier crawls, and may no longer visit it. So it stays in the index, without content. The way out runs through crawling: lift the block, set noindex, let Google visit.
Do I even need a robots.txt file?
Not strictly. If the file is missing and the server returns a 404, Google assumes there are no rules and keeps crawling. Small sites with a few dozen pages get by without one.
It earns its place when you have many URLs that only cost crawlers time: filter parameters, internal search results, cart, account. What matters is that an existing file is served cleanly, because a server error on /robots.txt is worse than no file.
How do I block a PDF file from Google?
With the X-Robots-Tag: noindex HTTP header, sent by your web server for that file. A PDF has no head section for a meta tag, so the header is the only route. A Disallow does not help: the file stays in the results, Google simply cannot read it.
What does "Indexed, though blocked by robots.txt" mean?
Google has taken the URL into the index but may not fetch it because of your robots.txt: it knows the address, not the content, and shows the page without a description. Whether that is a problem depends on your intent. If you wanted the page out of Google, lift the block and set noindex. If you only wanted to save crawl capacity, ignore the message.
Can I block CSS and JavaScript in robots.txt?
Better not. Google renders pages like a browser and needs stylesheets and scripts to do it. Blocked, they leave Google looking at an unstyled or half empty page, and where content is built by JavaScript the entire text can vanish. The advice to block folders like wp-content predates rendering.
How quickly does a robots.txt change take effect?
Google generally caches the file for up to 24 hours, longer if there are server problems. You can speed that up in the robots.txt report. The effect on the results is another matter: a new block removes nothing from the index, and a lifted block does not produce an immediate visit.
How do I remove a page from Google immediately?
Through the Removals report in Search Console, tab "Temporary removals", button "New request". The URL disappears within hours. The removal lasts about six months, after which the URL comes back. Use the time for the permanent fix: noindex, a password, or deleting the page with a 404 or 410.