A car parts retailer with 1,630 pages indexed, selling every day, had no robots.txt file at all. Not a permissive one. Not an empty one. The URL returned a 404.
That is not a crisis, and I want to be clear about that before anything else, because “you have no robots.txt” is the kind of finding that gets written in red and sold as urgent. No robots.txt means no restrictions. Google crawls the site. Everything works.
What it told me was something else. On a catalogue of that size, somebody should have had a reason to open that file at some point, to keep the crawler out of the cart, or the search results pages, or the account area. Nobody ever had. The absence was not the problem. It was the tell that the technical layer had never had anyone’s attention, and the rest of that audit bore it out: 1,277 orphan pages, 1,301 URLs with no H1, 1,344 pages under 500 words.
That is how I read robots.txt now. Not as a checklist item, but as the cheapest available signal of whether anyone has ever been in there.
How normal is a missing one
More normal than you would guess. HTTP Archive’s 2025 Web Almanac found that 13% of robots.txt requests return a 404, and of the files that do exist, 97.6% are under 100 bytes and 1.7% are completely empty.
Read that second figure again. Almost every robots.txt on the web is three or four lines long. The mental image most people have, a carefully maintained file full of considered directives, describes almost nobody. The realistic version is two lines and a sitemap reference, and for most sites that is genuinely all it should be.
So the honest framing is: an absent or tiny robots.txt is not evidence of neglect on a small site. On a large database-driven one, it is.
The mistakes that actually cost something
In fifteen years the damage I have seen from this file has come from four places, and none of them is the file being missing.
Blocking something the site depends on. A Disallow on a directory that also holds the CSS or the JavaScript. The pages still get crawled; they just get crawled looking broken, because Googlebot cannot fetch what it needs to render them. This is the single most expensive robots.txt mistake and it is invisible unless you check, because the site looks perfect to you.
Using it to remove a page from the index. It does not do that. Blocking a URL stops Google crawling it, which means Google cannot see the noindex you also put there, so the page can stay indexed indefinitely, usually with no snippet. Worth knowing that this is a different situation from crawled, currently not indexed, where Google fetched the page, read it, and decided against it. Here Google never got to read it at all and is indexing a URL it has never seen the contents of. If you want something out, let it be crawled and serve noindex. Blocking is how you make a page permanently un-removable.
A staging block that shipped. Disallow: / is correct on a staging site and catastrophic on the live one, and it moves across during a launch more often than anybody admits. It is a thirty-second check on the day and nobody does it on the day, because on launch day everyone is looking at the front end.
Assuming it is a security control. It is a public file listing the directories you would prefer people did not look at. Anyone can read yours.
Google’s own introduction to robots.txt is explicit on that last point and on the second one: the file manages crawler traffic, it is not a mechanism for keeping a page out of the index. Worth reading once properly; it is short, and most of the mistakes above come from assuming it does something it does not.
The new question: AI crawlers
This is the part of the file that has actually changed, and it is the one clients now ask about.
The same Almanac data shows how many robots.txt files name the AI crawlers: gptbot in 4.2%, claudebot in 3.4%, petalbot in 4.4% of files on mobile. Around one site in twenty-five has an opinion. The other twenty-four have not thought about it.
My position, and it is a position rather than a finding: for most of my clients, blocking them is the wrong call. A local clinic, a trades business, an ecommerce store. These are businesses that want to be found and quoted, and the AI surfaces are increasingly where being found happens. Blocking the crawler removes you from the answer without removing the question.
The businesses where it is worth a real conversation are the ones whose product is the content: publishers, course providers, anyone whose archive is the asset being sold. That is a commercial decision about licensing, not an SEO one, and I say so rather than pretending I can optimise it either way.
What I would not do is add the directives because a plugin offered to. Four lines you did not decide on is how the staging block gets shipped.
What I actually check, in about ninety seconds
- Does it return 200? A 404 is fine. A 500 is not. Google may treat a server error on robots.txt as a reason to slow or stop crawling the site entirely, which is the one way this file can genuinely take you off the map.
- Is anything blocked that renders the page? Fetch the CSS and JS paths directly and confirm they are reachable.
- Is
Disallow: /in there? Yes, really. Check. - Is the sitemap listed, and does that URL work? One line, frequently either absent or pointing at a sitemap that moved two rebuilds ago.
- Does anything blocked here also carry a
noindex? If so the two are fighting and the block is winning, which is the opposite of what whoever wrote it intended.
The short version
- A missing robots.txt is not a fault. 13% of sites 404 it and most of them are fine.
- On a large catalogue, its absence is a tell that nobody has ever been in the technical layer. Go and look at what else is true.
- Almost every real file is tiny. 97.6% are under 100 bytes. Yours probably should be too.
- Blocking rendering resources is the expensive mistake, and it is invisible from the front end.
- Blocking is not removing. A blocked page can stay indexed forever because Google never sees your noindex.
- A 500 on this file is worse than a 404.
- AI crawler directives are a commercial decision, not an SEO one. One site in twenty-five has made it deliberately.
If a site should be ranking and it isn’t, that’s the work I do. Technical SEO covers the crawling and indexing layer. If you’re not sure which of several plausible problems is costing you, that’s what an SEO audit is for.
