Every crawler tells you what is on your site. Log files tell you what Googlebot actually did about it, and those are not the same document.
This is the one technical check I rate highly and recommend rarely, because for most sites the effort is not worth it. I want to be straight about that before explaining what it is good for.
What only logs can tell you
- Which pages Googlebot has never fetched. Not “not indexed”. Never requested at all. A crawler tells you a page exists and is indexable. A log tells you Google has not been near it in three months. Those pages are invisible in every other report because there is no data about them to show.
- How often it comes back. Crawl frequency by section is a rough proxy for how much Google cares about each part of the site. If the blog is fetched daily and the service pages monthly, that tells you where the perceived value sits, and it is often the reverse of where the business wants it.
- What it is wasting requests on. Parameter URLs, old redirects, files nobody links to any more. You can infer this from the Crawl Stats report in Search Console, but the logs give you the actual URLs rather than a category chart.
- Whether it is really Googlebot. A meaningful share of traffic claiming to be Googlebot is not. Verifying by reverse DNS changes the numbers, sometimes a lot.
When I actually ask for them
Three situations, and outside these I do not raise it.
A large site where pages are not getting indexed and nothing else explains it. If a section is sitting in Discovered, currently not indexed at volume, which is a different status from pages nothing links to at all, logs answer whether Google is ignoring it or has never found a path to it. That is a different fix each way.
After a migration. Logs show whether Google has picked up the new URLs and how fast it is abandoning the old ones. Nothing else shows the transition in progress.
When somebody claims a crawl budget problem. Logs settle it in an afternoon, usually by showing there is no such problem. which there usually is not.
Google’s own threshold for the question being worth asking starts at a million pages updating weekly, or ten thousand changing daily. If the site is not near either, logs will confirm what the thresholds already told you, which is a legitimate reason to pull them when a client insists, and not a reason to start there.
Why I recommend it rarely
Getting the logs is the hard part and it has nothing to do with SEO. On managed hosting they may not be retained. Behind a CDN you need them from the CDN as well or you are looking at a fraction of the traffic. Someone has to be persuaded to hand over a file that also contains visitor IPs, which is a data protection conversation before it is a technical one.
Then the volume is awkward, enough to be annoying but not enough to justify tooling, and the analysis needs a person who knows what a normal pattern looks like.
For a local business or a store with a few hundred products, all of that effort buys you very little. The Crawl Stats report in Search Console covers the same ground well enough, for free, in two clicks. I would exhaust that first every time.
What I look at when I do have them
Requests by response code. A high share of 404s or 301s means Google is spending its time on URLs that no longer exist. That is the most common finding and the easiest to act on.
Requests by directory. Where the attention is going, at a glance, and whether it matches where the money is.
Requests by file type. Scripts and stylesheets against actual pages. I have seen sites where the majority of Googlebot requests were not pages at all.
That last one is less surprising than it sounds once you look at what a page now weighs. In HTTP Archive’s 2025 Web Almanac, drawn from 16.2 million sites, the median mobile home page ships 632 KB of JavaScript against 22 KB of HTML. Googlebot has to fetch the furniture to see the document, so a crawl log dominated by assets is partly just the modern web. The question is whether your ratio is worse than that, and by how much.
Pages in the sitemap with zero requests. The most useful single query in the whole exercise, and the one you cannot run any other way.
The short version
- Logs tell you what Googlebot did. Crawlers tell you what exists.
- Only logs show pages Google has never fetched.
- Worth it for large sites with unexplained indexing gaps, migrations in progress, and disputed crawl budget claims.
- Not worth it for most small and mid-size sites. Crawl Stats covers it.
- Getting the file is harder than reading it, and it is a privacy conversation first.
- Sitemap URLs with zero Googlebot requests is the query that justifies the exercise.
If a site should be ranking and it isn’t, that’s the work I do. Technical SEO covers the crawling and indexing layer. If you’re not sure which of several plausible problems is costing you, that’s what an SEO audit is for.
