Amrut Services

Mehul Dedhia

SEO Consultant · Amrut Services

Author: Mehul Dedhia

  • Soft 404s: Google is usually right

    A soft 404 is Google telling you that a page returned 200 OK and then failed to contain anything.

    People treat it as a bug in Google’s classifier. In my experience it is almost always correct, and arguing with it is the wrong instinct. The page really is empty. Google has simply noticed before you did.

    Google’s own documentation on this is blunt about the cause: a soft 404 happens when a URL returns a success code but the content signals that the page does not exist. The mismatch is the finding. Your server says “here you are” and the page says “nothing here.”

    Where they come from

    Four sources cover nearly everything I have seen.

    Empty category or collection pages. A category with no products in it. The template renders, the header and footer are present, the middle is blank. This is the commonest one on ecommerce sites and it multiplies quietly as stock changes.

    Out-of-stock product pages that hide the product. The URL persists, the theme replaces the buy area with a notice, and what is left is a title and a breadcrumb. To Google that is an absent page wearing a 200.

    Internal search results with no matches. “0 results for…” is a page that exists for every query anyone has ever typed. If those URLs are crawlable you are generating an infinite supply of them.

    Thin location or service pages. Forty pages with a place name swapped and eighty words of body copy. Some will be classified as soft 404s and the rest will be crawled and left out of the index, which is the same judgement arriving through a different door.

    The question that settles it

    Not “how do I stop Google flagging this.” It is: should this URL exist at all?

    If the answer is no, because the category is empty, the product is gone for good, or the search result page was never meant to be a page, then return a real 404 or a 410 and remove it from the sitemap. Google was right. You are agreeing with it, which is faster than disputing it.

    If the answer is yes, the page needs to contain something. An empty category with three sentences about the category and a link to the parent is no longer empty. An out-of-stock product with the specification, the reviews, and a link to the replacement is a page. The fix is content, not configuration.

    The wrong answer, which I have been asked for more than once, is to make the page look fuller to the classifier without making it more useful. That works on the tool and on nothing else.

    The one case where Google is wrong

    It happens, and it is worth knowing the shape.

    Pages that are genuinely useful but very short, such as a definition, a single specification, or a location with an address and a map, can get classified as soft 404s because they look skeletal. If you have a page like that and you believe it earns its place, the fix is to make the useful part unambiguous: real body text, not just fields, and internal links pointing at it so the page does not look abandoned as well as thin.

    But I would check the first explanation properly before reaching for this one. Most of the time the page is exactly as empty as Google thinks.

    And most pages, empty or not, were never going to earn anything. Ahrefs looked at 14 billion pages and found 96.55% get zero traffic from Google. A soft 404 on a page in that 96.55% is not a problem to solve. It is Google and reality agreeing with each other.

    How I check

    Search Console, Page indexing report, the row labelled Soft 404. Then the same triage I use for everything in that report:

    • Is anything in the list a page you meant to publish? Home, service, collection, product, location, blog. If none of those are there, this is platform exhaust and the work is to stop generating it.
    • Open two of them. On a phone. You will usually see the answer in about four seconds.
    • Check what the URL actually returns. If a deleted product is serving a 200 with an “unavailable” message, that is your template making the decision, and it is one fix for every product that ever sells out.

    The short version

    1. A soft 404 is a 200 with nothing in it. Google is usually right.
    2. Four causes cover most of them: empty categories, hidden out-of-stock products, internal search results, thin spun pages.
    3. Ask whether the URL should exist before asking how to clear the flag.
    4. If it shouldn’t, serve a real 404 or 410 and take it out of the sitemap.
    5. If it should, put something on it. The fix is content.
    6. Do not pad the page to satisfy the classifier. That fools the tool and nobody else.

    If a site should be ranking and it isn’t, that’s the work I do. Technical SEO covers the crawling and indexing layer. If you’re not sure which of several plausible problems is costing you, that’s what an SEO audit is for.

  • Crawl budget: almost nobody reading this has one

    Almost nobody reading this has a crawl budget problem, and I would rather say that at the top than sell you five hundred words of suspense.

    Google publishes the thresholds. Their large site owner’s guide to managing crawl budget says it is worth thinking about for sites with one million or more unique pages whose content changes about weekly, or ten thousand or more pages changing daily. They describe those as rough classifications rather than hard cutoffs, which is fair, but the order of magnitude is the point.

    Most of the sites I am called into have between two hundred and ten thousand pages and update monthly. They are not in either bracket and they are not close.

    And yet crawl budget comes up in almost every conversation, usually as the explanation for why pages are not indexed. So it is worth being precise about what it actually is, and about the one version of this that is real.

    Two things wearing one name

    Google splits it in two, and the distinction is the whole of the practical advice.

    Crawl capacity limit is how much Google is willing to ask of your server: how many parallel connections, held how long, before it decides it is being a nuisance. This one responds to how you perform. Slow responses and server errors bring it down. It is a technical ceiling.

    Crawl demand is how much Google actually wants your pages. Google names the inputs: perceived inventory, popularity, and staleness. This one responds to whether the pages are worth having.

    When somebody says “crawl budget” they almost always mean the first and almost always have a problem with the second. Google is not straining to keep up with your six hundred pages. It has looked at them and is not in a hurry.

    Two sites that looked like crawl budget problems and were not

    A fitness certification provider: 1,265 indexable pages. Inside that, 1,082 blog posts carrying an unnecessary /blog/ segment, 266 paginated pages with no canonical, 138 pages under 500 words, and a run of author archives several pages deep for authors with a handful of posts each.

    A classic car marketplace: 9,220 pages indexed, most of them expired vehicle listings nothing linked to.

    Both sites had the symptom people attribute to crawl budget: a large gap between pages that exist and pages doing anything. Neither was anywhere near a million pages. Neither was near ten thousand daily-changing ones. The crawl budget explanation was available, comforting, and wrong in both cases.

    What was actually true is duller. The extra URLs were manufactured by the CMS, nobody had decided to publish them, and Google had correctly declined to invest in them. That is not a budget being exhausted. That is Google crawling the pages and choosing not to index them, which is a different status with a different fix.

    The version that is real

    There is one, and Google names it in the same document: if a significant share of your URLs sit in Discovered, currently not indexed, crawl budget is worth looking at.

    Note which status that is. Not crawled and not indexed, but discovered and not indexed. Google knows the URL exists and has not spent the resource to fetch it. That is a queue problem rather than a quality judgement, and on a genuinely large site with a lot of manufactured URLs it is the thing crawl budget describes.

    Even then the fix is not “increase the budget”, because you cannot. The fix is to stop generating URLs that consume it. Google’s own list of what wastes it is worth reading in full: faceted navigation, duplicate content, soft 404s, hacked pages, infinite spaces, and low-quality or spam content. Every item is something the site is producing, not something Google is withholding.

    What I do instead of raising it

    Three things, and none of them is a crawl budget project.

    • Stop making the URLs. Turn off the archives nobody browses. Decide, per facet, which filtered views deserve to exist. This is upstream of everything and it is the only fix that stays fixed.
    • Use noindex rather than a robots block on the ones already out there. This is the mechanism people reach for last and should reach for first. HTTP Archive’s 2025 Web Almanac found meta robots tags on 47.9% of mobile pages, but a noindex directive on only 2.4%. Half the web is emitting the tag and almost nobody is using it to make a decision. Blocking in robots.txt does not remove a page, because Google cannot read a noindex on a URL it is not allowed to fetch.
    • Look at what Googlebot is actually spending its time on. Search Console, Settings, Crawl Stats, then the breakdown by file type. On more than one site I have found the majority of requests going to scripts and stylesheets rather than pages. That is a real crawl efficiency problem, it has nothing to do with a budget being too small, and it is two clicks from the report everyone is already looking at.

    When I do raise it

    Honestly? On large ecommerce catalogues with faceted navigation, and on listings platforms with a feed behind them. That is close to the whole list. On those, the URL space is combinatorial, since every filter combination is a page, and it can genuinely outrun what Google will fetch.

    If you are a local business, a clinic, a trades company, a professional practice or a store with a few hundred products, and somebody has told you your problem is crawl budget, ask them which of Google’s two thresholds you are near. It is a fair question and it usually ends the conversation.

    The short version

    1. Google’s own thresholds are 1M+ pages weekly or 10,000+ daily. Most sites are not close.
    2. Capacity limit is about your server. Demand is about whether the pages are worth having. People mean the first and have a problem with the second.
    3. “Crawled, currently not indexed” is not a budget problem. Google read the page and declined.
    4. “Discovered, currently not indexed”, at volume, might be. That is the one status worth the conversation.
    5. You cannot raise it. You can stop spending it.
    6. Use noindex, not a robots block. Only 2.4% of pages carry one and it is the right tool here.
    7. Check the crawl stats by file type before concluding anything about volume.

    If a site should be ranking and it isn’t, that’s the work I do. Technical SEO covers the crawling and indexing layer. If you’re not sure which of several plausible problems is costing you, that’s what an SEO audit is for.

  • The 1,630-page store with no robots.txt, and what that actually told me

    A car parts retailer with 1,630 pages indexed, selling every day, had no robots.txt file at all. Not a permissive one. Not an empty one. The URL returned a 404.

    That is not a crisis, and I want to be clear about that before anything else, because “you have no robots.txt” is the kind of finding that gets written in red and sold as urgent. No robots.txt means no restrictions. Google crawls the site. Everything works.

    What it told me was something else. On a catalogue of that size, somebody should have had a reason to open that file at some point, to keep the crawler out of the cart, or the search results pages, or the account area. Nobody ever had. The absence was not the problem. It was the tell that the technical layer had never had anyone’s attention, and the rest of that audit bore it out: 1,277 orphan pages, 1,301 URLs with no H1, 1,344 pages under 500 words.

    That is how I read robots.txt now. Not as a checklist item, but as the cheapest available signal of whether anyone has ever been in there.

    How normal is a missing one

    More normal than you would guess. HTTP Archive’s 2025 Web Almanac found that 13% of robots.txt requests return a 404, and of the files that do exist, 97.6% are under 100 bytes and 1.7% are completely empty.

    Read that second figure again. Almost every robots.txt on the web is three or four lines long. The mental image most people have, a carefully maintained file full of considered directives, describes almost nobody. The realistic version is two lines and a sitemap reference, and for most sites that is genuinely all it should be.

    So the honest framing is: an absent or tiny robots.txt is not evidence of neglect on a small site. On a large database-driven one, it is.

    The mistakes that actually cost something

    In fifteen years the damage I have seen from this file has come from four places, and none of them is the file being missing.

    Blocking something the site depends on. A Disallow on a directory that also holds the CSS or the JavaScript. The pages still get crawled; they just get crawled looking broken, because Googlebot cannot fetch what it needs to render them. This is the single most expensive robots.txt mistake and it is invisible unless you check, because the site looks perfect to you.

    Using it to remove a page from the index. It does not do that. Blocking a URL stops Google crawling it, which means Google cannot see the noindex you also put there, so the page can stay indexed indefinitely, usually with no snippet. Worth knowing that this is a different situation from crawled, currently not indexed, where Google fetched the page, read it, and decided against it. Here Google never got to read it at all and is indexing a URL it has never seen the contents of. If you want something out, let it be crawled and serve noindex. Blocking is how you make a page permanently un-removable.

    A staging block that shipped. Disallow: / is correct on a staging site and catastrophic on the live one, and it moves across during a launch more often than anybody admits. It is a thirty-second check on the day and nobody does it on the day, because on launch day everyone is looking at the front end.

    Assuming it is a security control. It is a public file listing the directories you would prefer people did not look at. Anyone can read yours.

    Google’s own introduction to robots.txt is explicit on that last point and on the second one: the file manages crawler traffic, it is not a mechanism for keeping a page out of the index. Worth reading once properly; it is short, and most of the mistakes above come from assuming it does something it does not.

    The new question: AI crawlers

    This is the part of the file that has actually changed, and it is the one clients now ask about.

    The same Almanac data shows how many robots.txt files name the AI crawlers: gptbot in 4.2%, claudebot in 3.4%, petalbot in 4.4% of files on mobile. Around one site in twenty-five has an opinion. The other twenty-four have not thought about it.

    My position, and it is a position rather than a finding: for most of my clients, blocking them is the wrong call. A local clinic, a trades business, an ecommerce store. These are businesses that want to be found and quoted, and the AI surfaces are increasingly where being found happens. Blocking the crawler removes you from the answer without removing the question.

    The businesses where it is worth a real conversation are the ones whose product is the content: publishers, course providers, anyone whose archive is the asset being sold. That is a commercial decision about licensing, not an SEO one, and I say so rather than pretending I can optimise it either way.

    What I would not do is add the directives because a plugin offered to. Four lines you did not decide on is how the staging block gets shipped.

    What I actually check, in about ninety seconds

    • Does it return 200? A 404 is fine. A 500 is not. Google may treat a server error on robots.txt as a reason to slow or stop crawling the site entirely, which is the one way this file can genuinely take you off the map.
    • Is anything blocked that renders the page? Fetch the CSS and JS paths directly and confirm they are reachable.
    • Is Disallow: / in there? Yes, really. Check.
    • Is the sitemap listed, and does that URL work? One line, frequently either absent or pointing at a sitemap that moved two rebuilds ago.
    • Does anything blocked here also carry a noindex? If so the two are fighting and the block is winning, which is the opposite of what whoever wrote it intended.

    The short version

    1. A missing robots.txt is not a fault. 13% of sites 404 it and most of them are fine.
    2. On a large catalogue, its absence is a tell that nobody has ever been in the technical layer. Go and look at what else is true.
    3. Almost every real file is tiny. 97.6% are under 100 bytes. Yours probably should be too.
    4. Blocking rendering resources is the expensive mistake, and it is invisible from the front end.
    5. Blocking is not removing. A blocked page can stay indexed forever because Google never sees your noindex.
    6. A 500 on this file is worse than a 404.
    7. AI crawler directives are a commercial decision, not an SEO one. One site in twenty-five has made it deliberately.

    If a site should be ranking and it isn’t, that’s the work I do. Technical SEO covers the crawling and indexing layer. If you’re not sure which of several plausible problems is costing you, that’s what an SEO audit is for.

  • 1,562 missing H1s and seven H1s on one page. Same issue, opposite problem.

    Two findings from my audit folder, both of which any crawler would file under the same heading.

    A Shopify homeware store: 1,562 URLs with a missing or empty H1.

    A chiropractic clinic: 7 H1 tags on the home page, several of them blank.

    Sort those by severity and the tool puts the first one two hundred times higher. Both are wrong in the same direction, because the heading structure does not describe the page, and neither is remotely as serious as the number implies. They are also, both of them, a single afternoon’s work, because in each case one template is producing every instance.

    This is the whole problem with issue counts. The number tells you how many pages use a template. It does not tell you how many decisions were made badly, and it does not tell you what anything costs you.

    The same label, five different problems

    Here is what “missing H1” actually looked like across five of these audits.

    • 1,562: a Shopify theme that never outputs an H1 on product or collection pages. One theme file.
    • 1,301: a car parts retailer, same shape, same cause.
    • 809: a continuing-education platform, again templated.
    • 141: a classic car marketplace, plus a separate list of URLs with duplicate H1s and another with duplicate page titles.
    • 48: a medical education provider.

    And then the clinic: six blog posts with no H1 at all, and a home page with seven of them.

    The clinic has the smallest number on the list and the most interesting problem, because those same blog posts also render with no header menu. A tool reports “H1 missing on 6 URLs” and moves on. What is actually happening is that the blog is structurally detached from the rest of the site: no heading, no navigation, no path to anything that sells. The H1 is a symptom of the template being broken, and the template being broken is the finding.

    No crawler will ever tell you that. It has no way of knowing the six URLs are the same six URLs in three different reports.

    The three questions I ask of anything a tool flags

    • 1. How many causes, not how many instances? Divide the count by the number of templates involved. Usually it comes out at one. An issue affecting 1,562 URLs from one template is a smaller job than an issue affecting nine URLs that were each hand-broken in a different way, and the second is far more likely to come back.
    • 2. Is this the same page-set as the finding above it? If two counts match exactly, they are describing one group of pages. On one site, 8,317 orphan pages and 8,317 canonical faults turned out to be the same URLs from one template. Chasing them as separate line items is how a 400-row report gets built and never implemented.
    • 3. What breaks if I do nothing? For a missing H1 on a page nobody was ever going to rank, nothing. For a missing H1 on the service page the business is built on, you have thrown away the clearest statement you get to make about what the page is. Same issue, same severity score, completely different answer.

    What tools over-weight

    Two things get flagged loudly and almost never deserve the position they are given.

    Missing meta descriptions. HTTP Archive’s 2025 Web Almanac found one on 67.2% of mobile pages, so a third of the web has none, including a great many pages ranking at the top of results you care about. It is not a ranking factor, Google rewrites it much of the time anyway, and on a 1,300-page catalogue writing them by hand is weeks of work for close to nothing. Write them on the twenty pages that earn money and leave the rest.

    Heading level order. An H3 following an H1 is reported as an error. It is not one. It is untidy, it matters for screen readers, and it has no bearing on whether the page ranks. I put it in audits under a different heading from the things that do.

    For scale on how routine that is: WebAIM crawled the top million home pages in February 2026 and found skipped heading levels on 41.8% of them, up from 39% the year before. Two pages in five. If it were a ranking problem, two fifths of the web would be broken.

    What tools under-weight, or miss entirely

    The inverse list is shorter and more useful.

    An invalid element inside the <head>. The Almanac found this on 10.3% of mobile pages, one in ten, most often a stray <img> or <div>. It matters more than almost anything on a standard issues list, because the browser closes the head at the first invalid element and everything after it is treated as body content. Your canonical, your meta robots, your hreflang: if they sit below the break, they are not being read. The tool will happily report that your canonical is present, because it is present in the source. It is just not in the head any more.

    Anything requiring a judgement about intent. Whether a page should exist. Whether forty location pages are a legitimate structure or a doorway. Whether the thing ranking for your best query is the page you wanted ranking. No tool has an opinion on any of that, and those are usually the findings that change the outcome.

    Whether the site says what the business is. Every crawler I have used will pass a home page that describes a mood rather than a company.

    So what do you do with the 400

    Not triage it top to bottom. Read it once for structure, then throw away the ordering.

    Group by cause rather than by issue type: which template, which plugin, which platform default. Most reports of four hundred rows collapse to somewhere between six and twelve actual causes, and that list is short enough to have an opinion about.

    Then sort those causes by what they are costing, which is a question about the business and not about the crawl. A template fault on the product pages outranks a template fault on the author archives even if the author archives generated more rows.

    Then do three of them. Three is usually the honest number for what actually gets implemented between an audit landing and the next quarter starting, and picking the three is the part of the job that the tool cannot do for you.

    The short version

    1. An issue count measures template reach, not severity. 1,562 and 7 were the same class of fault and the same size of fix.
    2. Divide by causes. Four hundred rows is usually six to twelve real problems.
    3. Matching counts across findings mean one page-set. Do not chase them separately.
    4. Ask what breaks if you do nothing. On most flagged URLs the answer is nothing.
    5. Meta descriptions and heading order are over-weighted. A third of the web has no meta description and ranks fine.
    6. An invalid tag in the <head> is under-weighted and silently disables everything below it. One page in ten has one.
    7. The findings that matter need a judgement about intent, and no tool has one.

    If a site should be ranking and it isn’t, that’s the work I do. What I check first is the sequence I run before I open any tool at all. If you’re holding a report you can’t prioritise, that’s what an SEO audit is for.

  • Four ways canonical tags get misused, and one of them is my own advice

    Four sites, four different things done wrong with one tag. I pulled these out of the audit folder because they make a set, and because the fourth one is a recommendation I wrote myself, twice, and got wrong.

    1. The tag is simply absent, at scale

    A classic car marketplace: 8,317 pages with no canonical URL, or with a canonical still pointing at the http version of a site that had been on https for years.

    A car parts retailer, same year: 1,347 pages in the same state.

    Neither of those numbers is a page-by-page failure. Nobody omitted a tag 8,317 times. One template was responsible for most of the site, that template had no canonical in it, and the omission printed itself once per URL. The number tells you how many pages use the template, not how many mistakes were made. There was one mistake.

    That distinction matters because it changes the size of the job. Presented as “8,317 canonical errors” it sounds like a quarter’s work and it gets deferred. Presented as “one template, one line” it gets done on a Tuesday.

    How common is missing a canonical at all? HTTP Archive’s 2025 Web Almanac found one on 67% of mobile pages, so a third of the web has none, and most of those sites are fine. Absence is not automatically a fault. It becomes one when the site also generates several URLs for the same thing, which is exactly what a listings platform does.

    2. The homepage canonical is wrong, or missing entirely

    The same two audits, both on the single most important URL on the site.

    On the marketplace, the recommendation reads: Canonical URL is not defined. It should be https://www.classiccardeals.com/. On the retailer: Canonical URL is incorrect.

    This one is worth calling out separately from the 8,317, even though it is the same tag, because the homepage is usually reachable at four or five addresses: with and without www, with and without a trailing slash, http and https, sometimes /index.php as well. Every other page on the site links to it, so whichever version Google settles on inherits the site’s internal link equity. Leaving that decision to Google is not catastrophic. It is just an odd thing to leave to chance on the one URL you care about most.

    It takes thirty seconds to check and I have found it wrong on sites that had been paying an agency for years.

    3. The canonical only exists after JavaScript runs

    This is the one that does not show up in a normal check, because when you view the rendered page the tag is there.

    The Almanac data has the tell in it. A canonical appears in the raw HTML of 64.3% of mobile pages, and in the rendered HTML of 66.1%. That gap, call it two pages in a hundred, is sites where the canonical does not exist in the served document at all. Something injected it in the browser.

    It usually works. Google renders. But you have taken a directive that could have been a static line in the <head> and made it conditional on a script executing correctly, on a crawler’s schedule, with a rendering budget attached. When it does not work you get no error anywhere. You get a page quietly canonicalising to nothing, and you find out months later.

    Check the raw source, not the inspector. view-source: and Ctrl+F, or fetch the URL with curl. If the tag is not in what the server sent, it is not as reliable as you think it is.

    4. Canonicalising paginated pages to page one, which is what I told two clients to do

    Here is the uncomfortable one.

    In a 2022 audit of a fitness certification provider I found 266 paginated pages with no canonical and wrote: The canonical URL for paginated pages should be the parent URL. For example: Canonical URL for /blog/page/1 should be /blog/.

    In another audit the same year, a pharmacy education site had paginated author and category archives that were self-canonicalising, each page of the series pointing at itself. I flagged it as a fault and recommended they point at the parent instead.

    Google’s position is the opposite. Their pagination guidance states it plainly: “Don’t use the first page of a paginated sequence as the canonical page. Instead, give each page its own canonical URL.” The reasoning is straightforward once you say it out loud. Page four of a listing is not a duplicate of page one. It has different items on it. Telling Google it is a copy of page one is telling Google those items do not exist.

    So the second client was already doing it correctly and I told them to change it.

    I do not know whether they did, and I have not gone back to ask, which is its own small lesson about follow-up. What I can say is that the instinct underneath the recommendation was not mad. Paginated archives genuinely do generate a lot of low-value URLs, and wanting to consolidate them is reasonable. The canonical tag is simply the wrong instrument. If page four of an author archive should not be in the index, the answer is noindex, or not generating the archive, or fixing why you have a nine-page archive for an author with eleven posts. It is not lying to Google about what the page contains.

    The same page also carried the rel=next and rel=prev advice that was standard at the time. Google no longer uses those either.

    What I actually check now

    Four things, in this order, and the whole pass takes about ten minutes on a site of any size.

    • Is there one at all, and is it in the raw HTML? Not the rendered DOM. The served document.
    • Does it point at the https, canonical-host version of itself? Most of the time a self-referencing canonical is the right answer and the interesting cases are the exceptions, not the rule.
    • Is the count suspiciously round, or suspiciously equal to something else in the crawl? 8,317 canonical faults and 8,317 orphan pages on the same site are not two findings. They are one page-set with one template behind it. I have written up what a 90% orphan page rate looks like separately. Chasing them as separate line items is how an audit becomes a spreadsheet nobody implements.
    • Is anything canonicalising away its own content? Paginated series, filtered views that show genuinely different products, location pages that are actually different. A canonical pointing somewhere else is a request for deletion from the index, and people use it as though it were a tidiness setting.

    The short version

    1. Missing at scale is one template mistake, not thousands. Say it that way and it gets fixed.
    2. Check the homepage separately. It is reachable at several addresses and every internal link points at it.
    3. Read the raw HTML. A canonical injected by JavaScript works until it doesn’t, and fails silently.
    4. Do not canonicalise page four to page one. Google says give each page its own. I told two clients otherwise and I was wrong.
    5. Matching numbers across findings mean one cause. Fix the template, not the list.
    6. A canonical is not a tidiness tag. Pointing it elsewhere asks Google to drop the page.

    If a site should be ranking and it isn’t, that’s the work I do. Technical SEO covers the crawling, indexing and canonicalisation layer. If you’re not sure which of several plausible problems is costing you, that’s what an SEO audit is for.

  • The client never knew I existed, and that was the point

    The best result I can show you from the last two years belongs to a law firm I have never spoken to.

    They were the client of an agency. The agency brought me in on a white-label basis. I set the strategy and handled link building, their team implemented alongside me. Twelve months of Business Profile data, a flat autumn, and a step up from February that held through the summer.

    The firm does not know my name. That is not a failure of the arrangement. It is the arrangement.

    What white label actually means in practice

    The phrase gets used loosely, so here is the version I work to.

    An agency has a client with a problem they cannot solve in-house: usually technical depth, sometimes link building, occasionally just capacity. They bring in a specialist. The specialist’s work goes out under the agency’s name. The end client deals with one supplier and one invoice, which is what they wanted when they hired an agency.

    Three things follow from that, and they are the parts people get wrong.

    The agency is my client, not the end client. I report to them, in their format, on their schedule. If the end client needs telling something uncomfortable, the agency decides how and when, because they own that relationship and they know how much bad news it can take.

    I do not get credit, publicly or privately. No case study with their logo, no mention in a pitch deck, nothing on my site that identifies them. The chart I mentioned above is on my site only because I asked and the agency agreed.

    The agency has to be able to explain my work as their own. This is the part that gets underestimated. If I hand over a recommendation the account manager cannot defend in a meeting, I have made their life worse regardless of whether the recommendation was right.

    Why agencies actually buy this

    Not the reasons the sales pages give.

    A client asked a question they could not answer. This is most of it. Plenty of the agencies I work with are good at what they do, whether that is design, content, campaigns or the client relationship, and SEO is simply not their strength. That is precisely why they have me. The question is usually technical: why did traffic drop after the migration, why is Google not indexing the new section, what is happening with the JavaScript. Bringing somebody in is faster than hiring and much faster than learning.

    They won an account that is bigger than their bench. A retainer that justifies specialist input for six months without justifying a permanent hire.

    They want a second opinion before committing. An agency about to recommend a six-figure rebuild sometimes wants someone outside the room to check the diagnosis first. This is the engagement I most enjoy and the one I am asked for least, because asking feels like an admission.

    None of those are “we want cheap labour”. The agencies I work well with are not buying hours, they are buying a judgement they do not currently have. When somebody is shopping purely on rate it usually does not work, for reasons in the next section.

    When it does not work

    • When the agency wants a deliverable rather than a diagnosis. “Send us a technical audit for this client” with no access, no context and no conversation produces a document, not an outcome. I can write it. It will be worth very little, and it will be my name-in-their-name on something I could not stand behind.
    • When there is no route to the developer. Most technical recommendations need somebody to change something. If the chain runs specialist to agency to client to their web team to an outsourced developer, nothing ships. I now ask early who actually deploys code and how long it takes them.
    • When the agency is reselling at a margin that requires me to be quick. If the price only works when I spend four hours on something that needs twenty, we will both be unhappy by month three, and the end client, who is not part of that negotiation, gets the worst of it.
    • When nobody has agreed what happens if I disagree with them. Occasionally I will conclude that the work the agency has already sold is not the work the client needs. That is a real risk of hiring a diagnostician. Worth deciding in advance who says what to whom, rather than at the point it happens.

    What I would ask before starting

    If you run an agency and you are considering this, the questions I would want answered are:

    • What has the client already been told, and by whom?
    • Who deploys changes, and what is the realistic turnaround?
    • What is already committed in the contract that I need to work around?
    • Am I allowed to talk to the developer directly, or does everything route through you?
    • If I find something that contradicts your existing plan, how do you want that handled?

    Five questions, most of them answerable in a sentence. The engagements that go wrong are almost always ones where nobody asked the last one.

    The honest trade

    For me the trade is straightforward and I would rather be plain about it. White-label work is steadier than direct client work, the person briefing me already understands the vocabulary, and I skip the entire business of selling. In exchange I give up attribution and I do not own the relationship. Those are real costs and I accept them knowingly.

    For the agency, the trade is that you keep the client relationship and the credit, and you get a capability you do not have to hire. In exchange you carry the coordination and you have to be able to explain my work as your own.

    It works when both halves of that are understood. It fails when one side thinks they are buying something else.

    The short version

    • The best result I can show is a client I have never met. The agency owns the relationship, I set strategy and did link building, their team implemented.
    • The agency is my client. They decide what the end client hears and when.
    • Agencies buy this for a question they cannot answer, an account bigger than their bench, or a second opinion before committing. Not for cheap hours.
    • It fails when there is no route to a developer, when the margin requires speed over accuracy, or when nobody has agreed what happens if I disagree with the existing plan.
    • Ask the five questions before starting. The last one matters most.

    If a site should be ranking and it isn’t, that’s the work I do. White label SEO is how this works commercially, and SEO for personal injury law firms has the twelve-month case from the engagement above.

  • Two things I recommended in most of my audits, and have stopped recommending

    I went back through eleven audits I have written since 2021 to count what I actually find, rather than what I remember finding.

    Two recommendations show up in most of them that I would no longer make. One is out of date because Google changed something. The other was never right, and I repeated it for years because everyone repeats it.

    Publishing this is mildly uncomfortable and I think it is the most useful thing in the whole exercise, because if you have had an audit from anyone in the last few years there is a good chance it contains one or both.

    One: add FAQ schema

    Recommended in 7 of my 11 audits, including one written in 2026.

    The reasoning was that FAQPage markup earns you an expandable FAQ block in the search result, taking up more space and pushing competitors down. For a while that was straightforwardly true and it was one of the cheapest wins available.

    Then it stopped, on a date, in writing. Google announced on 8 August 2023 that FAQ rich results “will only be shown for well-known, authoritative government and health websites. For all other sites, this rich result will no longer be shown regularly.” The same post was updated a month later to deprecate HowTo rich results on desktop as well. Then in May 2026 the feature went entirely. Google has since removed the FAQ documentation altogether, on the grounds that it “is no longer shown in Google Search results.” Search Console’s reporting for it is being switched off through this summer.

    So the visibility argument is gone. Not diminished. Gone. If somebody sells you FAQ schema in 2026 as a way to get more space in the result, they have not read the release notes.

    What is still true: Google has said it continues to read FAQPage markup to help understand a page. The markup is not harmful and if it is already there I would leave it. Marking up genuine questions and answers is a reasonable way to describe your content unambiguously.

    What has changed is that it cannot be sold as a visibility win, and it should not be near the top of anyone’s priority list. In my own audits it sometimes was.

    Two: nofollow your outbound links to stop link equity leaking

    Recommended in 8 of my 11 audits. Every ecommerce and B2B one.

    The wording varied but it amounted to: you are leaking PageRank through outbound links and social icons, so add rel="nofollow" and keep that authority inside the site.

    This is PageRank sculpting, and it has not worked since 2009.

    And the mechanic underneath it was formally retired years ago. Google’s September 2019 announcement states it plainly: “All the link attributes, sponsored, ugc, and nofollow, are treated as hints about which links to consider or exclude within Search,” with nofollow becoming a hint for crawling and indexing as of 1 March 2020. You cannot sculpt anything with a hint.

    Google changed the handling that year specifically to stop it. A nofollowed link still consumes its share of the page’s outgoing PageRank. The share is simply discarded rather than passed. You do not redirect it inward. You throw it away. Nofollowing your own Instagram icon achieves nothing except deleting a small amount of value.

    Since March 2020 nofollow has been a hint rather than a directive, so even the discarding is now uncertain, which does not make the tactic work. It makes it less predictable while still being pointless.

    I kept recommending it because it appears in a great many audit templates, mine included, and because it sounds mechanically plausible. Neither is a reason.

    The instinct underneath it was sound, and this is the part worth keeping. Sending site-wide links to low-value destinations is genuinely worth avoiding. A footer with forty links in it, a sidebar of unrelated partner sites, a header carrying every page you have ever made. Those are real problems, because they dilute the internal signal about what matters and they appear on every page.

    The fix is to have fewer links, not to nofollow the ones you have. Remove the link, do not neuter it.

    Why I am posting this rather than quietly updating the template

    Three reasons.

    The first is that both recommendations are still in circulation. FAQ schema advice from 2022 is all over the web and reads as current. PageRank sculpting has survived seventeen years past its expiry because it sounds like it ought to work.

    The second is that if you are holding an audit of mine from the last few years, you should know which two lines to ignore. That seems like the minimum.

    The third is that this is what auditing anything honestly looks like. I count what I find rather than trusting my memory, and occasionally the count tells me something I did not want to know. An audit that only ever confirms the auditor’s priors is not an audit.

    What did hold up

    For completeness, since two out of a long list is a decent hit rate: everything else in those eleven audits stands. The internal linking findings, the thin content findings, the heading structure problems, the schema gaps, the location strategy questions: all still what I would say today.

    The two that failed both failed the same way. They were mechanisms I had inherited rather than things I had checked.

    The short version

    • Counted across 11 audits since 2021: FAQ schema recommended in 7, nofollow-your-outbound-links in 8.
    • FAQ rich results were restricted in August 2023 and removed in May 2026. The markup is still read; the visibility win no longer exists.
    • Nofollowing outbound links does not redirect PageRank inward. It discards it. That has been true since 2009.
    • The instinct behind the second one is sound: fewer site-wide links to low-value destinations. Remove them rather than nofollowing them.
    • Everything else in those audits still stands. The two that failed were both mechanisms I had inherited rather than checked.

    If a site should be ranking and it isn’t, that’s the work I do. Technical SEO covers the crawling and indexing layer, and SEO consulting is where I get asked for a second opinion on somebody else’s recommendations.

  • 466 of 812 pages were thin. The fix was not to rewrite 466 pages.

    Two audits, two sites, same finding.

    One had 466 of 812 indexable pages classed as thin. Fifty-seven per cent. The other had 93 of 189. Forty-nine per cent.

    Different businesses, different platforms, different catalogue sizes. In both cases roughly half of everything the site had in the index was a page nobody would choose to land on.

    That is a normal result. I do not think I have ever audited a catalogue site and found the number to be small.

    What “thin” means, and what it does not

    A thin page is one that does not have enough on it to be the best answer to anything. Not a word count. A page can be four hundred words and perfectly adequate, and eight hundred words of padding is still thin.

    Google’s own framing supports that. Where the spam policies address this, under scaled content abuse, the definition is “many pages generated for the primary purpose of manipulating search rankings and not helping users”, explicitly including stitching content together from other pages and generating pages that add no value “no matter how it’s created.” Read what is absent from that: any number. Not a word count, not a minimum, not a ratio. The test is whether the page adds something, and 466 short pages that each genuinely answered something would not be in scope at all.

    Two things it does not mean, and both matter because both cause people to do expensive and unnecessary work.

    It does not mean the page is penalised. Thin pages mostly just fail to rank. That is a lost opportunity rather than an active harm, and the distinction changes the maths on whether it is worth fixing.

    It does not mean every one of them needs prose. This is the assumption I spend the most time arguing with, and it is what makes the finding feel unfixable. Told that 466 pages are thin, an owner hears “write 466 descriptions”, quite reasonably concludes that is not happening, and does nothing at all.

    Three questions before anyone writes anything

    The 466 is not a to-do list. It is a starting point for triage, and the triage is three questions.

    Should this page exist?

    A large share of thin pages should not be in the index at all. Tag pages, filtered variants, empty categories, paginated series, discontinued products with no replacement. On the site with 466, a substantial fraction were pages that existed because the platform generated them, not because anyone wanted them.

    Deleting or noindexing a thin page is a complete fix and it takes seconds. Do this pass first, because it usually removes the majority and it costs nothing.

    Does this page get any impressions?

    Search Console will tell you which of the survivors Google is already showing to somebody. A thin page picking up impressions is a page with demand and an inadequate response. That is the highest-return writing you will ever do, because you are not gambling on whether anyone wants it. You know they do.

    A thin page with no impressions in twelve months is a page nobody is looking for. Improving it is speculative.

    Is it a page you sell from?

    Category and collection pages are usually worth more than individual products, because they catch the broader query and because one of them serves many products. If you only fix twenty pages, fix twenty collection pages.

    After those three questions the 466 is usually a list of thirty or forty pages that genuinely warrant work. That is a project somebody can actually do.

    The manufacturer description question

    The specific version of this on product pages: should you rewrite the descriptions the manufacturer gave you?

    The received answer is yes, always, because duplicate content. I do not think that is right, and I have seen the “yes always” answer waste a great deal of money.

    What actually happens with a manufacturer description is that you are one of several hundred retailers publishing identical text. Google is not going to penalise you for it. It will simply have no reason to prefer your page, so you will rank on everything else: your authority, your reviews, your price, your delivery terms, your internal linking.

    Which means the question is not “is this duplicated” but “is there any version of this page that could win?”

    For a commodity product sold by four hundred retailers at the same price, probably not, and rewriting the description will not change that. Leave it. Spend the effort on the category page above it.

    For a product where you have something to add, because you stock it, you have used it, you know which of the four variants people regret buying, and you have fitting notes for the vehicle it goes on, then yes, and the win comes from the added information rather than from the text being unique.

    Uniqueness is a by-product of having something to say. It is not the goal, and pursuing it directly produces spun descriptions that are unique and still worthless.

    What I would do with 466 thin pages

    1. Count them properly and split by template. Products, collections, tags, blog, paginated. The distribution tells you what kind of problem it is before you look at a single page.
    2. Remove or noindex everything that should not exist. Usually the largest bucket.
    3. Cross-reference the survivors against Search Console impressions. Sort descending.
    4. Do the collection pages first, then the products with demand.
    5. Leave the long tail alone, and stop treating it as debt. A commodity product page that nobody searches for is not costing you anything by being brief.

    That turns an unfixable-sounding number into about a fortnight of work with a defensible order to it.

    The short version

    • Two audits: 466 of 812 pages thin (57%) and 93 of 189 (49%). This is a normal result, not an unusual one.
    • Thin pages mostly fail to rank rather than cause harm. The cost is opportunity, not penalty.
    • Most of them should be removed or noindexed, not rewritten. Do that pass first. It is free and it usually clears the majority.
    • Of what remains, write for the pages with existing impressions and for collection pages. Ignore the rest.
    • On manufacturer descriptions: rewrite where you have something to add, not because duplication is inherently a problem. Uniqueness is a by-product of substance, not a target.

    If a site should be ranking and it isn’t, that’s the work I do. Ecommerce SEO covers category and product architecture at scale, and Shopify SEO covers the platform defaults that generate a lot of these pages in the first place.

  • How to prove to Google that you are one business

    Most advice on this topic is one sentence long: keep your name, address and phone number consistent everywhere.

    Nobody disagrees with it and almost nobody can act on it, because it doesn’t say where “everywhere” is, what “consistent” means, or how you’d know if you’d failed. I’ve had clients tell me their details were consistent while their own contact page was linking to another company’s Google listing.

    So this is the procedure instead. It takes about twenty minutes and I run it on every local site before I look at anything else.

    What you’re actually proving

    Google is trying to answer one question about you: which real-world business is this?

    It has your website, your Business Profile, your social accounts, whatever directories have scraped you, and any citation your last agency built. From that pile it has to decide that all of it describes one entity, and if the pile contains contradictions, it resolves them without asking you.

    “NAP consistency” is a shorthand for that. But the failures I actually find are rarely a mistyped suite number. They’re a site that describes a different business in one or two places and nobody noticed, usually because the site was adapted from somebody else’s build and the details came along with it. I’ve written about the worst version of that elsewhere.

    So run the check as an identity check, not a spelling check.

    The twenty-minute procedure

    Open your Business Profile in one tab. That’s the reference copy. Not your letterhead, not what’s in your head. Whatever is on the profile is what Google already believes, and your job is to make everything else agree with it.

    Then work through these.

    1. The footer, on a page you didn’t build recently.
    Not the home page. Pick an interior page: a service page, a blog post. Footers get edited on the home page and forgotten elsewhere on badly built themes.

    Check the business name character for character, the street address, and the phone number. Then click the address. On most sites it’s a link, and on more sites than you’d expect it goes somewhere wrong.

    2. The contact page, as a separate check.
    The contact page usually predates the current footer and was written by a different person. Treat it as an independent source. Same three fields, same click-the-link test, plus the map embed. Open it and confirm the pin is your building and the label is your name.

    Take map embeds from your own Business Profile listing, using the Share → Embed option on the listing itself. If you get the embed by typing the address into Maps, you can silently pick up the unit next door or a previous tenant.

    • 3. Every social icon in the header and footer.
      Click each one. All of them. This is the single highest-yield thirty seconds in the whole procedure and nobody does it, because the icons look right and clicking your own social links feels pointless.
    • 4. Your schema, against your visible page.
      If you have LocalBusiness markup, read it and compare it field by field with what a visitor sees. Name, address, phone, hours, URL. Markup that contradicts the page is worse than no markup, and markup describing hours you don’t publish anywhere is a rich results violation.
    • 5. The name itself: is it one name?
      Businesses accumulate variants: a legal name, a trading name, a name with the town in it, a name with “Ltd” or “LLC” sometimes. Pick the one on your Business Profile and use exactly that everywhere. Not a version of it.
    • 6. Hours in three places.
      Profile, website, and any booking system embed. Booking widgets are the usual offender because they’re configured once and never revisited.
    • 7. Then, and only then, the directories.
      This is last on purpose. Citation cleanup is what most local SEO sells, and it’s the least of it. If your own site contradicts your own profile, no amount of directory tidying resolves the ambiguity. You’re paying somebody to correct third parties while the primary source is still wrong.

    What “consistent” actually means

    Stricter than people assume in one place, looser in another.

    Strict on the name. “Riverside Chiropractic” and “Riverside Chiropractic Clinic” are two strings. Pick one.

    Strict on the phone number. One number, in one format. Not a tracking number on the site and the real number on the profile. That’s a common and genuinely damaging mismatch, and if you need call tracking, use a solution that swaps the number dynamically while leaving the canonical one in the markup.

    Reasonable on the address. “Street” versus “St” is not what’s costing you. Google normalises that. A different unit number is a different address.

    Strict on entity links. Every link that claims to be you, whether maps, social or booking, must actually be you. This is the one where I find real failures, and it’s binary. It either points at your business or it doesn’t.

    What this doesn’t do

    I’d rather set the expectation properly.

    Getting this right will not lift you up the map pack. It’s a hygiene floor, not a lever. Proximity, review volume, category and the general strength of the site do far more for local visibility than consistent formatting ever will.

    What it does is remove a reason for Google to be uncertain about you, and remove a way for a customer to end up at a competitor’s listing from your own contact page. On the three clinic sites I looked at this year, that second thing was not hypothetical.

    It’s the check I run first because it’s cheap, it’s binary, and when it’s wrong it’s badly wrong.

    The short version

    1. Your Business Profile is the reference copy. Everything else agrees with it, not the other way round.
    2. Click every address link and every social icon. Thirty seconds, highest yield in the procedure.
    3. Take map embeds from your own listing, not by searching the address.
    4. Check the contact page separately from the footer: different age, different author.
    5. Compare schema against the visible page, field by field.
    6. One name, one phone number, exactly. Street/St doesn’t matter; a different unit number does.
    7. Directories last. Fix your own site before paying anyone to fix other people’s.
    8. This is a floor, not a lever. It removes doubt. It doesn’t create visibility.

    If a site should be ranking and it isn’t, that’s the work I do. Local SEO is where most of this sits. If you’re not sure which of several plausible problems is costing you, that’s what an SEO audit is for.

  • 659 tag pages, competing with the pages the store actually sells from

    A Shopify store I audited had 659 tag pages indexed.

    Not products. Not collections. Tag pages: the URLs Shopify generates automatically when you tag a product, of the form /collections/something/tag-name. Nobody had created them on purpose. Nobody knew they existed. They outnumbered the store’s real commercial pages several times over.

    What Shopify is doing when you tag a product

    Tags are an organisational feature. You add them so you can filter your own admin, drive collection rules, and let customers narrow a collection down.

    The side effect is that Shopify will serve a crawlable, indexable URL for the tag. Tag ten products with “waterproof” and you have a /waterproof variant of that collection, live, linkable and eligible for the index.

    Do that across a catalogue with a normal number of attributes, colour, material, size, use case, season and brand, and the count multiplies fast. It is a combinatorial problem, and it grows quietly because nothing about adding a tag feels like publishing a page.

    That is how you get to 659 without anyone making a decision.

    Why it is a problem, stated precisely

    I want to be careful here, because this topic attracts overstatement.

    It is not a penalty. Nobody is going to action a store for having tag pages. Google is generally competent at working out which of several similar URLs it wants, and much of the time it will simply pick one and ignore the rest.

    It is not primarily a crawl budget problem either. Crawl budget matters at a scale most stores never reach. If you have a few thousand URLs, Google is not struggling to get round them.

    The actual cost is more specific and less dramatic.

    You have handed Google a choice you did not want it to make. For a query like “waterproof jackets”, your store now offers a properly built collection page with a description, curated products and internal links pointing at it, and also a bare tag page listing the same products with no copy and nothing linking to it. Those compete. Sometimes the tag page wins, and the customer lands on the thinner of your two options.

    They are thin by construction. A tag page is a product grid with a heading. There is no description, no buying guidance, no reason for it to be the best result. Enough of them and you have a store where a large fraction of indexable URLs are pages you would be embarrassed to send a customer to.

    They dilute internal linking. Whatever authority flows around the store gets spread across URLs that were never meant to receive it.

    On this store the third point was the one that mattered. The tag pages outnumbered the collections. The signal about which pages the business actually cared about was buried.

    What I would do about it

    In order.

    • Work out how many you have. site:yourstore.com inurl:/tag/ will give you a rough count, and Search Console’s page indexing report will show you what has actually been indexed. Do this before deciding anything. Plenty of stores have twelve, and twelve does not matter.
    • Decide which ones you actually want. This is the part people skip. Some tag pages are genuinely useful and deserve to be real pages: if “waterproof jackets” is something customers search and you have thirty of them, that should be a proper collection with a description, not a tag page and not nothing.
    • Noindex the rest. In theme.liquid, conditionally output a noindex meta tag when the template is a tag-filtered collection. This is the standard fix and it is a few lines. Leave them crawlable. You want Google to see the tag so it drops the URL, and blocking in robots.txt prevents exactly that.
    • Do not canonical them to the parent collection and consider it solved. Shopify already sets a canonical on tag pages pointing at the parent collection. That is a hint, and Google will disregard it when the pages differ enough. If canonicals were sufficient, this store would not have had 659 of them indexed.
    • Then look at whether the collection pages are worth ranking. Removing a thin competitor only helps if the page that survives is good. On most stores I audit, the collection pages have a title and a product grid and nothing else.

    The wider version of the same problem

    Tag pages are the Shopify-flavoured instance of something you see on every ecommerce platform: the store generates URLs faster than anyone decides to publish pages.

    WordPress does it with tag and author archives. Faceted navigation does it with parameters. Shopify does it with tags and with the /collections/all/product duplication of every product URL.

    The common failure is not that these URLs exist. It is that nobody has ever gone and looked at how many there are. In the audits I have done, the number is almost always higher than the owner expects, and the first useful thing I do is simply count.

    For a sense of how many stores this is true of: HTTP Archive’s 2025 Web Almanac found ecommerce software on 19.2% of mobile sites, with Shopify accounting for 25.3% of those. Roughly one site in twenty on the mobile web is a Shopify store, and every one of them mints a tag URL the moment somebody types a word into the tags field. The 659 I found were not a misconfiguration by a careless store owner. They were the platform doing exactly what it was built to do.

    The short version

    • One Shopify store, 659 indexed tag pages that nobody had created deliberately.
    • Shopify serves an indexable URL for every tag on a collection. Tag combinations multiply fast.
    • It is not a penalty and it is rarely crawl budget. It is competition between your good page and your thin one, and dilution of the internal links pointing at what you sell.
    • Count them first. Promote the few worth having into real collections. Noindex the rest in theme.liquid: crawlable, not blocked.
    • Shopify’s own canonical is a hint, not a fix. Those 659 were indexed with canonicals already in place.

    If a site should be ranking and it isn’t, that’s the work I do. Shopify SEO covers the defaults that create this kind of problem, and Ecommerce SEO covers architecture at catalogue scale.