Three weeks after a routine platform migration, an outdoor gear retailer’s organic traffic started sliding. Rankings looked fine. Redirects checked out. Every page a human clicked on loaded exactly as it should. And yet Google Search Console kept reporting that hundreds of live, revenue-generating pages did not exist.
That contradiction is the whole story. When a site is broken, it’s usually broken for everyone, humans and crawlers alike. This time, only Googlebot was being lied to.
When “everything looks fine” is the warning sign
The retailer’s SEO lead had done the migration checklist correctly. Every old URL had a 301 redirect mapped to its new equivalent. Title tags stayed untouched, on purpose, since changing them mid-migration muddies the signal when something else goes wrong. Staging tests passed. The launch weekend was quiet.
Then, in early September, year-over-year organic sessions dipped about 20 percent. Two weeks later, after a round of “wait and see,” the dip became a 40 percent drop. No manual action notice. No algorithm update on record for that window. No obvious content quality issue. Just fewer visitors, arriving on fewer pages, month over month.
That last detail mattered more than it seemed to at first.
Isolating the actual mechanism, not the usual suspects
Most traffic-drop investigations start with a mental checklist: core update, penalty, cannibalization, lost backlinks. Running through all of them on a large catalog site can burn a week before you learn anything. A faster approach is to separate the two things a traffic drop can actually be made of:
- Fewer pages earning any clicks at all, which usually points to an indexing problem
- Fewer clicks per page that’s still ranking, which usually points to a content or relevance problem
The team pulled landing-page sessions from Google Analytics and page counts from the Search Console Page Indexing report, then plotted both over the same timeline. The answer was immediate. Clicks per indexed page had barely moved. The number of pages actually eligible to rank had fallen off a cliff.
That single chart eliminated an entire category of hypotheses. This wasn’t a content-quality problem or a relevance mismatch. Pages that stayed indexed were performing exactly as before. The problem was that pages were disappearing from the index in the first place.
Ruling out the obvious culprit
The next stop was the URL Parameters report, checking for what’s known as an infinite crawl space, the trap where faceted navigation or filter combinations generate an effectively endless set of near-duplicate URLs that drain crawl budget and choke out indexing of the pages that matter. It’s a common cause of this exact symptom on e-commerce sites with large catalogs.
It wasn’t the cause here. Parameter counts were proportional to the actual number of product variants. No runaway URL space, no crawl trap. Which meant the answer was somewhere else.
The moment the bug became visible
The Page Indexing report showed a growing list of URLs marked Not found (404), and every single one of them was a page the team had personally redirect-mapped during the migration. Clicking through to a handful, expecting to confirm Google was simply behind on re-crawling, the pages loaded. Correct content, correct layout, correct price, no errors, nothing a shopper would ever notice.
The confirmation came from opening the Network tab in Chrome DevTools and checking the raw server response headers rather than the rendered page. The server was returning a 404 Not Found status code on pages that were serving fully functional content.
That’s the phantom 404: a page that is visibly, humanly fine, while the HTTP header underneath it is telling every crawler, bot, and monitoring tool on the internet that it doesn’t exist.
Why this is rare, and why it’s dangerous precisely because it’s rare
Most indexing problems announce themselves. A blocked robots.txt line, an accidental noindex tag, a broken redirect chain these are visible if you know where to look, and they tend to affect a predictable, traceable set of URLs.
A status code mismatch hides in a different layer entirely: the CDN edge configuration, an application server rule that fires only for specific URL patterns, or a caching layer that serves stale headers while the page body updates normally. None of it touches what a person sees in a browser. It only shows up in what a machine reads before the page even renders.
Google’s crawler treats a 404 as a strong, fast signal. According to Search Console documentation, Google gives a page roughly 24 hours of grace before treating a 404 as confirmed and dropping it from the index. On a large site, that grace period isn’t nearly enough time for a human to notice a pattern across hundreds of URLs, especially when every one of those URLs looks perfectly healthy in a browser tab.
That combination, invisible to users, fast-acting on crawlers, is what makes this bug rare in the literal sense. Most sites never encounter it. The ones that do usually don’t find it until an SEO deliberately checks raw response headers instead of trusting what renders on screen.
The fix, and the second bug it exposed
The retailer’s development team traced the mismatch to a caching rule introduced during the platform switch. Certain URL patterns, mostly product pages with recently changed inventory status, were being served through a code path that defaulted to a 404 header whenever a specific product-availability flag was momentarily null during a cache refresh. The content rendered correctly because the front end pulled live data. The header didn’t, because it was set earlier in the request cycle from a stale cache read.
Fixing it meant aligning the two: making sure the HTTP status code was generated from the same data source as the page content, not from a separate, faster, occasionally-stale cache layer.
While validating the fix, the team found a second, smaller issue riding along with the first. Several of the original 301 redirects from the old URL structure were stripping query parameters, the kind used for tracking paid campaigns and internal promotions, on the way to the new URLs. It wasn’t causing the indexing drop, but it was quietly corrupting attribution data in the marketing team’s paid search reporting. Both issues were resolved in the same development sprint.
What changed after the fix
Once the corrected status codes shipped, the team submitted the affected URLs for re-indexing through Search Console and waited for Google’s crawl cycle to catch up rather than trying to force it.
- Indexed page count returned to its pre-migration baseline within roughly three weeks
- Organic sessions climbed back past year-over-year parity by the following month, and ultimately settled above the prior year’s levels once the previously-locked pages resumed accumulating rankings
- Paid search attribution accuracy improved immediately once the parameter-stripping redirect issue was corrected, unrelated to organic traffic but a real business impact from the same investigation
The most telling number wasn’t the recovery percentage. It was the gap between when the bug started (immediately at launch) and when a human would have caught it without checking response headers directly (likely never, since every visible signal said the site was healthy).
How to check your own site for this before it costs you a quarter
This bug is rare enough that most SEO checklists don’t include it by default. It’s worth adding deliberately, especially around migrations, CDN changes, or new caching layers.
- Compare page counts to session-per-page trends. If indexed pages are dropping while performance per remaining page is stable, the problem is indexing, not content or relevance.
- Cross-reference Page Indexing errors against your actual site map. A 404 listed for a URL you know is live and correctly redirect-mapped is the flag, not a coincidence to dismiss.
- Check raw response headers, not rendered pages. Chrome DevTools’ Network tab, or a command-line tool like curl, will show you the actual status code Google receives. A page can look flawless and still be sending a 404.
- Re-check after any CDN, caching, or platform change, even ones that seem unrelated to SEO. Header-level bugs almost always trace back to an infrastructure layer, not a content or template change.
- Don’t stop at the first bug you find. The redirect parameter issue in this case was only caught because the team kept investigating after the primary fix was already identified.
The broader lesson sits underneath the technical detail. Search Console’s Page Indexing report is often treated as a monitoring dashboard, something to glance at for red numbers. Used as a diagnostic tool instead, cross-referenced against real user data and verified at the raw HTTP level, it can surface the exact category of bug that no amount of content strategy or link building would ever have caught.
