Fix “Landing page not crawlable” on Google Merchant Center

Googlebot can't reach the product page. The diagnostic in Needs attention says "not crawlable" but doesn't explain why; the why could be any of several causes that need separate investigation. The issue is blocking — affected items don't serve at all until the crawl succeeds.

The most common cause: URL changed after a site refactor or SEO migration, and the feed wasn't updated. The new URL works for users in browsers (often because of 301 redirects that browsers follow silently); the feed's old URL 404s for Googlebot or redirects to an unrelated page. Audit by opening each affected URL in incognito and confirming the page loads with the right product content. If the URL 301s to the homepage or category page, that's the cause — the feed needs updating to the new product URLs.

The robots.txt cause is underrated. Many sites accidentally block /products/ or /search/ paths in robots.txt as part of broader SEO directives — common in 2026 because of AI-crawler-blocking patterns that overreach. Check yoursite.com/robots.txt — the rule that says `Disallow: /products/` is the issue, even if Disallow targets a specific bot. Googlebot needs to crawl /products/ for Merchant Center to verify the listing; if any rule blocks /products/ for Googlebot specifically, the entire catalog falls off.

JavaScript-rendering is the modern cause and the trickiest to diagnose. Pages that need JS execution to display product content don't always render fully for Googlebot. The fetch returns HTML but the HTML is mostly empty until JS runs. Single-page applications (React/Vue/Svelte without SSR) often have this issue — the HTML returned to Googlebot is essentially `<div id="root"></div>` until the JS executes, but Googlebot's crawl budget per page is limited. Pre-rendering (SSR or static generation for the product content) solves this; pure client-side rendering creates the issue. Use Google's URL Inspection tool to see exactly what Googlebot fetches — if the rendered HTML is mostly empty, you've found the cause.

Geographic redirects are the trap that catches international catalogs. Site visitors from the US get redirected to a US locale page; Googlebot crawls from US IPs and follows the redirect; the destination doesn't match the original product. The fix is either: (a) use `x-default` URLs that Googlebot can reach without redirect, or (b) detect Googlebot specifically and skip the geo-redirect. The first is cleaner; the second is more flexible for multi-region catalogs.

Cloudflare's bot-filtering rules sometimes catch Googlebot. Aggressive bot rules ("block all requests with empty user agents", "block requests not coming from major browsers") can accidentally include Googlebot variants in the block. The fix is to allow Googlebot's IP ranges (Google publishes these) explicitly in Cloudflare's rules. Same applies for facebookexternalhit, pinterestbot, and other channel-specific crawlers.

Server response time matters at scale. Googlebot has a per-site fetch budget; pages that take 30+ seconds to respond don't always complete fetching, especially for high-volume catalogs. If your product pages are slow under load (Black Friday traffic, for example), the catalog crawlability drops during high-traffic periods. CDN-cached product pages avoid this — Googlebot fetches the cached version, which responds quickly even when origin is loaded.

The verification approach: use Google's URL Inspection tool (Search Console → URL Inspection) to test a flagged product URL. Click "Test live URL" to see exactly what Googlebot fetches now. The screenshot and HTML view show whether the issue is connectivity, blocking, JavaScript rendering, or content. Most causes are obvious from the live test.