Soft 404 Fix

Turns an old-URL list and the mappings you have into a redirect map as JSON - one rule per old URL, chains collapsed to the final target, loops reported and never emitted, queries and percent-encoding kept as written, and old URLs with no named target listed as unmapped instead of guessed. Also writes the same map as a Cloudflare Pages style _redirects file. Knows the status codes (301 for a moved page, 308 where the method must be kept and what Next.js really sends for permanent true) and the cache trap, where a redirect added over a cached 404 keeps serving a 308 with no Location until the path is purged. Use when asked to plan or write redirects for a site migration, a redesign, a URL rename or a domain change, or to check a redirect list for chains and loops.

Decides for every URL variant of a page whether it is the canonical URL, gets a canonical tag to another address, a permanent redirect, a noindex, a 404 or 410, or cannot be settled from the facts given, answered as JSON only with the target and a short reason per URL. Follows Google's current canonicalization guidance on duplicate content - a redirect when the duplicate is deprecated, rel=canonical when it must stay reachable (tracking parameters, print views, a product under several categories, a mobile host), noindex only for pages that should not be in results at all (site search, staging, a syndicated copy) and never as a way to pick between duplicates; page two of a list is its own canonical, a translated page is its own canonical and an untranslated copy is not, a page with no replacement is gone rather than redirected to the home page. Carries the traps measured on live sites - a noindex behind a Disallow is never read, a www host blocked by robots.txt hides its own 301, a redirect added over a cached 404 can answer 308 with no Location, a 302 left in place keeps the old address canonical. Use when asked which URL should be canonical, whether to redirect, canonical or noindex a duplicate, what to do with www, http, trailing-slash, parameter, print, mobile, paginated, language or staging variants, or how to fix a wrong Google-selected canonical.

Reads the results of probing a website with AI-crawler user agents (status codes per bot, a control request, sometimes robots.txt or the site's settings) and says what they really show as JSON - whether training bots, search bots and user-triggered agents are open, blocked, mixed or unknown, what is behind a refusal, and whether the robots.txt that answered is the site's own file. Knows the traps - a bare bot name proves nothing and lies differently for each bot, a blocked training bot is a choice while a blocked search bot is a defect, Python-urllib and libwww-perl are refused by Browser Integrity Check and not by the site's policy, since 15 September 2026 Cloudflare refuses training bots on zones nobody touched, and a robots.txt of only comments is a stand-in from the edge. Use when asked whether a site blocks ChatGPT, Claude, Perplexity or Google AI, how to read curl results with bot user agents, or why an AI crawler gets a 403.

Reviews pasted Next.js App Router code and config that runs on Cloudflare through OpenNext, for caching and ISR mistakes that serve a wrong or stale answer, or make every visit read the database. It flags an unstable_cache key whose cached shape changed while the key stayed, force-dynamic pages that disable the browser back-forward cache, a wildcard Cache-Control override that makes the whole site dynamic, a dynamic route that never starts ISR because generateStaticParams is missing, a static page reading D1 at build time, a writer that evicts a tag on every event, s-maxage trusted to cap database reads, whole pages cached instead of the data, a redirect added over a path that was cached as 404, a middleware matcher that skips robots.txt and sitemap files, and a clock stored inside cached data. Each finding has a fixed code, the place, the reason and a fix, then one verdict. Use to review Next.js caching, ISR and revalidation code, a next.config headers or redirects block, or middleware before deploying to Cloudflare.

Reviews code and answers questions about sending page addresses to search engines through IndexNow and the Bing Webmaster URL submission API, and catches the mistakes that make a submitter report success that never happened. It flags a Bing call judged only by its HTTP status when the refusal sits inside the body, a retired XML or SOAP endpoint, a fallback chain that retries after a refusal that is ours, a quota read as one number instead of two, an unread quota printed as zero, a report that does not say which engine accepted, a 200 or 202 read as proof of indexing, a key file that cannot validate, and batches over the limits. For a question it answers in the JSON shape asked. Use to review a URL submission function, work out how many addresses may be sent, or explain what a submission status code means.

Reads a crawl export of a website (one row per page with its status, its robots setting and the links in its body, or a plain list of links from page to page), the sitemap list and the navigation and footer links, and reports the internal-link problems that keep pages from being found and indexed - orphan pages with no inbound link from another page, pages with a single contextual inbound link or links only from the footer, pages too many clicks from the home page or not reachable at all, links that point at a redirect or at a removed page, noindex pages listed in the sitemap, and a crawl that failed on some pages so that the orphan verdicts are unknown. Each finding has a fixed code, the address, the reason and a fix, worst first, or exactly No findings. Use for an internal link audit, to find orphan pages, to check pages with no inbound links before publishing, to check click depth, or to audit a blog or a catalog from a crawl export.

Reads what the owner of a site on Cloudflare Workers (usually Next.js through OpenNext) pasted - Search Console numbers, an audit of status codes, open-next.config.ts, wrangler.jsonc, data-fetch code, robots, sitemap and metadata code, wrangler tail lines, the owner's plan - and finds what stops Google from crawling, indexing or showing the pages, in a fixed diagnosis order. Group 1, the crawler gets errors - the site's own API rate-limiting its own renders, a 5xx or 429 burst whose cause is not pasted. Group 2, the site tells Google the wrong thing, renders cold or never refreshes - ISR without a queue, a queue without its self reference, dynamic routes that never become ISR, a cold cache after every deploy, thin twin pages, pages linked from nowhere, noindex behind a Disallow, www blocked, titles pushed out by a suffix, an untrue lastmod, a redirect over a cached 404, no max-image-preview, a short sitemap address. Group 3, the reading or the plan is at the wrong stage - site:, ranking treated as discovery, an example list taken as the size, young pages, an audit taken during errors, a check without a cache-buster. Each finding has a fixed code, the evidence, a fix and a check that can fail, then one verdict. Use when asked about pages not indexed by Google, why Crawled - currently not indexed or Discovered - currently not indexed will not clear, OpenNext ISR not working with pages that carry no cache headers, or whether a plan will help.

Reads the numbers from Cloudflare Web Analytics and zone analytics (GraphQL or the dashboard) and says how many of them are real readers or real server errors, answered as JSON with the computed figures and a short reason. Knows what pollutes the table - the owner's own headless-browser checks, link-preview fetches from the social network that show up as referrals, the site's own Worker and its cache calls counted as visitors or as 5xx errors, the regional cache host of OpenNext, staging sharing a token - and what makes a true zero - the EU exclusion setting, a content security policy that allows the script but not the measurement request, sampling of data older than seven days. Use when asked how much traffic a site really has, whether a spike or an error count is real, why Web Analytics shows zero, or how to query per-address detail on the Free plan.