Internal Link Orphan Audit

Decides for every URL variant of a page whether it is the canonical URL, gets a canonical tag to another address, a permanent redirect, a noindex, a 404 or 410, or cannot be settled from the facts given, answered as JSON only with the target and a short reason per URL. Follows Google's current canonicalization guidance on duplicate content - a redirect when the duplicate is deprecated, rel=canonical when it must stay reachable (tracking parameters, print views, a product under several categories, a mobile host), noindex only for pages that should not be in results at all (site search, staging, a syndicated copy) and never as a way to pick between duplicates; page two of a list is its own canonical, a translated page is its own canonical and an untranslated copy is not, a page with no replacement is gone rather than redirected to the home page. Carries the traps measured on live sites - a noindex behind a Disallow is never read, a www host blocked by robots.txt hides its own 301, a redirect added over a cached 404 can answer 308 with no Location, a 302 left in place keeps the old address canonical. Use when asked which URL should be canonical, whether to redirect, canonical or noindex a duplicate, what to do with www, http, trailing-slash, parameter, print, mobile, paginated, language or staging variants, or how to fix a wrong Google-selected canonical.

Turns an old-URL list and the mappings you have into a redirect map as JSON - one rule per old URL, chains collapsed to the final target, loops reported and never emitted, queries and percent-encoding kept as written, and old URLs with no named target listed as unmapped instead of guessed. Also writes the same map as a Cloudflare Pages style _redirects file. Knows the status codes (301 for a moved page, 308 where the method must be kept and what Next.js really sends for permanent true) and the cache trap, where a redirect added over a cached 404 keeps serving a 308 with no Location until the path is purged. Use when asked to plan or write redirects for a site migration, a redesign, a URL rename or a domain change, or to check a redirect list for chains and loops.

Turns a crawler policy written in words into a correct robots.txt, answered as the file only. Knows the mistakes that make a file do the opposite of what was meant - a crawler obeys only the one group that names it, so a GPTBot group silently drops every rule written for the star group and the private paths must be repeated; longest match wins and Allow wins a tie; Google ignores Crawl-delay; Host is a Yandex-only line; Sitemap must be an absolute address; Google-Extended is a control token for Gemini training and does not touch Google Search; training, search and user-triggered AI bots are separate tokens. Says in a NOTE comment what robots.txt cannot do, such as removing a page from Google. Use when asked to write, check or fix a robots.txt, to allow or block search engines, AI crawlers or AI training bots, or to keep private paths out of crawlers.

Reads what the owner of a site on Cloudflare Workers (usually Next.js through OpenNext) pasted - Search Console numbers, an audit of status codes, open-next.config.ts, wrangler.jsonc, data-fetch code, robots, sitemap and metadata code, wrangler tail lines, the owner's plan - and finds what stops Google from crawling, indexing or showing the pages, in a fixed diagnosis order. Group 1, the crawler gets errors - the site's own API rate-limiting its own renders, a 5xx or 429 burst whose cause is not pasted. Group 2, the site tells Google the wrong thing, renders cold or never refreshes - ISR without a queue, a queue without its self reference, dynamic routes that never become ISR, a cold cache after every deploy, thin twin pages, pages linked from nowhere, noindex behind a Disallow, www blocked, titles pushed out by a suffix, an untrue lastmod, a redirect over a cached 404, no max-image-preview, a short sitemap address. Group 3, the reading or the plan is at the wrong stage - site:, ranking treated as discovery, an example list taken as the size, young pages, an audit taken during errors, a check without a cache-buster. Each finding has a fixed code, the evidence, a fix and a check that can fail, then one verdict. Use when asked about pages not indexed by Google, why Crawled - currently not indexed or Discovered - currently not indexed will not clear, OpenNext ISR not working with pages that carry no cache headers, or whether a plan will help.

Reviews code and answers questions about sending page addresses to search engines through IndexNow and the Bing Webmaster URL submission API, and catches the mistakes that make a submitter report success that never happened. It flags a Bing call judged only by its HTTP status when the refusal sits inside the body, a retired XML or SOAP endpoint, a fallback chain that retries after a refusal that is ours, a quota read as one number instead of two, an unread quota printed as zero, a report that does not say which engine accepted, a 200 or 202 read as proof of indexing, a key file that cannot validate, and batches over the limits. For a question it answers in the JSON shape asked. Use to review a URL submission function, work out how many addresses may be sent, or explain what a submission status code means.

Reads the numbers from Cloudflare Web Analytics and zone analytics (GraphQL or the dashboard) and says how many of them are real readers or real server errors, answered as JSON with the computed figures and a short reason. Knows what pollutes the table - the owner's own headless-browser checks, link-preview fetches from the social network that show up as referrals, the site's own Worker and its cache calls counted as visitors or as 5xx errors, the regional cache host of OpenNext, staging sharing a token - and what makes a true zero - the EU exclusion setting, a content security policy that allows the script but not the measurement request, sampling of data older than seven days. Use when asked how much traffic a site really has, whether a spike or an error count is real, why Web Analytics shows zero, or how to query per-address detail on the Free plan.

Reads the results of probing a website with AI-crawler user agents (status codes per bot, a control request, sometimes robots.txt or the site's settings) and says what they really show as JSON - whether training bots, search bots and user-triggered agents are open, blocked, mixed or unknown, what is behind a refusal, and whether the robots.txt that answered is the site's own file. Knows the traps - a bare bot name proves nothing and lies differently for each bot, a blocked training bot is a choice while a blocked search bot is a defect, Python-urllib and libwww-perl are refused by Browser Integrity Check and not by the site's policy, since 15 September 2026 Cloudflare refuses training bots on zones nobody touched, and a robots.txt of only comments is a stand-in from the edge. Use when asked whether a site blocks ChatGPT, Claude, Perplexity or Google AI, how to read curl results with bot user agents, or why an AI crawler gets a 403.

Writes an llms.txt file in the shape the llms.txt proposal defines, from a site description and a list of pages, answered as the Markdown file only. One H1 with the site name, a blockquote summary, optional plain notes, then H2 sections that hold lists of links in the form "- [name](address): notes". Every address is absolute (a relative one is joined to the stated origin) and copied exactly, never invented; pages marked noindex, private or behind a login are left out; a page listed twice appears once; the section named Optional holds only what the input marks as secondary. Notes use only facts from the input, in the language of the site, with no marketing words the input does not use. Use when asked to write, generate, fix or check an llms.txt for a site, a documentation area or a shop.