A site can answer every single request correctly and still fail on half its pages the moment something asks for many at once. On our own site the cause was ordinary: product pages pulled their data from an API that the same site publishes, and whose rate limit was written for outside callers, and a burst of page renders used it up. A one-by-one check never gets there. Only a parallel audit, plus a look at the live Worker log while it runs, shows it.

Method

The starting point was a Search Console screen on 18 October 2026. It listed 77 indexed pages and 51 not indexed. Of the 51, 32 were "Crawled - currently not indexed" and 8 were "Discovered - currently not indexed"; the rest were pages with a noindex tag, a redirect, a 404 or a correct alternate. The sitemap had been read with 307 addresses found, while 330 were live. Over 28 days the site had 0 clicks and 121 impressions at an average position of 19.9. Not one reason in that list was a server error.

We audited every one of the 330 addresses, three requests in parallel, and read each answer. Then we asked for the failing pages again, one at a time and twelve at a time. During a repeated burst we watched the Worker's live log with wrangler tail.

Results

The audit that found the 500s

Of 330 addresses, 162 answered 500. They were product pages and their test pages only; none of the 59 articles failed. The errors started after roughly the 115th request and got denser. Single requests, and twelve in parallel right afterwards, all returned 200: a check made page by page would have called the site healthy.

What the Worker log said

In a repeated burst, 56 of 240 requests failed. Every failure was the page's data function throwing because the catalog API had answered 429. The site fetched its catalog from the API hostname that outsiders use as well, and a single product page needed between two and three calls. That hostname limits callers per address.

We cannot say which address the API saw. Cloudflare describes two situations: a Worker call that stays inside its own zone reaches the target with the address of the original visitor, while a call crossing into a different zone reaches it with one fixed address. We did not record which situation applied to us. Whichever it was, a limit built for outside callers was being drained by page renders of the site itself. If it was the first, the bucket belongs to whoever is asking, an auditor or a crawler, and every page fetched costs that visitor between two and three calls to the API.

A second fault in the same log

Another line in the same log, FatalError: Dummy queue is not implemented, came from the placeholder queue, which refuses every send. The OpenNext config named an incremental cache but no queue, so an expired ISR page had no way to be rebuilt behind the scenes. The OpenNext page says time-based revalidation needs one. Our product pages went out with no cache-control header, even though the route had revalidate and generateStaticParams.

Orphans that were not orphans

The internal-link audit of that same run reported 76 test pages with no inbound links. After the fix, the same audit reported 0. Those pages had always been linked; the page that linked to them had answered 500, so the audit never saw the link. An audit run while a site errors invents findings of its own.

The fix, deployed the same day

  • The site now identifies itself to its own API with a secret header and is counted in a separate bucket. A wrong key is treated like no key: it falls into the shared outside limit, with no error of its own.
  • A Durable Object queue was added to the config, together with a service binding that points the Worker at itself. Without that binding the queue handler threw. Our written plan had left it out, and only the deploy exposed it.
  • The regional cache was set to its long-lived mode.
  • A script now warms the cache after every deploy. Each deploy gets a new build id, so it begins with nothing cached, and one week brought 29 builds.
  • 120 test pages that only repeated part of the product page became noindex,follow and left the sitemap; their links stayed.

After the fix

The warm-up fetched 330 of 330 addresses with 200 in 201 seconds. A product page went STALE and then HIT, with s-maxage=298. A burst of 240 product pages, three at a time, gave 0 errors, where the same burst before the fix gave between 56 and 127. The Worker log showed no more 429 from the API and no more dummy-queue errors.

The Workers Rate Limiting binding keeps its counters per location and is documented as approximate, so a test that waits for the first request over the limit to be refused passes or fails by luck. We checked the key instead: 80 requests with a wrong key still met 4 refusals of 429, which the site's own bucket would not give at that volume. 80 requests with no key met none, though they go to the same public limit; the count depends on which location answers.

Would a model have found it?

We turned the diagnosis order into a skill and tested Claude Sonnet and Claude Haiku with and without it, one run per model and situation, scored by hand.

Set one had 35 situations, 30 with a fault and 5 sound, each with the telling evidence pasted. Sonnet scored 35 of 35 with the skill and 35 of 35 without it. Haiku scored 35 of 35 with it and 32 of 35 without.

Set two had 11 situations, written before any run, that begin where our incident began. Each holds just the material available on day one: Search Console counts, a status audit, the config, and a plan of next steps, and nothing that points at the culprit. Both sides had to put the main cause at the top, and a string of maybes ahead of it earned no credit. Sonnet scored 11 of 11 with the skill and 9 of 11 without. Haiku scored 11 of 11 and 8 of 11.

Here is where bare Sonnet slipped. Facing ISR without a queue, which was our real incident, it opened with a generateStaticParams that returns an empty list. On this stack that empty list is what makes the route ISR in the first place: without it every request renders. Shown a robots.txt read soon after a deploy, it claimed a .txt file escapes the cache, yet Cloudflare's own page says robots.txt is cached unless you change that. Bare Haiku overlooked the queue, the missing self reference and the robots.txt cache. With the skill, a Sonnet answer ran near 300 words; without it, near 1,500.

Two things need saying plainly. After the first run we moved two problem codes out of the group where the crawler sees errors and into the group that needs repair before anyone leans on it. The reason was that facts inside the skill backed Sonnet: OpenNext catches the failed queue send, so the crawler is handed the stale page with a 200, not an error. Those situations, and the sound ones, were run again. Separately, the format check built into the skill rejected three answers from each model: for example a misspelled code, a finding with no check attached, or a ranking line nobody needed.

The mechanical scoring erred in both directions. Four faults in it are written down and were deliberately left unfixed, so the instrument is not tuned to the answers. That is why every count here was read by hand.

The honest conclusion: when the evidence is pasted, Sonnet does not need the skill. Its gain appears only at the hard start, 2 of 11. Haiku gained 3 of 35 with the evidence pasted and 3 of 11 at the hard start. Every count comes from one run.

What we did not measure

  • Whether Google noticed. Search Console listed no server errors, and the fix went live the same day, so we have not seen Google's reaction. We do not claim the 500s caused the unindexed pages, and we do not claim indexing improved. Google says a persistent server error slows the crawl; we have no record of what its crawler received.
  • Which address the API saw. We did not record the zone case described above.
  • The noindex decision. We hold no index counts from before and after for the 120 pages. Google prefers a canonical link for true duplicates, and our pages were a subset, not copies.
  • Our incident on both sides. The skill and the hard set were both written from the same day's work. A skill built from an incident has an easy time with situations drawn from that incident; cases from someone else's site could show a smaller gain. Only two models, both from one vendor, were tested.
  • One site, one run per case. All 46 situations are ours, with one result per model and situation, so a single answer could move on another run.
  • The sitemap gap. Why 307 addresses were read out of 330 is still unexplained.

Read this post as Markdown: /blog/google-indexing-cloudflare-500s-under-load.md · Atom feed.