Soft 404 Fix: test results

Verdicts

Date

2026-11-05

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 5 cases, read by hand. On a Netlify single-page app it named the wildcard rewrite to index.html, kept the API proxy and the 301, and rewrote only the real routes with a 404.html; on a Next.js site it found that loading.tsx starts the stream before notFound() runs and left the manuals route alone; on an Express server it named the catch-all that sends the shell and added a last 404 handler; on an Apache site it named the ErrorDocument with a full URL that turns into a redirect. On the clean Workers setup with a 404 page it answered exactly No findings.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on 4 of 5 cases by the skill's own format checks, read by hand, but it missed the one-line format on the Express case: the cause, the fix and the real routes were correct, yet it added a code block and two paragraphs after the finding line. It named the right layer on the Netlify, Next.js and Apache cases and answered exactly No findings on the clean Workers setup. On the Next.js case its Verify line named the not-found heading instead of the home page; the check was widened to accept that.

With and without the skill

Tested 2026-11-05.

Scored on content only, with a fence removed first. Read by hand, Sonnet without the skill already named every layer, kept the real routes, wrote the control that a real page still answers 200 and gave a clean answer on the Workers setup; it also added useful extras such as missing assets falling into the same catch-all. Haiku without the skill did the same on all five. So on these five cases the skill did not add content for either model; what it adds is a fixed code per cause, one line per finding, a Verify line and the exact sentence for a clean input, which the side without it does not give.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Five cases written by us (four planted, one clean): a Netlify single-page app, a Next.js site with a loading file, an Express server, an Apache site and a clean Workers setup. Each planted case gives only what the owner has before the diagnosis (a report line or a Search Console notice, the hosting name, the config) and carries one safe look-alike (an API proxy rewrite, a manuals route that calls notFound before streaming, an API router ending in a 404, a local-path error page and a specific 301). Each answer is scored by code for the skill's format, and the same answers are scored for content only (cause named, fix keeps the real routes, a control that a real route still answers 200) so that the side without the skill can be compared. Facts were read on 6 November 2026 from the public documentation of each host and framework; there are no owner-measured facts. Two checks were widened after the run for both sides: the 404-body element of the Verify line also accepts a not-found page or heading instead of the home page, and a real-route control phrased as 'should still be 200' counts; one forbidden-pattern check no longer matches the word change. One run per model and case.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.