Secret Handling Review: test results

Verdicts

Date

2026-10-29

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 3 cases, read by hand. On the forecast script it named the config object printed at start-up and the key placed in the query string, and left out the line that prints only that the key is present, its length and its last four characters, and the key read from the environment. On the shop it found the mail password typed into a committed file and the admin token under the public prefix in a client component, and left out the publishable payment key and the CAPTCHA site key under the same prefix. On the clean reports route it answered exactly No findings.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on all 3 cases, read by hand. It gave the same pairs of findings as Sonnet on the forecast script and on the shop, with the same safe look-alikes left out, and answered exactly No findings on the clean route. Two of its lines ended with a full stop and one answer was indented by four spaces, which the checks accept.

Note

Light test: three cases (two planted, one clean), one run per model; no comparison without the skill. The code was written by us with made-up values only; each planted case carries a safe look-alike (a presence, length and last-four print, a publishable key and a site key under the public prefix), and neither planted case has a comment pointing at the defect. A fifth code the brief suggested, a secret passed as a command-line argument, was dropped because no official page read on 29 October 2026 supports it. Facts come from OWASP cheat sheets and the Next.js, Vite and Stripe documentation, read on 29 October 2026; there are no owner-measured facts.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.