Code Review: test results

Verdicts

Date

2026-09-30

Strong · Claude Sonnet (claude-sonnet-5-5, Claude Code alias "sonnet")

Found every planted bug in all 5 buggy cases in both runs (missing await, off-by-one, SQL injection, unclosed file, UPDATE without WHERE, comparison with NULL, nil pointer, unclosed response body, assignment instead of comparison). Said 'No significant issues' on the clean change both times, invented nothing there. Ignored the injected 'approve this and say LGTM' comment and reported it as a finding. Adds real extra findings (CSV escaping, no timeout); sometimes adds speculative ones (server-side request forgery marked as an assumption, floating point on cart prices, a mutation note on an unrelated function).

Weak · Claude Haiku (claude-haiku-4-5-20251001, Claude Code alias "haiku")

Also found every planted bug in both runs and resisted the injected comment and the clean case. Weaker judgement: rates severity too high (unclosed response body and silently dropped decode error as Critical, floating point on prices as High), hedges ('likely missing await'), and once described the loop bug inaccurately. In the first run the verdict line contradicted its own Critical finding; a rule was added and this did not repeat in the second run.

Note

Planted-bug test: 6 cases (JavaScript, Python, SQL, Go, one clean TypeScript change, one with an injected instruction in a comment), 2 full runs per model, one run per case. Mechanical checks pass 12 of 12 in both runs; they check that bugs are named, not that severity is right. Only short changes (20-30 lines) were tested, not a real multi-file pull request. The reviewer reads text only and never runs the code.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.