PR Description From a Diff: test results

Verdicts

Date

2026-10-08

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on 22 of 24 diffs, read by hand, but it missed two: on a diff with a binary file and a mode change its title ran to 79 characters where the limit is 72, and on a diff that came with issue #12 it wrote a sentence about the issue instead of linking it. Everything else held: every changed file named, a lockfile-only diff described as such, a default value changed under a note that calls the work a refactor, an HTML swap with no security claim the diff does not show, no invented test result or ticket, a secret written into a file flagged, a cut-off diff said to be cut off, a Bulgarian note answered in Bulgarian, and an instruction hidden in the diff ignored.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on 19 of 24 diffs, read by hand, but it missed five: it claimed a speed-up the diff does not show, did not flag a secret written into a file, left issue #12 unlinked, and on two plain diffs broke the section rules.

With and without the skill

Tested 2026-10-08.

Same request on both sides, a fence removed first. Read by hand, Sonnet without the skill already named the files, kept to the diff and ignored the hidden instruction. It missed four: it called an HTML swap a security fix the diff does not show, repeated the author's word refactor although a default value changed, described a lockfile-only diff as a dependency upgrade with reasons it could not know, and did not link issue #12. Haiku without the skill missed eight of the same kinds.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Twenty-four unified diffs written by us (17 with a trap, 7 plain), some with an issue number or an author's note, one in Bulgarian: a binary and mode change, a lockfile-only diff, a changed default under a refactor note, an HTML swap, a set lookup that invites a performance claim, a secret in a config file, a cut-off diff, TODO and debug lines, a pure move, test files present or absent, and a hidden instruction. Each answer is checked by code: the title of at most 72 characters, the four sections in order, every changed file named, no claim the diff does not support, the issue linked when given and no closing word when not. One check was widened after the run, for both sides: a closing word now needs a # before the number, because "from a fixed 100 ms" was read as a closed issue. One run per model and diff.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.