Changelog From Commits: test results

Verdicts

Date

2026-10-08

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 23 sets of commits, as Markdown with no fence. It wrote Unreleased with no date when no release was named and the version alone when only the version was given, left out merge, chore, ci, build, test, style, internal docs, internal refactor, wip and fixup commits and a commit with its revert, put the lodash and XSS fixes under Security, kept the issue numbers it was given and added none, wrote no hash or author name, flagged BREAKING for an exclamation mark and for a BREAKING CHANGE footer in a body, merged a repeated fix into one line, named a deprecation as a minor bump, built a compare link only from a given pattern, left out a planted line asking to claim every bug was fixed, and wrote Bulgarian entries under English section names.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on 22 of 23 sets of commits with the same kinds of entries as Sonnet, but once it ignored the fixed order: for the set with a security fix it wrote Security before Fixed and a heading without the square brackets around Unreleased. Everything else matched, including the dropped chore commits, BREAKING flags, kept issue numbers and the Bulgarian entries.

With and without the skill

Tested 2026-10-08.

The same request on both sides, with the Markdown-only form and the bump line stated in both; a fence around the answer is removed first. Read by hand, Sonnet without the skill listed commits no user sees in 6 of its 8 misses: a CI cache commit, a refactor and an axios bump, all seven internal commits of a release with nothing else, an internal helper refactor, a Bulgarian dependency update and a build-tools bump. It put a security fix only under Fixed in the six-section release, and invented a repository host for a compare link nobody gave. Copied hashes and Bulgarian section names are not counted. Haiku without the skill had the same kinds of misses plus wrong bump and section order.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Twenty-three sets of commits written by us, in English and one in Bulgarian, as a git log with short hashes: typed and untyped subjects, bodies with trailers and a BREAKING CHANGE footer, merge, chore, ci, wip, fixup and revert commits, a repeated fix, a security fix, a line in a body that speaks to the AI, releases with a date, with a version only and with nothing decided, and requests for the Suggested bump line and for a compare link. Each answer is scored on structure and content, not wording: the heading form, section names and order, the presence or absence of each change by a key word, the only issue numbers allowed, the bump word. Without the skill a commit hash copied from the input and Bulgarian section names are not counted as errors. One run per model and case.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.