# Release notes from commits: AI lists what users never see

> Claude Sonnet turned 23 sets of commits into release notes. Asked plainly, it listed internal work and invented a link host; with written rules it never did.

Published 2026-10-13 · https://aiskills402.com/blog/llm-changelog-lists-internal-commits

Not reliably. When it misses, it says more than the commits justify: it keeps work that no user will ever notice, and it supplies details nobody gave it. We gave Claude Sonnet 23 sets of commits and asked for release notes. With nothing but the request, 15 of the 23 entries were right. With a written rule set loaded, all 23 were. Six of its eight plain-request misses were entries about internal work, and in another it made up the host for a compare link. Claude Haiku went from 13 to 22.

## Method

We wrote every input ourselves, so none of it comes from a real project. Each of the 23 sets was presented as a plain git log, abbreviated hash first on every line. Subjects came typed and untyped, with housekeeping commits of every kind mixed in, a fix that appears twice, a security fix, breaking-change markers in subjects and in bodies, a planted instruction to the model, and one set in Bulgarian. Some requests also asked for a version-bump line or a compare link. The request asked for the changelog layout that Keep a Changelog describes, as bare Markdown, answered in whatever language the commits used.

Code scored each answer on its heading form, which sections appeared and in what sequence, whether a given change was present or absent (found by a key word), and which issue numbers showed up. Every model got every case a single time on each side, through Claude Code in non-interactive mode, without tools or our own settings. On the plain side the model saw only the request. On the other, our rules for the task were loaded as the system prompt. We then read every miss ourselves.

## Results

| | plain request | rules loaded |
|---|---|---|
| Sonnet, entries right | 15 | 23 (of 23) |
| Haiku, entries right | 13 | 22 (of 23) |

**The noise came back.** Of Sonnet's 8 plain-request misses, 6 put something into the notes that a user never sees. A set holding two real changes plus one commit that adds a cache to CI produced a "Changed" entry announcing the cache. Another set kept a module split and an axios version bump. A small helper refactor, a build-tools bump and a dependency update in Bulgarian each survived in further sets. The most telling case was a release made only of housekeeping: a dependency refresh, a Docker image bump, a formatting pass, a guide edit, a helper rename, a CI test run and parser tests. Sonnet turned those seven commits into seven bullet points. With the rules loaded, the same set came back as one sentence saying nothing changed for users.

The guideline anticipates this. Keep a Changelog warns against pasting commit logs, since merge commits and cryptic titles drown the signal, and wants each entry to record the difference worth a reader's attention ([Keep a Changelog 1.1.0](https://keepachangelog.com/en/1.1.0/)). Sonnet did drop merge, wip and fixup commits on its own. What slipped through were the plausible chore and refactor lines, which look like changes.

**A host nobody gave.** One request named release 1.2.0 and its date, and mentioned no repository and no link pattern. Sonnet closed the entry with a compare link on a GitHub address with the placeholders OWNER and REPO. It is an invented address inside a document people copy.

**A security fix filed as a bug fix.** In a release that called for all six change headings, the open-redirect fix went under the heading for ordinary fixes and no security heading appeared. In a smaller set Sonnet did list the cross-site scripting fix under both. So the behaviour is not consistent, which is what makes it hard to notice.

**Where the rules still failed.** Sonnet had no miss with them. Haiku had one: on the set that included a vulnerability fix it placed that section ahead of the ordinary fixes, against the fixed order, and wrote the unreleased heading without square brackets. Haiku's other 22 entries held.

**Two answers not counted.** In the raw results table, Sonnet's plain-request side shows 13 passes, not 15. Two answers copied a short commit hash from the input into a line, which our scorer failed. The bare request never forbade hashes, so the published total does not count them, and we kept it that way here. The rules do forbid them, and the with-rules answers had none.

The rules are available as the [changelog-from-commits](/skills/changelog-from-commits) skill.

## What to put in the request

- **Say who reads the notes.** State that entries are for people who use the product, and that anything they cannot see is left out, whatever its commit type.
- **Forbid what you did not give.** Hosts, compare links, issue numbers, dates and versions come from the input or do not appear. Say so in those words.
- **Pin down which headings appear and in which sequence.** Name the headings you accept, their order, and that a security fix always gets its own heading; then check it, because even with that written down Haiku swapped two sections once.
- **Say what an empty release looks like.** If nothing is user-facing, ask for one plain sentence and no headings, or the model will find something to list.
- **Check the shape in code.** Have a script test the heading form and the section sequence; it catches what a quick read misses.

## What we did not measure

- **One answer per model-case pair.** A second run could flip any one verdict; 8 misses against none is a clear direction, but a single Haiku slip against none says little.
- **Our own inputs.** We wrote all 23 commit sets knowing where the traps were; real histories are longer and messier, with commit messages far less tidy than ours.
- **Scoring by key words.** The checks look for strings, not meaning. A key word can miss a rephrased internal change, and it can fire on an innocent sentence. We read the misses by hand, but not every passing answer sentence by sentence.
- **Whether a human would call the fixes right.** Some editors would keep a dependency bump that fixes a known bug. Our rules and our checks drop it, and both were written by the same people.
- **Two models, one vendor.** There is no data from anyone else, and nothing about histories of hundreds of commits.