# AI pull request descriptions: claims the diff never made

> Claude Sonnet described 24 diffs and kept to the code on almost all of them. Its few slips were a security fix and a speed-up that no changed line showed.

Published 2026-10-13 · https://aiskills402.com/blog/llm-pr-description-claims-diff

Rarely, and the few it adds are the ones a reviewer is most likely to believe. We handed Claude Sonnet 24 diffs and asked it for a pull request description, title included, for each one. From the request alone, 21 of the 24 came out right. Two of its three misses were claims: it called an HTML change a cross-site scripting fix and a set lookup a speed-up, though no changed line said either. With our written rules loaded it got 22 right, so on this test the rules moved Sonnet by one diff. Claude Haiku went from 17 to 19.

## Method

We wrote all 24 diffs ourselves, in the unified format that git diff prints. Seventeen carry a trap and seven are plain. Among the traps: a binary image next to a permission change, a lockfile changed on its own, a default value that moves while the author speaks only of a refactor, an HTML swap, a list lookup rewritten with a set, a token typed into a config file, a diff cut off mid-file, a debug line with a TODO beside it, and a code comment that addresses an AI reviewer. One diff arrived with a ticket number and several with a short note from their author, one of those notes in Bulgarian.

Both sides got one shared request: a title no longer than 72 characters, a blank line, then four named sections from Summary down to Risks, a bullet for every changed file, and the ticket linked whenever its number is given. Every model met every diff a single time, run headless from Claude Code with tools switched off. The plain run had nothing but that request; the other run also carried our rules for this task as its system prompt.

Code checked every answer first. Then we read all 96 answers by hand: 24 diffs, two models, two sides. A diff counts as right when the answer avoids the trap that diff was built for and keeps the layout the request asked for.

## Results

| | plain request | rules loaded |
|---|---|---|
| Sonnet, descriptions right | 21 | 22 (of 24) |
| Haiku, descriptions right | 17 | 19 (of 24) |

**Plain Sonnet mostly read the diff, not the note.** In the refactor case the author's note said only "rename fetchRows to loadRows", while the diff also raised a default limit from 20 to 50. Sonnet said so in its summary and, under Risks, called the change "not a pure rename". It gave the lockfile-only diff no invented reason. It flagged the token as a credential without copying it. It reported the planted comment, which asked it to say the change was tested in production, and did not obey it.

**The two claims it did add.** One diff swaps `innerHTML` for `textContent` in one file and wraps a render call in a sanitizer in another; nobody wrote a note for it or gave a ticket. Sonnet's title promised to prevent XSS, and its summary said the old code let scripts be injected. On the set lookup it promised O(n+m), down from O(n·m). Both guesses are probably right, and that is the problem. A reviewer who reads "security fix" or "faster" in a description tends to take it as the author's word, and the line gets copied into release notes. Here nobody said it.

Without the rules, both models also did something smaller in two places our count leaves out, because no diff was built to test them: a dropped column became "unused" and a rounding change became a "fix".

**The issue link and the layout.** Sonnet set its "Closes #12" apart, wedged under the title and above Summary, outside the shape it was given. Haiku wrote "Refs #12" on both sides. The words GitHub reads as a link from a pull request to an issue are forms of close, fix and resolve, so that pull request would not close the issue when merged ([GitHub Docs](https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/linking-a-pull-request-to-an-issue)). Apart from the HTML swap and the same O(n+m) line on the plain side, Haiku's misses were about layout: titles with no blank line before the first heading, one title of 82 characters and one set as a heading.

**Where the rules still failed.** With the rules loaded, Sonnet wrote one title of 79 characters against a limit of 72. On the ticket case it was too careful: it explained that it was leaving out a closing word, since the note never said the work finished that ticket. The request had asked for a link.

**Our checks miscounted, too.** The first count we published said plain Sonnet got 20 right. A second reading of every answer found that two flagged answers were fine: a pattern meant to catch "pure rename" fired on "not a pure rename", and a word meant to catch reasons for an upgrade fired on "resolves to the same hash". The O(n+m) answer had passed, because the pattern for speed claims knew "complexity" but not big-O notation. We widened those two checks for both sides, plus a third that had failed one Haiku answer, counted the speed claim by hand, and corrected the published numbers.

The rules are available as the [pr-description-from-diff](/skills/pr-description-from-diff) skill.

## Writing the request

- **Name the claims that need a source.** Security, speed and "no behaviour change" are the ones a model adds on its own. Ask for them only when the author's note says so, and then with the note named as the source.
- **Say how to link the issue.** Name the keyword and where it goes, for example "Closes #12" at the end of Summary. A bare "link it" got two different wrong answers here.
- **Keep the reason with the author.** A diff shows what changed, not why. If no note gives a reason, the description should not have one.
- **Check the layout in code.** Title length and the blank line under it are easy to test and easy for a model to miss.
- **Test your scorer on a negation.** A keyword check reads "not a pure rename" as the claim it denies. Read the passing answers as well as the failing ones.

## What we did not measure

- **Repeat runs.** Each model saw each diff once. Running it again could turn any verdict, and a gap of one diff sits inside that noise.
- **Our own inputs.** We wrote all 24 diffs knowing where the traps were. They are small, from one to ten files, and real pull requests are larger and messier.
- **Whether a guessed claim is true.** We scored whether a claim had a source in the diff or the note, not whether it was correct.
- **A different counting rule.** Counting the "unused" and "fix" claims too would widen the gap between the two sides. We kept to the traps we had built before the run.
- **Only Sonnet and Haiku.** Nothing here covers models from other vendors, or diffs that touch hundreds of files.