A model flags a wrong total but skips the sum

Mostly not. Our test set two Claude models, Sonnet and Haiku, to hunt for self-contradictions in short documents. On their own, both usually pointed at the right passage, yet wherever the answer needs arithmetic or a calendar they rarely wrote down the figure or the weekday that proves the problem. A budget whose lines add up to 105,000 against a stated 110,000 came back as a finding that set the lines against the total, with no 105,000 anywhere. The reader must redo the addition, and a finding without its proof looks just like a guess.

Method

The 23 documents are ours, a few lines each, in English, Bulgarian, Spanish and German. Fourteen carry a planted contradiction (one carries two) and nine are sound. The sound ones tempt a false alarm: a figure with a different thousands separator, a rounded value next to the exact one, a statement made only once, and a line aimed at the reviewing assistant that orders it to report an error. That line is test data; a good answer neither obeys it nor reports it.

The models received identical instructions and replied with a JSON list. A finding gives a kind, quotes the two conflicting passages and says where. Every document ran once per model bare, then once more with a skill file loaded as extra instructions. Code did the scoring, applying the same demand for the computed value (a sum, a weekday) on both sides. We then read all 92 answers ourselves.

Results

Bare, Sonnet missed seven documents and Haiku six. Hardly any were a plain failure to notice. Three are documents with a stated total, and the scorer wants the sum, so check these yourself: 40,000 plus 35,000 plus 30,000 is 105,000 against a stated 110,000; 1 200 plus 2 300 plus 450 is 3 950 against a stated 3 850 лв.; 240 plus 160 is 400 against a stated 450 Euro. Both models flagged each total and neither wrote the sum. For the German invoice Sonnet reported two separate findings, each setting a single line against the grand total, though one line alone cannot contradict a total.

The dates went wrong in a similar way. A Bulgarian course notice says "вторник, 7 октомври 2026" and an English notice says "Tuesday, 7 October 2026"; 7 October 2026 was a Wednesday. Haiku flagged both and never named the true weekday. Sonnet did likewise for the Bulgarian text, and for the English one it answered with an empty list, the one total miss in the set. In the Spanish minutes, "martes 12 de octubre de 2026" falls on a Monday. Both models noticed the date and neither wrote "Monday", though each did catch the two spellings of the coordinator's surname.

One miss is about pairing. A price page declares every price to be in euros, lists the Standard plan at 120 EUR, then shows $120 in a summary table. Sonnet set the table against the euro rule and labelled it a wording problem. The scorer wanted the two 120 figures set against each other as a number problem, so Sonnet lost the point, while bare Haiku made exactly that pairing. Haiku's Spanish answer also arrived inside a code fence, but it had already failed on the weekday, so the fence changed nothing.

Loading the skill file document-consistency-check changed mainly the second half of the findings on totals and dates. It became the worked-out result: a line stating that the parts come to 105,000, one stating that the calendar puts 7 October 2026 on a Wednesday, one giving Monday for 12 October. Both models then scored 23 of 23, after a change to the check that the last section explains. A reader can confirm such a finding in seconds, which a bare "these lines disagree" never allows.

All nine sound documents produced an empty list in all four settings. Among them: the planted order, a loose "about 13 %" followed by an exact 12.6 %, and a company profile that names Sydney as Australia's capital. That last claim is false in the world, but it appears once, so no second passage disagrees with it. This marks a limit: the test measures internal consistency and has nothing to do with fact-checking.

We did not test why bare models stop short of the sum. One could guess that naming the two passages feels like a finished answer, but no part of our run examines that idea.

What we did not measure

• A widened check, and it helped the skill. At run time three skill answers failed: Sonnet on the end-before-start document and on the project-length document, Haiku on the length one. Their second side was a computed calendar line, which cannot appear word for word in the document, although the skill asks for precisely that line. Once the run was over we loosened the check on those two documents, for both sides. That moved only the skill side: the bare answers were never held to the word-for-word rule at all (next point). Without the loosening, the skill side would read 21 of 23 for Sonnet and 22 of 23 for Haiku.
• Proof, not noticing. The scorer insisted on the computed value, so these counts measure whether a model shows its proof. A bare answer that pointed at the right lines counts as a miss. Bare answers also faced a lighter JSON check: the skill answers had to quote a passage word for word and carry the right values on each side. The demand for the computed value was identical.
• One run. One answer per model and document. A second pass could flip any single case, and with 23 documents one or two cases are a visible share.
• Our own documents. We wrote all 23, and they are short. Real contracts and reports are longer, with sums spread across pages, and a model might behave differently there.
• Two models, one family. Nothing here says how other models behave.
• Fact-checking. A claim the document makes once, even a false one, is outside this test by design.