Document Consistency Check: test results

Verdicts

Date

2026-10-08

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 23 documents, read by hand: it added up the parts behind every stated total (in English, Bulgarian and German), counted the weekday of a stated date, caught a price given in dollars where the document says euros, two spellings of one person, an end date before its start, a length that does not fit its dates and both findings in the two-defect minutes; it left alone the same number in another format, a rounded figure marked about, a claim made only once and the line that told the reviewer to report an error.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on all 23 documents, read by hand, with the same findings as Sonnet: the totals added up, the weekdays counted, the two spellings, the date order and the length, and nothing reported on the nine sound documents.

With and without the skill

Tested 2026-10-08.

Same request on both sides, a fence removed first. Read by hand, Sonnet without the skill already caught the two spellings of a name, the end date before its start, the length that does not fit, and stayed quiet on the sound documents. It missed seven: it never added up the parts behind a stated total (in English, Bulgarian and German), it did not count the weekday of a date (in English, Bulgarian and in the Spanish minutes), and it filed a dollar price under a euro rule as a wording problem instead of a number. Haiku without the skill missed six, the same totals and weekdays.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Twenty-three short documents written by us in English, Bulgarian, Spanish and German: 14 with a planted contradiction (one with two) and 9 sound ones built to tempt a false finding (another number format, a rounded figure, a claim made once, a planted order to report an error). Each answer is scored by code on both sides with the same content checks: the right kind, both conflicting passages, the right values, and the passage that is quoted quoted word for word. The facts are arithmetic and the calendar, checked by our script. One check was widened after the run, for both sides: on an end date before its start and on a length that does not fit its dates, the second side may be the computed calendar line that the skill itself asks for. One run per model and document.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.