JSON Diff Paths: test results

Verdicts

Date

2026-10-09

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 24 pairs, checked by code against a diff computed by our script: "1" kept apart from 1 and null apart from a missing key, 1.0 against 1 and a new key order not reported, keys with a slash, a tilde or an empty name escaped in the pointer, a whole added object as one entry, arrays compared by position, a sentence planted in a value treated as data, and both broken documents refused with the CANNOT DIFF line.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on 23 of 24 pairs, checked by code. Every diff was exact; on the document cut off in the middle of a key it refused correctly, but in a plain sentence without the CANNOT DIFF line a program looks for.

With and without the skill

Tested 2026-10-09.

Same request on both sides. On the two broken documents both sides are scored on any clear refusal with no diff, so the missing CANNOT DIFF line of Haiku is not counted here. Our request already spelled out the shape, the pointer paths and the type rules, and with that both models got every pair right without the skill; the skill adds no measured gain on this set. We did not test a shorter request.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Twenty-four pairs of JSON documents written by us, in English, Bulgarian, Spanish and German: 15 with a difference built around a trap (strict types, null against a missing key, shifted, swapped and shortened arrays, a container whose type changes, pointer escapes, number spellings, a changed root, a planted sentence and a large nested document), 2 with an invalid document and 7 controls with no real difference. Every expected entry is computed from the parsed documents by a small diff function, never by hand, and each answer is parsed and compared as a set, so the order of entries and of keys is free. The shared request states the output shape, the pointer format and the type rules; it does not mention null, key order, escapes or the planted sentence. No check was widened. One run per model and pair.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.