JSON Repair: Fix It, Keep Every Value: test results

Verdicts

Date

2026-10-07

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Passed all 15 cases with bare JSON equal to the expected value: every key, value, type and order. It dropped a code fence and chat text, removed comments but kept the double slash inside a URL, turned a Python dict into JSON without obeying a line that told the AI to set approved to true, set a cut-off string and a cut-off number to null and dropped a cut-off key, kept the zip 01234, the id 007, the hex 0x1F and the ambiguous price 1,250 as strings, left curly and Bulgarian quotes inside values untouched, returned already valid JSON unchanged, and answered an HTTP 429 error message with a CANNOT REPAIR line instead of an invented object.

Weak · claude-haiku-4-5-20251001 (Claude Code alias "haiku")

Got the values right in 12 of 15 cases: the same nulls for cut-off values, the same strings for 01234, 007, 0x1F and 1,250, the planted instruction ignored, valid JSON left alone, and the same CANNOT REPAIR line for the error message. But it wrapped the JSON in a Markdown code fence in 7 of 15 answers, so only 7 answers are both right and parse as they are. It straightened curly quotes inside values in two cases, which changed one value and broke the JSON in the other, and once kept a cut-off quantity of 1 instead of null.

With and without the skill

Tested 2026-10-07.

One run per model and case, the same checks on both sides. Without the skill both models kept cut-off values as if they were complete, turned 1,250 into 1250 and built a JSON object out of an HTTP 429 error message; Haiku also finished a cut-off sentence, added a price of 0 and a renew flag of true that the input never had, and put the long id in quotes. Haiku fences JSON with or without the skill, so strip fences if your agent runs on it.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Fifteen fictional inputs, one per kind of damage: a fenced answer with trailing commas, comments next to URLs, a Python dict with a planted instruction, a string, a number and a key cut off at the end, unquoted keys with missing commas, raw line breaks and inner quotes, numbers that would lose information, a decimal comma, NaN and undefined, curly quotes, a Bulgarian Python dict, valid JSON that must stay as it is, and an error message with no JSON at all. The check parses the answer and compares the whole value with the expected one. One run per model and case.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.