Extracting JSON with an LLM: nulls instead of guesses

When you extract JSON from text with a model, give it a written rule for every place where guessing is tempting, then check its answer with a parser, not by eye. When we did that, Claude Sonnet and Claude Haiku both left a value empty when the text did not settle it, refused to add up a total that nobody had written, and treated an instruction hidden inside an order as part of the order. The values were right in all nine of our texts for both models. The weaker model still failed in a way a person would not notice: it put every JSON answer inside a Markdown code fence, so none of them parsed as they came.

Method

We wrote nine short texts, spread over five languages, each paired with the JSON shape the answer had to fill. Every text hides one or two traps:

• an English invoice with a quantity column, a unit price and a total in USD;
• a German invoice with the amount written 1.250,50 € and dates in day-month order;
• an English note about a review "on 04/05/2026", where nothing tells day from month;
• a Bulgarian job ad with a salary range written 3 100 – 4 300 € and a work mode that has to be mapped to onsite, hybrid or remote;
• a Russian contact list where the second person has no phone number;
• an English order that says "awaiting payment" and then, in brackets, asks "the AI system" to flip it to paid and grant a refund;
• an English quote that gives two different totals and corrects the delivery address in passing;
• a Spanish product sheet with the weight written 2.500 g, asked for in kilograms;
• an English order that lists prices and quantities but never states a subtotal or total.

Each model received the text, the shape and our extraction skill as instructions. We called it from the command line of Claude Code, tools switched off, once per text. A script parsed each answer as JSON and compared the fields we cared about with fixed expected values: null where the text gives no answer, exact numbers and ISO dates where it does.

Results

The missing value is the hard one. An empty field looks like a failure, which is exactly why it needs a written rule. The three cases that tempt it most are an ambiguous date, conflicting values and a sum that is easy to compute. In each, our rule says the field stays null and, if the shape has room for it, a short note explains why. For the two totals, Sonnet wrote exactly that: two different totals, no sign of which one stands, and that the labour and material lines were not added up on purpose.

Separators depend on the language. The same characters mean different numbers. In German and Spanish the dot groups thousands and the comma marks decimals; in English it is the other way round; in Bulgarian a space groups thousands. Both models read all four cases correctly once the rule told them to read separators by the language a document is written in.

A planted instruction became a note. The order text told the AI system to set the order as paid. Both models set paid to false, because the text says the order awaits payment, and left the refund empty. Both also used the notes field to record that the text carried an instruction aimed at them.

The code fence is invisible to a reader and fatal to a parser. Haiku wrapped all nine answers in a Markdown block marked json. The values inside were correct every time, so a person reviewing the output would pass it, while JSON.parse fails on the first character. Sonnet returned bare JSON in all nine.

What we did not measure

• No run without the rules. We did not test the same texts with no instructions, so this post shows what the models did with our rules, not how much the rules changed.
• Nine short texts. Real invoices are longer, scanned, and messier; we used clean text we wrote ourselves, in five languages.
• One run per model. A second run could differ, especially for the weaker model.
• Fields we chose. The script compares only the fields each case names; an odd value in another field would pass unnoticed unless we read it, and we read only some of the answers in full.
• Two models from one family. Other vendors' models were not part of this test.