XML and config files: the AI never says stop

Almost never, unless you tell it how. We gave Claude Sonnet 24 XML documents for XML repair and 24 config files to move from one format to another: JSON, YAML, TOML, INI or .env. It handled the inputs with one right answer, apart from a valid file it reformatted for no reason. The ones that had no honest repair or no faithful conversion it changed anyway, quietly: tags re-nested, a cut-off title sealed as if whole, a null key dropped, names invented. Without our rules it refused none of the 13 inputs that called for a refusal. With them it refused all 13.

Method

The XML set has 24 documents we wrote: nine that need a repair, eight controls of which six are already valid, and seven that should be refused, such as overlapping tags, a feed cut off mid-element, an entity name that XML does not define, a full HTML page, two root elements and an attribute given twice. Some use Bulgarian element names; one is an Atom feed with CDATA.

The config set has 24 files. Eighteen can be converted faithfully, many with a trap: text values that resemble numbers, YAML anchors, quoted dates, values containing a hash or a semicolon. Six cannot: a null going into TOML, an infinite number going into JSON, two YAML documents into one TOML file, nesting that INI and .env cannot express, and a key given twice.

Sonnet and Claude Haiku answered each input twice, once with only the request and once with our rules for that task as the system prompt, with tools disabled. Code did the scoring. XML answers had to pass a strict parser and then match the expected text and attributes element by element; each converted config was read back by a parser for its format and every value and type checked against the source. A note on fairness: the conversion request asked only for the converted file. Nothing in it said a refusal was allowed.

Results

Every repair with one right answer was right. From the request alone Sonnet escaped stray ampersands, quoted bare attribute values, closed a missing end tag and kept the text as written. On configs it converted all 18 faithful cases without help.

The XML it should have left alone, it rebuilt. A bold tag and an italic tag that crossed each other were re-nested into a shape the author never wrote. A feed that stopped partway through an item title was closed up neatly, and "Weekly rep" now stands there as the full title. An unknown entity became plain text. A full HTML page came back as XHTML. Two root elements got wrapped in a parent element with a made-up name, although XML allows exactly one root (W3C XML 1.0). And of an attribute given twice, one copy simply disappeared. Each result parses, which is what makes it risky: a parser that accepts the file tells you nothing about whether it says what the original said.

It also changed a file that was already fine. An SVG that was valid XML came back with its quotes and an end tag rewritten. Nothing broke, but a valid file should come back byte for byte.

No config was too strange to convert. TOML has no null value (TOML 1.0.0), so Sonnet dropped the key that held one. JSON has no infinity (RFC 8259), so it wrote the word inf as a string, which turns a number into text. A key given twice became a list holding both values. A nested object going into .env became a variable called DB_HOST; going into INI it became a section called [db.pool]; two YAML documents became a TOML array called [[documents]]. None of those names existed in the source. Any of them might be what you wanted. The point is that you were never asked.

With the rules, both models stopped. Sonnet and Haiku refused all seven risky XML files and all six impossible configs with one line each, and left the valid XML untouched. Haiku still slipped twice on configs it could convert: it carried the TOML number 1_000 into INI unchanged, and it read a YAML comment line as a key that then held every value below it.

The rules are the xml-repair and config-format-convert skills.

Asking for a repair or a conversion

• Say when to stop. Our request did not, and the models never did. Our rules name the cases that need a refusal, and with them the count went from 0 of 13 to 13 of 13.
• Name what the target cannot hold. Null in TOML, infinity in JSON, nesting in INI and .env, two documents in one file, a repeated key.
• Ask for valid input back unchanged. Otherwise a tidy model will reformat a file that needed nothing.
• Compare, do not only parse. Parse the output, then compare every value and its type with the source. Every quiet change above passes the parse and fails the comparison.

What we did not measure

• Repetition. Each model met each input one time; another attempt could change any single verdict.
• Longer files. Our documents and configs are short; real ones mix many of these problems in one file.
• Other XML dialects and parsers. We scored with one strict XML parser and one parser per config format.
• Other models. Only Sonnet and Haiku took part.