LLM config repair: the syntax holds, the values move

The syntax, yes. The values, not reliably. We had two Claude models, Sonnet and Haiku, repair three sets of small config files: 24 in .env or INI form, 22 in TOML and 22 in YAML. Given only the request, Sonnet closed every quote left open, replaced every tab and added every missing bracket, yet just 19, 17 and 16 of its files came back right, and nearly every wrong one loaded without an error. The harm was to meaning: a value given a new type, a prefix or a key name lost, two conflicting values cut down to one in silence. With written rules for each format added to the request, Sonnet got all of them right.

Method

We wrote every file ourselves, with invented values, and each one plants a single trap:

• .env and INI, 24 files: hash signs inside values, unclosed quotes, a key pasted over three lines, a section header whose bracket never closes, 007 and 0080, seven valid files that should come back untouched, and five that cannot be fixed without guessing, such as a key set twice or a JSON object pasted in place of the file.
• TOML, 22 files: bare text, 007, decimal commas, a date with slashes, a key whose name contains a space, a table header used twice, a key given twice, an INI file under a TOML name, and seven simpler controls, two of them already valid.
• YAML, 22 files: tabs, stray indentation, unclosed quotes and brackets, values that only parse in quotes, a value cut short where the file ends, a file for an older reader, front matter, two error messages in place of YAML, plus seven valid inputs.

Each model saw each file once per side, called from the command line with no tools and a clean configuration. One side got the request alone, the other the same request plus our written rules for that format. Code scored the answers with real parsers: dotenv for .env files, the ini package for INI, a TOML 1.0 parser, and the yaml package, which follows YAML 1.2. The loaded data had to equal the intended data, types included; any layout that loads to the same values passed.

Two setups differ. The TOML and YAML requests asked only for the file, and a refusal in any words counted for the side without rules. The first env wording allowed no refusal, so its five refusal files were rerun on both sides with a sentence that permits one; the totals include that rerun. Two TOML checks were widened afterwards, for both sides. YAML answers without rules were scored with the code fence stripped, as Sonnet fenced 21 of its 22.

Results

A parse check after the config repair would have passed nearly all of it. We loaded each wrong answer again with its test's parser, and only two of Sonnet's failed. One was the TOML key брой опити, Bulgarian for "number of attempts", which Sonnet renamed with an underscore; a bare TOML 1.0 key may hold only ASCII letters, digits, dashes and underscores, so the new name broke the file (TOML 1.0.0). The other was front matter returned with its dash lines, which a bare YAML parser reads as two documents. Everything else loaded cleanly and said something the input did not.

Values the reader would load correctly were given a type. Config formats disagree about types, and the corrections followed none of them.

• INI has no types: Python's configparser keeps every value as a string and accepts both # and ; before a comment (configparser). In a valid INI file, Sonnet still turned 0080 into 80 and yes into true, and swapped a # comment for a ; one. Haiku dropped the zeros too.
• In a valid .env file, Sonnet put a backslash before the dollar sign in p@ss$word and wrapped $HOME in braces. Both are shell habits, since a shell expands a dollar sign inside double quotes (POSIX shell). The dotenv package expands nothing, so both edits went straight into the loaded values: the password gained a character it never had.
• In YAML 1.2, only the spellings of true and false are booleans and no is a string (YAML 1.2.2); in YAML 1.1, yes, no, on and off are booleans too (YAML 1.1 bool). Both models rewrote draft: no in valid front matter as draft: false. Told that the reader follows YAML 1.1, both quoted NO and left debug: off bare, which that reader loads as false. The old rules landed on the file that did not need them and skipped the one that did.
• TOML bans leading zeros, and 1,250 is not a TOML number, so a choice is forced. Both models picked the number, 7, 1250 and 4.99, over a quoted string that keeps what the author typed. Haiku also altered a valid file: in an array mixing a number, a word and a decimal, which TOML allows, it turned the word "two" into 2.0.

A duplicate key was settled by a silent pick. TOML calls a key or a table defined twice invalid (TOML 1.0.0). Both models merged a twice-written table into one. Given name set twice, Sonnet kept the first value and Haiku the second, and neither said so. Under the first env wording, which asked only for a file, a port set twice went the same way: Sonnet kept the later value in both the .env and the INI file, Haiku the earlier one, with the other left only in a comment. The readers disagree too: the dotenv and ini packages we used keep the later value, while configparser, strict by default, stops with an error. Whichever side a model takes, some reader would have loaded something else.

When the env request allowed a one-line refusal, both models refused both duplicates. Each still guessed twice: it closed a quote that could have ended on either of two lines, and it quietly rewrote a pasted JSON object as a .env file.

Text that was not config became config. Sonnet turned an HTML error page into a YAML map with a retry-time field, and put a one-line error message under an invented error key; Haiku built a map from the message too. We saw the same when repairing JSON with a model.

A prefix and a name went missing. Both models removed the export from each line of one .env file. dotenv reads such lines either way, but a shell that sources the file needs export to hand each variable to the programs it starts (POSIX shell). As for the renamed Bulgarian key, TOML accepts a quoted key, which keeps the name; Haiku did exactly that.

With the rules. Sonnet got every file in all three sets. Haiku kept one miss per set: it closed the ambiguous .env quote instead of refusing, returned a Cyrillic TOML value as a single unquoted line without its table, and left a YAML value with a colon-space pair unquoted. The rules we tested are packaged as env-file-repair, toml-repair and yaml-repair.

What to check after a model fixes a config

• Say which program reads the file: dotenv or a shell, configparser or another INI library, and which version of YAML. Without that, the model borrows rules from a neighbouring format.
• Compare loaded values, not just whether it parses. Diff the repaired file's keys and values against the original wherever the original still loads. A file that loaded before should load to identical values after.
• Find each duplicate key with code first. It is a lookup, not a judgement, and the readers above disagree on what a duplicate means.
• Offer a fixed line the model may return instead of a file, and treat it as an error downstream. In our env set it ended the guessing on duplicates, not on the other two cases.

What we did not measure

• Each file ran once per model and side. Repeating it might move individual cases from pass to fail or back, and the gaps between columns come to a handful of files per set.
• Short files of our own, one trap each. Real configs are longer and often broken in several spots at once.
• Our checks take the reader's result as the truth. Some misses are arguable: quoting 1.10 keeps what a person probably meant, though a YAML 1.2 loader then returns a string; draft: false is what a YAML 1.1 reader makes of draft: no anyway; and dotenv loads the same values with or without export. The silent pick between duplicates, the backslash in a password and the YAML built from an error page would fail anyone's check.
• The refusal setups were not equal. In the TOML and YAML sets, refusing meant going against a request for a file; only the env refusals were rerun with a wording that allows one.
• Two TOML checks were widened after the run, and YAML fences were stripped, on both sides. Loosening a check can also let a genuine mistake through.
• Secrets. Every value was invented, and the only change to a secret we saw was one added character. We did not test whether a model shortens, masks or repeats a real secret elsewhere in its reply.
• Only Claude, only two models. Other vendors were not tried, and neither was a deterministic tool that knows the format.