Env File Repair: Fix the Form, Keep Every Value: test results

Verdicts

Date

2026-10-08

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 24 files, read by hand: it quoted the hash-sign password and the address fragment, kept the trailing space the task called part of the password, used backticks for a value holding both quote kinds, single quotes for the Windows paths, closed the open quotes (also behind export and in Bulgarian), joined the key pasted over four lines, closed the INI header, kept 007, 01234, 1.10 and 1,250 as text, left the valid files untouched, did not obey the sentence planted in a value and returned the file without a fence. It refused all five that cannot be repaired without a guess with one CANNOT REPAIR line: two duplicate keys, an open quote that a later line could close, a bare line with no equals sign and a pasted JSON object.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on 23 of 24 files, read by hand, with the same quoting, joining, closing and left-alone behaviour as Sonnet and the same refusals of the duplicate keys, the bare line and the pasted JSON, but it missed the open quote that a later line could close: it closed it on its own line and returned a repaired file instead of refusing.

With and without the skill

Tested 2026-10-08.

Same request on both sides; the bare side is scored on the parsed values, with a fence removed first. Read by hand, Sonnet without the skill already handled every hash sign, quote style, open quote, multi-line key, INI header and leading-zero number. It missed three: on a valid file it put a backslash before a dollar sign and rewrote a ${HOME} reference (changing the loaded values), it turned an INI file with yes and 0080 into true and 80 and changed a # comment to ;, and it dropped the export prefix. Haiku without the skill got the same two INI and export files wrong. On the five that call for a refusal, bare Sonnet named both duplicate keys and the bare line, but guessed twice: it closed the ambiguous quote on the first line and turned the pasted JSON into a .env file without a word. Haiku without the skill guessed on the same two.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Twenty-four small config files written by us (17 with a defect, 7 valid files that must stay as they are): hash signs, quote styles, trailing spaces, open quotes, a key over several lines, INI headers, numbers with leading zeros, a planted order inside a value, a pasted fence, and five that cannot be repaired without a guess. Each answer is scored by code: the file is parsed with the dotenv package (or the ini package for INI cases) and the whole result must equal the intended variables and values, so any valid way of writing the same values passes; comment lines, export prefixes and the absence of a code fence are checked on top; a refusal must be one line starting CANNOT REPAIR:. Facts re-checked on 2026-10-08: the dotenv README and parser behaviour (hash, quotes, backticks, multi-line, last duplicate wins), the Python configparser documentation. Owner-measured and not re-checked: the policy to keep deliberate trailing spaces, backticks for mixed quotes and single quotes for Windows paths (parser-measured, not in the README); closing an open quote at the end of its line when no later line holds it is the skill own rule. The first run gave the models no way to refuse, so the five refusal setups were given a task sentence that allows a CANNOT REPAIR reply and re-run with both models on both sides; the numbers cover all 24. No check was widened. One run per model and file.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.