PO File Repair: Make msgfmt Accept It Again: test results

Verdicts

Date

2026-10-09

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 24 files, compared by meaning with an answer msgfmt --check accepts: quotes escaped and closed, an unknown backslash doubled, newlines matched at the right end, a wrong placeholder type replaced, fuzzy added where a placeholder or a plural form was missing, msgstr indexes put right, charset and Plural-Forms fixed from the Language header, an identical duplicate dropped, comments, flags and #~ lines kept, empty entries left untranslated despite a comment asking otherwise, four files that already compile left alone, and a CANNOT REPAIR line for two different translations of one entry and for a plural formula with no language named.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on 20 of 24, but twice it dropped the header entry while repairing, once added a newline at the wrong end of a translation, and refused one file in a plain sentence instead of the CANNOT REPAIR line.

With and without the skill

Tested 2026-10-09.

Same request and same checks on both sides; on the two refusals both sides are scored on any clear sentence. Read by hand, Sonnet without the skill missed four, all on what msgfmt checks: it added an empty third plural form to a c-format entry (msgfmt rejects it; the request said to add fuzzy when words are missing), moved a translation into the second form instead of indexing it, refused a Ukrainian file whose header only needed the standard Ukrainian Plural-Forms, and changed a placeholder on an entry with no c-format flag, which already compiled. Haiku without the skill missed eight.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Twenty-four .po files written by us in Bulgarian, Russian, Ukrainian and German: 18 broken in a way msgfmt --check rejects (quotes, escapes, newlines, c-format and python-format placeholders, plural forms, msgstr indexes, charset, Plural-Forms, a duplicate, several faults at once, a comment asking to translate, a fuzzy entry and obsolete lines to keep), 2 that must be refused and 4 that already compile. The case script runs msgfmt --check (GNU gettext) on every input, ideal and control, and refuses to write the cases unless each broken input is rejected and each ideal and control accepted; it caught one of our own ideals (an empty plural form added to a c-format entry is rejected). Each answer is compared with the ideal by meaning: entries, translations after unescaping, flags, comments and obsolete lines. The shared request states what may change, when to add fuzzy and when to refuse. No check was widened. One run per model and file.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.