PO Translate: Gettext Files That Still Compile: test results

Verdicts

Date

2026-10-08

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right in all 24 files, checked by the structure of the .po file: every msgid, msgctxt and comment line came back unchanged, Russian, Ukrainian, Polish and Czech entries got three msgstr lines, Arabic six and Japanese one, a missing Plural-Forms line was added and an English one replaced, Turkish and Japanese word order used numbered %1$s placeholders, and the obsolete #~ entries, including an empty msgstr[2], were left as they were.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right in 23 of 24 files, with the same plural counts and header fixes as Sonnet. It missed one: in a Turkish file it put the arguments of "Sent %d reports to %s" in the other order with plain %s and %d, which formats the wrong value, and it wrapped that file in a code fence although the skill says not to.

With and without the skill

Tested 2026-10-08.

Same request on both sides, scored by the content of the first code block because the bare answers were fenced. Read by hand, Sonnet without the skill had two misses. It filled the empty msgstr[2] of an obsolete #~ entry and said so in a note, which changes a block that must stay untouched. In the Arabic file it dropped %d from the zero, one and two forms; the gettext manual allows that for a form that serves one number, so this one is a strict check, not a clear error. Haiku without the skill kept an English two-form header or none and wrote two msgstr lines for Russian, Polish, Japanese and Arabic, and filled the obsolete entry too.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Twenty-four .po files written by us for a fictional app, translated into Russian, Ukrainian, Polish, Czech, Arabic, Japanese, Turkish, Bulgarian, German, Spanish and Italian. They hold plural entries with 1 to 6 forms, headers with the right, a wrong (English) or no Plural-Forms line, msgctxt pairs, developer and translator comments, source references, c-format, python-format and brace placeholders, numbered %1$s placeholders, multi-line and escaped strings, an entry left fuzzy, symbols with no words, and obsolete #~ entries. The check parses the answer as a .po file: entries and msgctxt/msgid/msgid_plural identical and in order, #. and #: lines and #~ lines unchanged, the header kept except Plural-Forms, the msgstr count equal to the language, every msgstr filled and different from the source, placeholders the same, numbered when the order changes. The bare models wrapped almost every file in a code fence with notes around it, so the numbers are scored by content (the first code block), not by the fence. One run per model and file.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.