Reply Language Lock: Answers in the User's Language: test results

Verdicts

Date

2026-10-09

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 24, checked by code on the language of the answer and on words and letters of the neighbouring language: Bulgarian under an English build log, a Russian forum post, a Russian error message and a planted line asking for English; Ukrainian, Polish, Czech, Spanish, Portuguese, German, Turkish, French, Italian, Japanese and Russian under material in another, often neighbouring, language; an email written in English when asked; and Cyrillic for a Bulgarian user who typed in Latin letters.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on 21 of 24, but three times it answered in Bulgarian where it should not: a Ukrainian user asking about a Russian text, a Russian user asking about a Bulgarian one, and an English user asking about a Bulgarian one.

With and without the skill

Tested 2026-10-09.

Same request and same checks on both sides. Read by hand, Sonnet without the skill drifted into the language of the material three times: it answered a German user in Dutch, a Portuguese user in Spanish and a Russian user in Bulgarian, each time summarising the pasted text in its own language. It also answered the Bulgarian user who typed in Latin letters in Latin letters; the check accepts either script, so that is not counted. Haiku without the skill missed six, mostly on neighbouring languages.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Twenty-four single-turn cases written by us: the user writes a short request in one language and pastes material in another, often a close neighbour (Bulgarian with English and Russian, Ukrainian with Russian, Polish with Czech, Czech with Slovak, Spanish and Portuguese both ways, Italian with Spanish, German with Dutch and English, Russian with Bulgarian, Turkish, French and Japanese with English), plus a planted line asking for English, an explicit request for an English email, a Bulgarian user typing in Latin letters and 3 controls. The bare request is only the user's message, with no instruction about the language of the answer. Checks: the language of the answer and forbidden marks of the neighbour. Checks widened after the run, for both sides, each after a right answer was refused: an English email may come with words to the user in Bulgarian and with a Subject line, and the length limit on the one-sentence controls was loosened. One run per model and case.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.