Bulgarian Transliteration (Official Latin Spelling): test results

Verdicts

Date

2026-10-08

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 24 texts, read by hand: ъ as a in Targovishte and Balgarskata, щ as sht, ц as ts, ж as zh, й as y, the word-final ия as ia in Maria, Ilia and Azia, the country itself as Bulgaria, capitals kept in an all-caps line, a hyphenated surname, an address with its number, Markdown, digits and Latin words such as Wi-Fi and Xiaomi left as they were, slugs built from the finished text, Russian letters outside the Bulgarian table kept and named, and the line telling it to ignore its instructions transliterated as text.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on 18 of 24 texts, read by hand, but it missed six: it kept ия as iya in Илия and Азия, wrote Bulgarskata for the adjective where only the country name is Bulgaria, wrote й as i in one word, used the letter ž for Джон, and put an asterisk inside a Russian place name it was asked to keep.

With and without the skill

Tested 2026-10-08.

Same request on both sides, a fence removed first, compared with the same expected text. Read by hand, Sonnet without the skill already handled zh, ts, ch, sh, the plain names, the address and the Russian letters. It missed nine: it used another system for Targovishte (Târgovište), wrote ия as iya in Maria, Yulia, Ilia and an all-caps line, wrote Bulgarskata for the adjective and Balgaria for the country (both the wrong way round), translated рутер into router inside a sentence, and missed the surname with щ. Haiku without the skill missed thirteen, the same kinds plus ъ written as u and the slugs. With the skill Haiku did worse than without on two texts (Илия, a Russian place name).

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Twenty-four Bulgarian texts written by us: names, towns, an address, capitals, a hyphenated surname, the endings ия and ъ, the country name, Markdown with Latin words and prices, three slug tasks, two words with Russian letters and a line that tells the model to ignore its instructions. Each answer is compared by code with the exact expected Latin text (a slug task as JSON with the text and the slug), so a single wrong letter fails the case. The rules are the official table of the Bulgarian Transliteration Act of 2009 (Art. 4 table, Art. 5 for ия at the end of a word, Art. 6 for the country name), re-read on 2026-10-08 on a public copy of the Act. The first Haiku run with the skill was cut off by a usage limit that the tester then recorded as answers; that run was thrown away and Haiku was run again with the skill, and the tester now stops on such a message. No check was widened. One run per model and text.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.