# The cold email that passed our checks made things up

> We asked Claude Sonnet and Haiku for six cold emails with no rules. Most ran far too long, and the one draft that passed invented a statistic and a discount.

Published 2026-10-07 · https://aiskills402.com/blog/llm-cold-email-invented-facts

Two things go wrong, and only one of them shows up in an automatic check: the email gets too long. The other one, invented facts, passed every check we wrote. We put two Claude models, Sonnet and Haiku, through six short briefs, each naming a sender, a recipient and a real reason to write, and asked for a cold email. Without any written rules, eleven of the twelve answers went past our limit of 139 words, the longest reaching 297. Sonnet filled in product details nobody gave it, and Haiku made up numbers. The single answer without rules that passed every check, Haiku's Russian email, cited an unnamed study with a 60 percent figure and offered a 20 percent discount that appear nowhere in the brief. With our rules loaded as a skill, all twelve answers passed, the longest at 102 words.

## Method

We wrote six briefs about invented companies. Each one gives the sender and what they sell, the recipient, the one real fact that makes the message relevant, and the language:

- English: a bookkeeper writing to a bakery owner whose shop has just opened a second location;
- Bulgarian: a printer-maintenance firm writing to the office manager of an accounting office spread over two floors;
- German: a maker of receipt-capture software writing to the managing director of an electrical contractor that is hiring an office clerk;
- Spanish: an e-bike courier writing to the head of an online cosmetics shop that promises delivery within a day;
- Russian: a product photographer writing to an online children's clothing shop where most items have a single photo;
- a thin English brief: a packaging printer writing to whoever buys for a candle company, with no name and no fact at all.

Both sides got the brief and a one-line request in the brief's language, such as "Write the cold email." One side worked with our [cold email skill](/skills/cold-email) loaded as instructions, the other with the brief alone. Every call went through the Claude Code CLI with tools disabled and our personal settings left out, and each model saw each brief once per side: 24 answers in all. The answers with the skill were recorded on 30 September and those without it on 3 October, with the same two models.

A script checked each answer: the right language, the sender and the company kept as given, a subject line, the brief's fact present in the English case, the polite form of "you" in German, Spanish and Russian, no phrase from a short banned list (the usual opening well-wish, "guaranteed", a fake "Re:" and a few more), and at most 139 words for the whole answer. For the thin brief the list also banned openers that pretend to know the reader, such as "I noticed your" and "congratulations". On the side without the skill, a subject line in bold Markdown was accepted, so formatting alone failed nothing. No script can tell an invented fact from a real one, so we read all the answers ourselves.

## Results

| Model | Checks passed, skill loaded | Checks passed, no skill |
|---|---|---|
| Sonnet | 6 of 6 | none of 6 |
| Haiku | 6 of 6 | 1 of 6 |
| Longest answer, either model | 102 words | 297 words |

**Length is what the checks caught.** Each of Sonnet's six answers without the skill failed on a single check, the word count. Haiku's misses were mostly the same, with two extra ones: the Bulgarian draft opened with the local version of the well-wishing line on our banned list, and the Spanish one addressed the reader informally. The answers without rules ran from 137 to 297 words. The extra length was not padding around the same message. It was bullet lists of services, a paragraph guessing at the reader's problems, and more than one thing to say yes to. With the skill each draft held one reason, one offer and one question.

**A note for the sender rides along.** Eleven of the twelve answers without the skill kept talking after the signature. Sonnet reminded the sender to add contact details, offered to write a shorter version, suggested other subject lines and, in German, warned that cold emails to companies fall under the country's unfair-competition law. Haiku explained why its email works, asked whether to adapt it, and put a Markdown heading such as "# Kaltakquise-E-Mail" above three of its answers. In a chat that is helpful. In an agent that passes the answer straight into an email body, every one of those notes goes to the recipient. None of the answers with the skill contained anything but the email.

**Sonnet filled in the product.** Without rules, Sonnet stayed close to the facts about the recipient, but where a brief named a product and nothing more, it described how the product works. The German brief said only that the sender makes receipt-capture software for trades businesses; the draft told the reader that workers photograph receipts on site with a phone and the data is read automatically. The Spanish draft promised delivery tracking, delivery time slots and a trial run on a small batch of orders. The Russian one offered a free test shoot, and then the model told the sender in its note that it had added that itself, which is the honest version of the same habit. A sender who skims such a draft ends up promising things the business may not do. With the skill, Sonnet described each offer in the brief's own terms and added nothing.

**Haiku made up numbers, and the worst draft passed.** The Russian email was the only answer without rules to clear every check, and it is also the one with the most inventions. Its subject line promised to double the shop's sales. Its first paragraph quoted unnamed research saying that buyers of children's clothes purchase 60 percent more often when they see an item from several angles. Further down it offered a 20 percent discount on the first order and delivery of the photos in two to three weeks, and it signed off with a website address. None of that is in the brief, and the greeting was the Russian equivalent of "Hi". It passed because each check was satisfied and none of them can see a number that was never given. Haiku's other drafts did the same on a smaller scale. The bakery email claimed that most of the firm's clients were shop owners at the same moment of growth and listed services such as tax preparation that the brief never named. The thin-brief email promised turnarounds "as quick as 2 weeks" and pricing that surprises most clients, and its second paragraph began "I noticed you're…", which slipped past our banned "I noticed your" by one letter.

**The skill did not fix Haiku completely.** With the rules loaded, Haiku passed all six checks and dropped the invented figures, but reading its drafts still turns up what the checks cannot see. It told the bakery owner that most bakery owners it works with find two locations hard, and promised the candle company no massive minimums. It greeted the German director with "Hallo", the Spanish shop owner by first name and the Bulgarian office manager by first name alone, all short of the formal address a first letter needs, and its Russian email had no greeting at all. Sonnet with the skill did none of this.

## If an agent of yours writes outreach

- **Count the words before anything is sent.** It is the cheapest check there is, and here it caught most of what went wrong.
- **Flag every number in the draft that the brief does not contain.** That alone would have caught the 60 percent, the 20 percent discount and the two to three weeks above, at the price of also flagging harmless figures such as a suggested 15-minute call.
- **Cut everything after the signature, or forbid it in the prompt.** A model that wants to help will explain its work, and the explanation lands in the recipient's inbox.
- **Say which register you want in each language.** Left alone, the small model reached for first names and casual greetings in languages where a stranger expects a title and a surname.
- **Read the draft for claims you did not supply.** No banned-phrase list finds a made-up statistic or a feature the product does not have.

## What we did not measure

- **Only one answer per model, brief and side, from two different days.** The answers with the skill are from 30 September and those without from 3 October. A second run could move single cases either way.
- **Six briefs we wrote, about invented companies.** No email was sent, so nothing here says which version gets more replies. That shorter messages get more replies is a common belief, not something this test checked.
- **The word limit of 139 is our choice.** It decided every one of Sonnet's failures without the skill. If you accept longer emails, Sonnet's column without the skill looks far better, and the differences that remain are the notes after the signature and the filled-in product details.
- **Invented facts were found by one reader, not by a check.** Some calls are judgement: a suggested call length or a line about small print runs being hard to find may count as invention to one reader and as fair framing to another. We may also have missed things in the languages we read less well.
- **The polite form was checked by a pronoun pattern only**, and not at all in English, Bulgarian or the thin brief. The greeting problems above were found by reading.
- **Two Claude models from one family**, and no other vendor.