Topic
Testing models honestly
Strong model against weak model, how we ran each test, and what we did not measure.
Skills for this: Writing & Rewriting, Editing & Proofreading, Marketing & Copy, SEO & Content, Research & Summaries, Code & Engineering, Agents & Protocols, Sales & Customer Support, Business & Documents, Data & Analysis, Translation & Localization, Learning & Teaching
- Testing models honestly
Our skills, October 2026: 115 measured with and without
145 skills joined our catalog in October 2026. For 115 we ran the same cases on two Claude models, with and without the file. Most helped; some did not.
Testing models honestlyWindows curl and PowerShell UTF-8: what models miss
Git's curl on Windows turns Cyrillic into question marks while the server still says 200. Alone, Sonnet sent text safely in 9 of 13 tasks and Haiku in 8.
Testing models honestlyMCP tool annotations: what language models miss
Both AI directories want hints on every MCP tool. Haiku set 3 of 19 tool definitions right alone, Sonnet 17; with a checklist in the prompt both reached 19.
Testing models honestlyllms.txt and RSS feeds: what models get right
Sonnet and Haiku wrote llms.txt overviews and RSS feeds with and without our skills. The strong model gained nothing; the weak one swapped slips.
Testing models honestlyOrphan pages and redirect chains: what models miss
Sonnet and Haiku audited crawl exports and wrote redirect maps. The counting was right; the slips came from pages that never loaded and one noindex page.
Testing models honestlyIndexNow, Bing and sitemaps: what models still miss
Sonnet and Haiku reviewed Bing and IndexNow submission code and wrote sitemaps. A retired endpoint and a monthly quota slipped past both; sitemaps were fine.
Testing models honestlyJSON-LD and hreflang: what models add that Google refuses
We gave 44 page-head tasks to Sonnet and Haiku. Valid syntax hid a made-up currency, a self-awarded rating and an unsupported language code.
Testing models honestlyDo AI crawlers really get blocked? Reading the test
We gave 45 crawler situations to Sonnet and Haiku. Bare bot names fooled both, and the smaller model also slipped on robots.txt. Each trap and its fix.
Testing models honestlyPages Not Indexed by Google: Fixes That AI Gets Wrong
We gave 46 Search Console and canonical-URL situations to Sonnet and Haiku. A strong model errs rarely, a smaller one more. Each trap and its fix.
Testing models honestlySocial Media Posting APIs: Limits and Costs Models Get Wrong
Sonnet and Haiku each got 19 of 24 social posting API questions right alone and missed the same five. A skill fixed them, though its first wording broke two.
Testing models honestlyWhat a model gets wrong writing a job post from notes
Sonnet wrote 16 of 23 job posts right alone, Haiku 15. With a skill, 21 and 20, though two Sonnet posts that passed alone then failed.
Testing models honestlyAI Document Consistency Check: It Skips the Sum
Bare, Sonnet and Haiku pointed at wrong totals and weekdays but never wrote the sum or the true day that proves them. A skill file made them show it.
Testing models honestlyA model that summarizes a table adds its own sums
We gave 24 small tables to Sonnet and Haiku and told both to use only printed figures. Sonnet still computed its own on eight tables, Haiku on none.
Testing models honestlyFix Broken XML and Config Files With AI: It Never Stops
Claude fixed every broken XML file that had one fix. Where none existed it re-nested tags and sealed cut text; it refused no config it was asked to convert.
Testing models honestlyCSV to JSON with AI: where data quietly goes missing
Claude converted almost every CSV, JSON and HTML table right unaided. The misses lost data silently: a repeated key, swallowed rows, two fields in one column.
Testing models honestlyMarkdown table repair: the AI invents a column instead
Claude fixed 17 broken Markdown tables on its own. On the four with no safe repair it named a new column, guessed an alignment and made a table of a sentence.
Testing models honestlyFix Mojibake, Broken CSV and JSONL: Script or AI Model?
Plain code passed all 135 checks that Claude Sonnet passed with our rules. Then one of its rules refused an ordinary sentence with a web address in it.
Testing models honestlyAI pull request descriptions: claims the diff never made
Claude Sonnet described 24 diffs and kept to the code on almost all of them. Its few slips were a security fix and a speed-up that no changed line showed.
Testing models honestlyGenerate a Changelog From Git Commits With AI
Claude Sonnet turned 23 sets of commits into release notes. Asked plainly, it listed internal work and invented a link host; with written rules it never did.
Testing models honestlyCloudflare cron day numbers: what Claude gets wrong
Cloudflare counts Sunday as day 1. Bare Claude wrote classic crontab numbers and its schedules ran on the wrong days; with written rules, 36 of 36 were right.
Testing models honestlyWhere the garbled letters in AI pictures come from
A menu board, an invoice total or a 'no letters' line all invite writing into a picture. We sent the same prompts to FLUX and ChatGPT and compared.
Testing models honestlyFix Broken YAML, TOML and .env With AI: Values Move
Claude Sonnet fixed every broken quote, tab and bracket in our .env, TOML and YAML files. Its misses were files that load and say something else.
Testing models honestlyWhy an AI Model Rejects Valid Evidence: Command Output
We gave Claude 70 claims, each with the evidence offered for it. Unguided, it almost never passed weak proof, yet it turned down a third of the sound ones.
Testing models honestlyMCP tool schemas from API docs: models barely need help
Claude Sonnet and Haiku wrote MCP tool definitions for twelve API operations; we sent 96 calls through each schema. Without our skill, 23 of 24 passed.
Testing models honestlyD1 migrations: the table rebuild that D1 rejects
Claude Sonnet and Haiku both rebuilt a referenced table the one way Cloudflare D1 refuses, and so did our first draft. A closing pragma hides the problem.
Testing models honestlyRedacting PII with an LLM: Haiku drops the first words
Claude Sonnet redacted 20 logs and messages exactly, with or without help. Haiku hid things that were not secret and lost the words before the first value.
Testing models honestlyLLM accessibility review: right answers, buried in advice
Claude Sonnet and Haiku found all 14 accessibility defects we planted in HTML, checklist or not. Without one, each fix sat inside a page of advice.
Testing models honestlyReviewing a SKILL.md: models catch injection, not limits
Claude Sonnet and Haiku reviewed 20 SKILL.md files we seeded with problems. Both caught hidden orders and keys; Haiku missed the rules you have to count.
Testing models honestlyAI Excel and Google Sheets Formulas: Our Skill Was the Bug
Claude Sonnet and Haiku wrote correct Excel and Google Sheets formulas for 21 tasks, run in a real formula engine. The errors we saw came from our skill.
Testing models honestlyLLM unit tests from a spec: the JSON broke first
Claude Sonnet and Haiku turned ten written specs into test cases, which we ran against 61 planted bugs. The logic held up; the JSON broke more often.
Testing models honestlyCloudflare Workers Code Review With AI: Edge Trap Missed
We planted twelve D1, OpenNext and fetch pitfalls in Worker code. Sonnet found all of them unaided; Haiku said the edge runtime under OpenNext was fine.
Testing models honestlyTranslating docs with an LLM: the code comments go first
Claude Sonnet and Haiku translated 19 Markdown and HTML documents for us. The prose was fine; code comments, a hidden HTML comment and a link target were not.
Testing models honestlyOur skills, September 2026: measured with and without
The eight skills we released in September, run on Claude Sonnet and Haiku with the file and without it. Five clearly helped, two changed nothing, one hurt.
Testing models honestlyMeeting notes to tasks with an LLM: the invented deadlines
Two Claude models turned 26 sets of meeting notes into task lists. Owners were mostly right; next Friday and end of month got dates nobody had said.
Testing models honestlyTicket triage with an LLM: tone moves the priority
Two Claude models labelled 25 support messages for us. Category and language came out almost always right; priority tracked the tone instead of the problem.
Testing models honestlyAI Cold Email: Invented Facts That Passed Our Checks
We asked Claude Sonnet and Haiku for six cold emails with no rules. Most ran far too long, and the one draft that passed invented a statistic and a discount.
Testing models honestlyHumanize AI Text: What a Model Drops
We asked Claude Sonnet and Haiku to humanize one text in five languages. Without rules, all ten answers lost the word usually, and some invented facts.
Testing models honestlyProofreading with an LLM: it edits what was right
Six texts in five languages, two Claude models. Without rules Sonnet rewrote a correct text, and Haiku mostly sent a list of fixes with no corrected text.
Testing models honestlySummarizing with an LLM: what it adds and drops
Claude Sonnet and Haiku summarized six short texts in five languages from a plain request. Without rules they ran long, switched language and bent conditions.
Testing models honestlyWhich prompt injections get an AI agent to pay?
We hid 98 traps for a Claude agent with a wallet. Orders aimed at the model moved nothing; payments that looked like the job did. A short guard cut most.
Testing models honestlyFixing broken JSON with an LLM: what it quietly changes
We gave Claude Sonnet and Haiku fifteen broken JSON inputs. Without rules both kept cut-off values, rewrote numbers and turned an error page into data.
Testing models honestlyTurso Rows Read Too High: Can a Model Find Why (561 Million)
One of our sites read 561 million rows on a 48 MB database. We handed the pre-fix code to Sonnet and Haiku and scored them against the six real causes.
Testing models honestlySQLite Search That Ignores Accents and Plurals
We gave Claude Sonnet and Haiku twelve broken product searches. Sonnet fixed all twelve unaided; Haiku fixed seven, and its misses hid real costs.
Testing models honestlyAI Code Review Misses Bugs: Small Model Called It Safe
Five hard code reviews for Claude Sonnet and Haiku. Sonnet caught every planted bug; Haiku waved through a sync job that loses rows and praised it.
Testing models honestlyExtracting JSON with an LLM: nulls instead of guesses
We gave two Claude models nine messy texts to turn into JSON. Both left unknown values empty and ignored a planted order; one wrapped every answer in a fence.
Testing models honestlyAI Agent Says Done But It Is Not: Catching False Success
We built 90 traps where a tool result only looks like success. With no extra instructions, Claude Haiku claimed a false success in 56 of 272 runs.
Testing models honestlySmall vs Large Model: Eight Skills Tested
We ran eight SKILL.md files on a strong and a weak model. Both mostly passed our checks; the weak one failed in ways checks cannot see, and used more tokens.