Safe Refactoring: Refactor Without Changing Behavior: test results

Verdicts

Date

2026-11-13

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 20 runs, scored by running the code it wrote next to the original. In each of the nine cleanup cases it kept the old result for explicit None or null, zero and empty strings, or named the change: a discount list that may arrive as None kept its None check, a logger and a fetch helper kept working with a null options object, and a page size of zero kept its rule. The control was modernized with no difference. Every answer was a real rewrite and ended with a Behavior section.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on all 10 cases, scored by running the code it wrote next to the original: no empty input changed its result without a line saying so in the Behavior section, the control was modernized with no difference, and each answer rewrote the code rather than leaving it as it was.

With and without the skill

Tested 2026-11-13.

Same bare request and same scorer on both sides. Read by hand, Sonnet alone named its changes on a CSV export, a React avatar, a retry policy and a settings loader. It missed a discounts function where codes=None became an empty tuple called safe, so passing None raises; a logger and a fetch helper whose null options object now throws; a paginator where a branch it called dead still handled a page size of zero; and one of two runs on a price formatter (strictly 10 of 20, since the other run counted only for a general sentence). A probe with only the name and short description also gave 10 of 10: the gain comes from pointing Sonnet at empty values, and the body added nothing measurable on these cases. Haiku alone missed three: the null logger options, a null avatar name and backoff=None in a retry policy.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Ten cleanup and modernize requests written by us in Python, JavaScript and React (nine where a rewrite can quietly change what a missing or empty value becomes, one control), each a bare request with no word about empty values. Scored by a script that runs the code from every answer and the original on the same inputs (explicit None or null, 0, an empty string or list, a left-out argument, a null options object) and lists each difference; whether the answer named a difference was read by hand against a rubric written before the first run. The pattern check in the case file only asks for an answer, so the count is that hand reading, not the generic comparison script. The language rules are public reference facts; nothing here is owner-measured. No check was widened after the runs. Sonnet two runs per case on each side, Haiku one.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.