API Breaking Change Check for OpenAPI: test results

Verdicts

Date

2026-10-18

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 33 pairs of API descriptions, read by hand: it listed removed endpoints, new required fields and parameters, narrowed limits, removed response fields, a response field that may now be absent, null newly allowed in a response in both the 3.0 and the 3.1 spelling, a replaced success status, auth added on one operation or on the whole API, type changes, a new response enum value and a changed default of an optional query parameter. It left out the safe look-alikes that sat beside each break (a renamed path variable, a field moved between allOf branches, null newly allowed in a request, a deprecated flag) and answered exactly No breaking changes. on all seven controls, including the one where a response enum lost a value.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on all 33 pairs with the skill, read by hand: every planted break came out as one coded line with the place, old and new, and which clients are hit, the safe look-alikes stayed out, a removed response enum value was judged safe, a changed default of an optional parameter was listed, and each of the seven controls got exactly No breaking changes.

With and without the skill

Tested 2026-10-18.

Same request on both sides. Read by hand, both bare models already handled removed endpoints, new required fields, narrowed limits, removed response fields, nullable in 3.0 and 3.1, status changes, added auth and type changes, and ignored a renamed path variable, an allOf refactor, null newly allowed in a request and a deprecated flag. Two situations differ. A default of the optional limit parameter moved from 20 to 50: Sonnet saw it and left it out on purpose as a behaviour shift, Haiku did not mention it. The value shipped was removed from a response enum: both called it breaking; the skill says safe, since the response stays inside the old values. That is the skill's policy, not the OpenAPI standard, and a reader could side with the bare model.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Thirty-three pairs of an OpenAPI description written by us from one bookshop description (OpenAPI 3.0 and 3.1, YAML and JSON): 26 with a planted breaking change, each carrying a safe look-alike change as bait, and 7 controls with only safe changes. Sixteen of the pairs are the real starting point, two documents and no release note pointing at the change. With the skill each answer is scored by code: the exact set of finding codes, the place named, one line per change, the fixed sentence for a control, no fence. Without the skill the same request is scored on the same points by content checks in any wording, with a fence removed first. After reading every difference by hand the content checks were widened for both sides: (1) a reported break also counts when it says clients get a value "without it/them", "absent" or "omit..."; (2) it also counts with "refus..."; (3) the nothing-breaks check for controls also accepts "Nothing in AFTER breaks..." and "No changes break existing clients."; (4) a line starting "Nothing breaks." is no longer read as calling a safe change breaking. Each widening was proven able to fail: removing it turned the control script red. Before the widening the counts were Sonnet 32 against 28 and Haiku 33 against 27; after reading by hand the gain shrank to 2 for each. All facts come from the public OpenAPI 3.0.3 and 3.1.0 and JSON Schema 2020-12 texts, read on 18 October 2026; there are no owner-measured facts. One run per model and case.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.