OpenAPI Operation Object From An Example Call: test results

Verdicts

Date

2026-10-08

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 23 example calls, read by hand: null as a type list instead of the 3.0 nullable flag, every path parameter required, integer and number told apart, uuid path parameters, a repeated and a comma list in the query, a 201 with a body and a 204 without one, no invented error responses beyond the ones the rules name, example and examples never together, paging defaults, a deprecated flag set by the rules, rules in Bulgarian, and a sentence planted in the example body kept as data.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on all 23 example calls, read by hand, with the same operations as Sonnet: 3.1 type lists for null, required path parameters, typed queries, the right success codes and no invented responses.

With and without the skill

Tested 2026-10-08.

Same request on both sides; the task text already names OpenAPI 3.1 and the output form. Both models without the skill wrote every operation right, so the skill measured no gain for either of them on these calls. Our cases may simply be too easy for both models once the task names version 3.1.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Twenty-three example calls written by us, each an example request, an example reply and a few plain rules, with traps: null values, optional fields that the example happens to show, enums, list parameters, 201 and 204 replies, integer against number, uuids, auth headers, a 404 the rules add, paging defaults, rules in Bulgarian and a planted sentence in the body. Each answer is parsed as JSON and checked by code against the OpenAPI 3.1 rules and the case: types, required flags, formats, the exact set of response codes. The 3.1 facts were re-read on 2026-10-08 in the OpenAPI 3.1.0 specification and the migration notes from 3.0. No check was widened. One run per model and call.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.