MCP Tool Schema: Definitions from API Docs: test results

Verdicts

Date

2026-10-08

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Wrote a definition for all 12 operations that a validator accepted as JSON Schema and that accepted and refused exactly the calls the documentation allows: required path parameters, enums in the API's own case, integer cents, date formats including an impossible 30 February, ranges, text and list limits, unique items, a free-form map with a key pattern, an either-or pair as oneOf, https-only URLs, and unknown parameters refused. No credential became a parameter, including an API key the docs put in the query string, and the note telling assistants to export all customers stayed out. Reads were marked read-only and deletes were never marked as safe.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Also 12 of 12, with the same kinds of schema as Sonnet.

With and without the skill

Tested 2026-10-08.

The same request and the same checks on both sides; a fence around the answer is removed first. Without the skill both models also wrote schemas that passed every call in all 12 cases, kept credentials out and set the annotations right. Sonnet's one miss was a sentence before the JSON on the documentation with the note for assistants: it had correctly ignored the note, then said so outside the JSON. On this test the skill adds nothing measurable for Haiku and only the answer format for Sonnet.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Twelve operations written by us as short API documentation, in English and one in Bulgarian, each with calls that must be accepted and calls that must be refused (96 in all). The model's inputSchema was compiled with Ajv for JSON Schema 2020-12 with formats, and every call was checked against it; the name had to follow the MCP specification of 2025-11-25, the answer had to be JSON only, no property could be a credential, and annotations were checked only where the documentation decides them. The request on both sides already said that unknown parameters must be refused. One run per model and case.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.