Test results

MCP Tool Schema: Definitions from API Docs: test results

Tested 2026-10-23, skill version 1.1.1 at the time of loading this page. We run every skill on a strong and a weak model before it is listed, and publish both verdicts, including where the weak one fails.

Verdicts

Date
2026-10-23
Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")
Wrote a correct definition for all 19 operations: a schema that a validator accepted and that accepted and refused exactly the calls the documentation allows, a title at the top and in annotations, and all four hints as true or false with the values the documentation decides. It marked reads of the caller's own account as closed world and a page fetch from any URL as open world, an SMS and a USDC transfer as destructive and open world, and both of our 402cron tools right: running a task now spends a credit and calls the destination, and creating a task with a name in use overwrites it. It split a records endpoint with an action field into one tool per operation, and split a purge with a dry_run switch into a read-only preview and a destructive purge.
Weak · claude-haiku-5-5 (Claude Code alias "haiku")
Also 19 of 19 in content: every hint the documentation decides had the right value, every tool had a title and all four hints, and the records endpoint was split into one tool per operation. It kept the purge as one tool with dry_run, marked destructive, which is the other right answer. One answer of the 19 came wrapped in a Markdown fence, which the skill says not to do.

With and without the skill

Tested 2026-10-23.

Results with and without the skill, for Sonnet and Haiku
SonnetHaiku
withwithoutwithwithout
Right schema, title and all four hints (19 operations)19/1917/1919/193/19

The same request and the same checks on both sides; a fence around the answer is removed first. Without the skill Sonnet got the schemas and most hints right; its two misses were both on our 402cron tools, which it marked as not destructive although running a task spends a credit and creating one with a name in use overwrites it. Without the skill Haiku left the title out of annotations on 16 of 19 tools (5 had no title at all), and on 10 of 19 it got the tool itself wrong: reads of the caller's own account marked as open world, an SMS and the two 402cron tools marked as not destructive, writes without readOnlyHint or idempotentHint, and the records endpoint kept as one tool with an action switch. On the documentation with a note for assistants it ignored the note correctly, then explained that in a sentence after the JSON.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Nineteen operations written by us as short API documentation: the twelve from version 1.0.0, four new ones for the hints (a page fetch from any URL, an SMS to any number, a USDC transfer, a records endpoint with an action field), and three tools from our own MCP servers whose right answer we had decided and checked live (402cron run now and create task, 402post validate). Each comes with calls that must be accepted and calls that must be refused, 136 in all, checked with a JSON Schema 2020-12 validator. Every tool had to have a name of at most 64 characters, a title at the top and in annotations, and all four hints as true or false; the hints the documentation decides were checked for their value. For the purge with a dry_run switch both answers count: one destructive tool, or a read-only preview plus a destructive purge. The request on both sides said that the server will be listed in the Claude and ChatGPT directories. A control run checks that a correct answer passes and that 380 broken ones fail. One run per model and case.

What was not measured

  • Models other than the two named above were not run.
  • Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
  • Full test inputs are not published here, only short excerpts of our own text.
  • Results on your own texts, languages and domains can differ.

Back to MCP Tool Schema: Definitions from API Docs · Card (JSON)