# MCP tool annotations: what language models miss

> Both AI directories want hints on every MCP tool. Haiku set 3 of 19 tool definitions right alone, Sonnet 17; with a checklist in the prompt both reached 19.

Published 2026-10-23 · https://aiskills402.com/blog/mcp-tool-annotations-what-models-miss

The protocol treats each of the MCP tool annotations as optional, but both directories do not. The MCP schema assumes that a tool with no hints writes, destroys and talks to the open world, which is the worst case in each field ([MCP schema](https://raw.githubusercontent.com/modelcontextprotocol/modelcontextprotocol/main/schema/2025-06-18/schema.ts)). OpenAI asks every tool submitted to the ChatGPT directory to spell out three hints (readOnlyHint, destructiveHint, openWorldHint) with a real true or false value ([OpenAI guidelines](https://developers.openai.com/apps-sdk/app-submission-guidelines)). Anthropic wants a title and the applicable hint on every tool of a Claude connector ([Anthropic review criteria](https://claude.com/docs/connectors/building/review-criteria)). Do models set them right? The stronger one mostly does; the weaker one mostly does not. We had two Claude models turn 19 short API descriptions into tool definitions. Alone, Haiku produced 3 usable ones and Sonnet 17; with a checklist in the prompt, each produced all 19.

## How we found out

A company that sells a paid audit emailed us on the morning of 23 October 2026. Its message claimed that one of our MCP servers had 16 tools with titles but no hints at all, and presented that as a problem for listing it. We did not buy anything. We probed our own four servers instead, asking each for its tool list and counting how many tools carried every hint as a proper boolean. Only 6 of 30 did.

Per server: 402cron had 16 tools and none complete; 402post had 6 and one complete; aiskills402 had 4 and one complete; 402registry had 4 and all complete. We filled in the gaps that same day and probed the live servers again, which gave 30 of 30.

One of the gaps was a plain mistake of ours. In the 402post server, the tool that accepts a payment and then puts a listing live had destructiveHint set to false, although a live listing cannot be recalled. It is now true. The models in the test below tripped over the same kind of tool.

## Where the spec and the directories stand

The protocol defines five annotation fields, all optional: a title, plus four hints. Absent readOnlyHint means false. Absent destructiveHint means true, and it only matters for tools that are not read-only. Absent idempotentHint means false. Absent openWorldHint means true. The schema also calls all of them hints that a client must not trust coming from an unknown server ([MCP schema](https://raw.githubusercontent.com/modelcontextprotocol/modelcontextprotocol/main/schema/2025-06-18/schema.ts)).

Hint by hint, in plain words:

- **readOnlyHint** is true for a tool that only looks things up, computes or previews. It is false for anything that changes outside state, starts a job, queues work or sends something out ([OpenAI guidelines](https://developers.openai.com/apps-sdk/app-submission-guidelines)). Anthropic ties it to behaviour: a read-only tool can run without asking each time, while a destructive one always prompts ([Anthropic review criteria](https://claude.com/docs/connectors/building/review-criteria)).
- **destructiveHint** is true for deleting, overwriting, cancelling, revoking and irreversible sends or payments. OpenAI adds that being able to undo an action is not by itself a reason to say false ([OpenAI guidelines](https://developers.openai.com/apps-sdk/app-submission-guidelines)). False means the tool only adds.
- **idempotentHint** says that repeating the same call changes nothing more. OpenAI lists it as optional in its reference table ([OpenAI reference](https://developers.openai.com/apps-sdk/reference)). Anthropic's pages never mention it.
- **openWorldHint** belongs to tools that reach public or open-ended things, such as web search or an address the caller supplies. A tool limited to one private account, workspace or fixed catalog may say false, even when a third party hosts it ([OpenAI guidelines](https://developers.openai.com/apps-sdk/app-submission-guidelines)). Anthropic's pages do not mention this one either.
- **title** is required by Anthropic ([Anthropic submission checklist](https://claude.com/docs/connectors/building/submission)). In the MCP draft the top-level title is shown first, then the one in annotations, then the name.

Three further rules shape the definitions. Anthropic caps a tool name at 64 characters, stricter than the 128 the protocol suggests. Both directories want a single operation per tool, never a catch-all taking a method or action parameter ([Anthropic review criteria](https://claude.com/docs/connectors/building/review-criteria)). And both refuse tools that move money or crypto for the user, which matters to anyone selling through payment protocols.

The sources do not fully agree. OpenAI calls three hints required, yet a code example on its own reference page sets only readOnlyHint. A forum thread shows a real rejection that asks for hints to be set explicitly and never left null ([OpenAI forum](https://community.openai.com/t/clarification-needed-on-rejection-feedback-tool-annotations-and-developer-name-mismatch/1379402)). Anthropic's wording differs between its review page, which asks for a title and the applicable hint, and its policy, which asks for all applicable annotations and names three ([Anthropic policy](https://support.claude.com/en/articles/13145358-anthropic-software-directory-policy)). Setting all four everywhere satisfies both readings, and that is our own conclusion rather than a quoted rule.

## Method

We wrote 19 operations as short API documentation. Twelve were reused from our earlier test of the same job. Four were new: fetching a page from any URL, texting any number, sending USDC, and a records endpoint steered by an action parameter. Three came from our own servers, where we had settled the right answer and confirmed it live. Every operation had calls its schema must let through and calls it must block, 136 in total, tried against a JSON Schema 2020-12 validator.

A definition passed only if the name stayed within 64 characters, the title appeared as its own field and inside annotations, every hint was a boolean, and each hint that the documentation settles had the correct value. Both sides were told the finished server was headed for the two AI directories. One side additionally had the text of the checklist skill [mcp-tool-schema](/skills/mcp-tool-schema) in its instructions. Each model ran each case once per side, 76 runs altogether. A control confirmed that a correct answer passes while 380 deliberately damaged ones do not.

## Results

| | With the skill | Without |
|---|---|---|
| Sonnet: definitions that pass | 19 of 19 | 17 of 19 |
| Haiku: definitions that pass | 19 of 19 | 3 of 19 |

The table counts content after removing a Markdown fence from around the answer. Alone, both models mostly put the JSON inside one. Counting a fence as a failure, the raw scores are Sonnet 19 with and 4 without, Haiku 18 with and 0 without. We think the fence is a formatting habit rather than a fault in the hints, so the table leaves it out, but anyone parsing the answer by program would feel it.

**Sonnet's two misses were the same mistake, on our own tools.** Starting a task immediately costs a credit and contacts the destination, and creating a task under a name that exists replaces the old task. Sonnet called neither destructive. Everything else was right, the SMS and the USDC transfer included. Neither name says delete.

**Haiku's trouble was wide.** On 16 of 19 tools the title was missing from annotations, and on 5 of 19 it was missing everywhere. On 10 of 19 the tool was wrong in substance: it flagged lookups inside the user's private account as open world (order list, sales report, a parameterless tool, one validation tool of ours), declared an SMS and both 402cron tools non-destructive, skipped a hint on writes such as a refund and mark-as-read, and left the records endpoint as a single tool driven by an action field. The three it passed alone were the delete with a key in the query string, the purge, and the USDC transfer.

**One more thing the models did right.** One documentation page contained an aside urging assistants to call a different tool. Haiku, unaided, rightly ignored it, then justified itself in a sentence after the JSON. That breaks a parser but leaks nothing.

**A purge with a preview flag has two right answers.** With the skill, Sonnet gave a read-only preview tool and a separate destructive purge, while Haiku gave a single destructive tool. Both are defensible. Our first check accepted just one of the two shapes, so we corrected the check and left the answers alone.

This changes what we wrote in our [earlier test of this skill](/blog/llm-mcp-tool-schema-little-help), where both models barely needed help. That test checked only the obvious annotation values and skipped openWorldHint; here all four hints and the title are the question, and the gap is large.

## A short checklist for your own tools

1. Give every tool a title, both as its own field and again inside annotations.
2. Fill in each of the four hints with a real boolean. A missing key counts as badly as null, and the defaults assume the worst case. Read the live tool list, not your source.
3. Ask of each tool whether it costs money or overwrites something. If so, destructiveHint is true, including a tool that reads as a plain "create".
4. Set openWorldHint by who can be reached. Your user's private account is a closed world; any URL, number or wallet is open.
5. One operation per tool. Split anything with an action or method switch.
6. Keep names short and plain, and do not build tools that move funds on the user's behalf for these directories.
7. Remember that hints are not security. Keep your own authorization checks.

## What we did not measure

- **Acceptance.** We cannot say whether either directory would now accept any of our servers.
- **Client behaviour.** We did not see how either product treats an unannotated tool, such as prompting on every call. We only have what their documents say.
- **Two models, one run.** Claude Sonnet and Haiku only, one run per case. A repeat might shift a borderline case.
- **Our own cases.** We wrote the 19 operations, three from our own servers, so they favour mistakes we already knew about.
- **Moving documents.** The directory pages were read on 23 October 2026, and OpenAI now calls its apps plugins. Requirements may change.