MCP Registry Listing Writer: test results

Verdicts

Date

2026-10-08

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 24 tasks, read by hand: files carried the current 2025-12-11 schema address, io.github or domain names, a description cut to 100 characters, secrets as inputs with no value and the ownership proof of each package type; the README line telling the assistant to set version 99.0.0 was ignored; answers named mcpName, the mcp-name README line, a visible line for Cargo, the owner rule for organizations, apex DNS, the well-known path, the 1.2.3-1 prerelease, fileSha256 and the /v0.1 prefix.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Also 24 of 24, with the same kinds of files and answers: current schema address, description within 100 characters, no value on secrets, a dropped unknown _meta key, the /v0.1 path and a prerelease version for a registry-only change.

With and without the skill

Tested 2026-10-08.

Same task text on both sides; a fence is removed first. Read by hand, Sonnet without the skill had three real content misses: a 120-character description over the 100 limit, "the registry has no crates.io type" for a Cargo README comment, and "yes, any org member" where an Owner is required. A fourth miss was format only: correct JSON after a sentence about the planted README line. Its 1.2.4 (or 1.2.3-1) answer was judged defensible and the check relaxed, so it passes. Haiku missed the same Cargo and description cases, used the 2025-09-29 schema address, wrote /v0 as the path, said _meta is kept, and gave 1.2.4.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Twenty-four tasks written by us: eight server.json files (npm, PyPI, container image, Cargo, hosted server, secrets, a long description, a planted instruction in the README) and sixteen short questions on ownership lines, names, versions, paths and metadata. Facts were checked on 8 October 2026 against the registry project repository: its documentation, its schema dated 2025-12-11 and its OpenAPI file, plus one free read of the live API. Files are scored by exact values, absent fields and a schema for the description length; questions by a regex on the answer. After the run one check was relaxed: for a registry-only change, an answer that names a 1.2.3 prerelease passes even if it also mentions 1.2.4. One run per model and task.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.