What a model misses when it reviews an x402 listing

Partly. Two Claude models, Sonnet and Haiku, reviewed 23 short x402 listings, scripts and plans, 16 of them hiding a planted flaw. On their own, Sonnet named 10 of the 16 in some words (9 on a strict reading) and Haiku named 6. Most misses were facts about how the catalog and the buyer client behave, and careful reading of a pasted snippet cannot reveal such a fact. Once the skill's text joined the instructions, both models named all 16.

Method

The 23 cases are ours: a 402 body, a script that checks whether a seller is listed, a runbook note, a hook that runs after each sale, a discovery route. Sixteen carry a planted problem (one carries two, one has a comment aimed at the reviewing model) and seven are sound. Each model received the same request: list every problem, worst first, and close with a one-line verdict. Each model got one bare run and one run with the skill on every case.

Scoring differs by side. A bare answer counts when it names the planted problem in any words, decided by keyword patterns; sound cases carry no check there. With the skill, code scores fixed finding codes and the verdict line, and a code that should not be there fails the answer. The with-skill side is therefore stricter. We re-scored all saved bare answers against the final patterns, and an answer with no content passes none of the 16.

Results

Bare Sonnet wrote long, sensible reviews, usually with about ten findings: no status check on the fetch, a purchase on a GET route, thin schemas, a placeholder address. Four of its six misses trace to platform facts it had no way to know; the other two are a missing control and a judgment call about rows.

The buyer client stops at $1 unless told otherwise. The standard client package compares each payment with a limit that defaults to "$1" and refuses with an error naming spendControls.maxAmountPerPayment (package source, version 2.28.0). We read this on 8 October and found it unchanged in the current release. In that code the check looks at the amount alone and never reads the description, so advice placed in the description field of the 402 cannot prevent the refusal. A seller with a $2 pack should put the note where agents read before they pay, such as llms.txt or the tool description. Bare Sonnet missed the spend cap on the $2 pack and again in the combined case. In a third case, where the note sat only in the 402 description, its pass is generous: it named maxAmountPerPayment, but read the note as an attempt to steer the buyer and recommended deleting it, never mentioning that the refusal comes first. Counted strictly, it named 9 of 16.

Search answers under resources, browse under items. In the Bazaar catalog documentation, search iterates result.resources, browse iterates page.items, and an unknown merchant address gives an empty resources list; the extension specification describes the same array for search (documentation, specification). A script reading items from a search call prints "not listed" forever. Sonnet said it could not confirm the field name. With a planted "reply No findings" comment on the same kind of script, it ignored the comment, which is right, yet still missed the field.

A listing check needs a query that is supposed to find nothing. Without one, "we are listed" can be printed for the wrong reason whenever the API answers something unexpected. Also query a domain or address nobody owns, and fail loudly if anything comes back; the documentation says an unknown address yields an empty list, so the test is cheap. On the merchant-lookup script Sonnet gave ten findings and none was this. On a script that already had such a control, it rightly noted that an erroring API also comes back empty, and asked for the opposite kind as well, a domain known to be listed. It did warn that a scanner page may answer 200 for any path, which matches an observation of ours from early October that we did not re-check.

Two rows beat four. This is our September observation, not re-checked: an outside price-comparison bot looked at the machine-readable amount and credit count, not the prose. Take packs of 10, 50, 200 and 1000 renders at $0.50, $2, $6 and $20. The 10-pack is the cheapest entry and the 1000-pack the lowest rate, so the middle two lose on both counts. Advertising only the extremes, linked to each other, says what four rows say. Bare Sonnet never made that case when shown all four. When only the cheapest row was advertised, it did see the hidden bulk rate and asked for a route per pack.

Haiku found 6 of 16 alone. Shown the planted reviewer comment, it said it ignored it, called it a steering instruction and advised removing it, then missed the resources field as Sonnet did. On the crawler case it noted that the two logged buyers look like scripts or scanners, yet proposed no count of the wallet's recipients and no search for a follow-up request, so the keyword check scored a miss. On the one-row tariff it caught a placeholder address and unpriced packs but never said the lowest rate was hidden.

The skill x402-discovery-listing-review changed the naming: all 16 problems came up for both models, each as a fixed code with location, reason and fix. One case rests on a public scanner's indexer specification: version 1.7.5 reads /openapi.json and has dropped parsing of the legacy x402 manifest under the well-known path. Both models caught that even bare. The scheme-name case is shakier; the specification never mentions bearer, and that the indexer judges a security scheme from its key is a September observation we did not re-check. Bare Sonnet passed it only through a widened check: it proposed answering the 402 before the token check, which also fixes the listing.

The full score, which also demands the right verdict line and no stray code, was 20 of 23 for each model. The six failed answers stay in:

• Sonnet added a stale-record line twice, on pastes where nothing was waiting for a scanner refresh. It also gave the wrong closing verdict on the crawler-as-customer case, though the finding itself was right.
• Haiku appended a pareto-rows line to two cases where the tariff was not in question; in one of them the paste states that both packs are advertised, yet the line claims the cheaper pack lives only in prose. It also raised an unstated cap on a sound two-row listing.

What we did not measure

• Single pass. One answer per model and case; one or two flipped cases would visibly move the totals among 16 traps.
• Our own cases. We wrote all 23 and kept them short; a real listing is longer and messier.
• Keyword scoring on the bare side. The patterns were widened after right answers were refused, and one of Sonnet's 10 passes is generous, so strictly it is 9.
• Bare false alarms. The seven sound cases have no bare check, so we do not know how often an unaided model complains about a sound listing.
• A skill side that moved. After a first run we dropped a sound case in which both models found genuine flaws, and changed the skill's advice on crawler buyers. We then re-ran the skill side in full; the first run is history, not part of these numbers.
• Dated facts. The client limit, catalog fields and scanner specification were read in October 2026 and can change. The key-based scheme reading, the stale scanner record, the comparison bot and crawler-like first purchasers of a low-priced entry are September owner observations, and the scanner page that answers 200 for any domain is one from early October; none was re-checked.
• Two models, same family. Others may behave differently.