Partly. Asked 24 short questions about posting to six services, Claude Sonnet and Claude Haiku each answered 19 correctly with no help, and they failed the same five. The five were not random. Several were things the services changed or never wrote down. Their plan would have been sound on how the protocols work and wrong on a few facts: one paywall from this year, one rule that runs against intuition, and Pinterest behaviour that no help page states. With a skill's text added, Sonnet reached 24 and Haiku 23, but the first wording of that skill also made Sonnet fail two questions it had answered correctly alone.

Method

Our test covers six services: Mastodon, Bluesky, Meta Pages, Pinterest, Hashnode and X. We wrote 24 questions, 16 of them built around a trap, meaning a fact that is new this year or missing from the documentation, and 8 as controls with plain answers. A reply has to be JSON with named fields, and a script checks each field against a single value. Wrapping the JSON in a fence, or adding a field, fails it. Claude Sonnet and Claude Haiku saw identical wording, ran inside Claude Code with every tool switched off, and answered each question one time.

The right answers have two origins. Most were taken from each service's published pages on 8 October. A few come from our own posting tests, on feeds we connected ourselves. We re-read the public pages on 17 October, and the results below say which answer rests on which origin.

Results

Alone, five misses, identical for both models. Of 48 answers, 38 passed.

  1. Hashnode reads. Both said read-only queries are still free without a paid plan. They are not: an announcement dated May 2026 says every request, queries included, now needs a Pro plan (Hashnode). Oddly, when another question described a free publication whose requests had been refused since mid-2026, both picked the paid plan as the cause. Given the symptom, they reasoned their way there; asked directly, they believed the old rule.
  2. Meta posts after going Live. Both said that a month of test-phase posts to our own Page stays hidden and must be sent again. Meta's page says that such data turns visible to people outside the app's roles once it is switched (Meta).
  3. and 4. Which Pinterest board receives pins when the feed settings name none, asked twice with different boards.
  4. How soon the first pins appear.

The board answers show no rule at all. In the first question Sonnet chose the newest board and Haiku the first one listed. In the second they swapped: Sonnet took the first listed, Haiku the newest. Neither chose what we saw in our feeds, which is the board that comes first by alphabet. Pinterest's help page is silent on a default, so this rests only on our own observation.

Timing is a judgment call. Both models said about 24 hours. Pinterest promises content "within 24 hours" (Pinterest), yet pins showed up in under an hour on our feeds. The expected value is our observation set against the documentation, so a reader may fairly count this miss as the models being right about what is written. Sorting the five: Hashnode and Meta contradict what the vendors publish, the two board questions lean on nothing but our feeds, and timing is our observation set against the vendor's promise.

What they knew alone. They handled the mechanics that are written in docs or code. Both said posting through the X API is never free and put a post with a link at $0.20 a request; the pricing page lists $0.200 for that and $0.015 for a plain post (X pricing). They treated a Mastodon 202 on upload as success (Mastodon media), knew a token without the media scope gets refused, and knew a retry key lasts one hour (Mastodon statuses). On Bluesky they measured the length limit in graphemes, not in JavaScript string length, answered both byte-offset questions right and knew a link card's picture must be uploaded first. Our reading, as an interpretation and not a measured rate: they were right on written mechanics, wrong on a recent policy change, wrong on undocumented behaviour, and wrong on one rule that is documented but counter-intuitive.

With the skill, and where it hurt. Sonnet scored 24 of 24 and Haiku 23 of 24. Haiku still missed one board question, choosing Kitchen, the board listed first. The skill is social-api-reality-2026. The headline hides a harm. With its first wording, Sonnet failed both byte-offset questions it had passed alone. On the rocket example both ends landed two bytes late, 12 to 33 where 10 to 31 is correct; on the plain-text one the end went one byte too far, 6 to 28 where 6 to 27 is correct. We changed the Bluesky wording, and three re-runs of those two questions were all correct. Bare models, re-run three times as well, stayed correct. So Sonnet's 24 is a merge of runs: the first run supplies 22 questions, the re-runs the two offsets.

If you are building this

  • Read the service's own page the day you build; a plan from memory can be a policy behind.
  • Test for any success code in the 2xx range, not a single one.
  • Name the Pinterest board in the feed settings. Do not lean on any default.
  • Do not republish Facebook posts after the switch to Live.
  • Look at an empty Pinterest board the same day, not tomorrow.

What we did not measure

  • Runs. One run per model and question, apart from the byte-offset re-runs.
  • A fix checked where it was found. The new Bluesky wording was written after the two misses and checked on the same two questions, not on fresh ones.
  • Our questions. We wrote all 24. A different author would probe other corners.
  • Fixed options. Answers were JSON with set choices. In free text a model may know more, and with an "I am not sure" option it may guess better.
  • The Pinterest default and timing. Both rest on a few feeds we connected ourselves, not a count we can publish as a rate.
  • Date of the facts. We read them on 8 October and again on 17 October; they can change later. The Bluesky rate limits were not read again.
  • Coverage. We ran two models only. The test leaves out Telegram, TikTok, LinkedIn and Reddit, and we measured nothing about reach or clicks.

Read this post as Markdown: /blog/social-api-facts-models-get-wrong.md · Atom feed.