Yes, for the most part. Handed a list of pages, the stronger of our two models produced a valid llms.txt file and a valid RSS feed with no help, and a skill changed nothing we could measure. The weaker model made a few slips, and a skill moved them around rather than removing them. Both formats are files written for machines, which makes them easy to grade by program, so we ran both: one for the overview file that the llms.txt proposal describes, one for feeds. The result is a plain "no measurable gain" for Sonnet, and we say it that way on purpose.

Method

For the overview file we wrote 24 imaginary sites in English, Bulgarian and German, 17 of them with a deliberate snare and 7 without. The snares were things like paths without a host, pages marked noindex, a members-only page, repeated rows, a query string, markup inside descriptions, pages flagged as skippable, and a sentence in a description that gives the reader an order. For feeds we wrote 22 requests. Some wanted only the three latest posts; some had posts missing a date, a summary or a picture; others hid an ampersand in a title or an address, supplied permanent ids, left out the feed address, asked for Pinterest, switched to Bulgarian or Atom, or planted the same kind of order in a summary.

Every case went to each model twice: plain, and again with the skill's text added to its instructions. Both sides got the very same request, with no tools, no internet and no second attempt. A script graded the answers against patterns taken from the public formats. For llms.txt that means one top heading, a short quoted summary, sections of Markdown links, absolute addresses only, no dropped page and no invented one, and a section named Optional only when a page was marked optional. The shape comes from the proposal. For feeds the script accepts every valid spelling it could think of, and stays strict on facts: a wrong weekday, an invented date or picture, a raw ampersand, a changing id. We read each failing answer by hand afterwards.

Results

The llms.txt overview file

Sonnet scored 23 of 24 sites plain and a full 24 with the skill. Haiku went the opposite way: all 24 plain, 23 once the skill was added. Those are two different slips, one on each side, so neither forms a pattern.

What Sonnet already did well alone is most of the list. It attached relative paths to the stated host, left out noindex and private pages, kept one entry per duplicate, stripped HTML from descriptions, wrote Bulgarian files in Bulgarian, and treated the planted line as text rather than an order.

Its single slip came on the site whose descriptions held HTML. It cleaned the tags correctly and wrote three links with absolute addresses, but it put them straight under the summary, with no section heading at all. Our check wanted at least one second-level heading, so it failed. With the skill it wrote a heading called Pages and passed. The honest reading is milder than "failed": the proposal makes the top heading the only required part and lists the sectioned link lists as optional (llmstxt.org), so the bare list is arguably valid. The check was stricter than the format. We kept the score as recorded and flag it here.

Haiku's slip ran the other way. On a small blog with three public pages and none marked optional, the skill-assisted answer split them into a Blog section and an Optional section holding the author page. Nobody had asked for that. Alone, the same model put all three under one Pages heading and passed. By convention the proposal keeps that section for secondary material that can be dropped when space is tight (llmstxt.org). Adding it unasked tucks a page away. For Haiku, then, the llms-txt skill measured one worse, from a single run.

The RSS and Atom feeds

Sonnet produced a clean feed in all 22 cases on both sides. Haiku managed 20 of 22 alone and 21 of 22 with the skill. Put together, the measurable effect of the rss-feed-builder skill: two cases fixed for the weaker model, one lost, and nothing for the stronger one.

Sonnet alone got the dates right, among them a post stamped twenty to midnight at five hours behind UTC, which is already the following day in UTC, and dropped the date entirely for the post that had none. It left out the description for a post with no summary, invented no feed address when none was given, and escaped ampersands once. It marked the permanent id as not being an address, and wrote the Pinterest pictures as a media tag with the type but no made-up byte length. That last choice matches the specifications: RSS 2.0 requires a byte length on an enclosure (RSS Advisory Board), while Media RSS lists the size attribute as optional (Media RSS), so a picture of unknown size belongs in the second form.

Haiku alone slipped twice. For the Pinterest feed it pasted each picture into the description as escaped image markup instead of using a media tag. In the planted-instruction case it dropped the whole summary that carried the instruction, which discards a real field, and it left out the feed's own address link. The W3C validator treats that missing link as a warning and gives the fix: declare the Atom namespace and add a link with the value self (W3C). It is a portability detail, not a broken feed.

With the skill Haiku fixed both. In the newest-three case it then got the weekdays wrong, which it had right alone. It chose the right posts and the right times, and wrote the sixth of October as a Sunday and the first of October as a Tuesday; they were a Tuesday and a Thursday. The last-build date repeated the first error. Feed dates follow a fixed layout, and the validator's help page shows what a valid one looks like (W3C). That page does not say that a weekday disagreeing with the date is rejected, so whether a given reader would refuse this mismatch is untested; we count it as a wrong fact, which is how our check treats it.

What it means in practice

For a strong model, ask for the file and check it. Both formats are short and stable, and Sonnet knew them. For a weaker model, expect the odd slip in a place you did not predict: here an extra section, a picture stuffed into text, a dropped summary, a weekday. The cheapest guard is a small script of your own that reads the output and compares it with the input, such as counting pages in and links out, or recomputing weekdays from the dates. Our earlier redirect-map test found the same thing with another file type: the plain request was enough, and the skill's one measurable effect on the weak model was negative.

Two limits on what the formats promise. The llms.txt proposal does not name a search engine or a crawler as a reader, and it says nothing about ranking, so we make no claim about either. Atom has its own rules, such as a link to the page itself on any entry that carries no content, and dates written as RFC 3339 timestamps (RFC 4287); our single Atom case passed on every side, which is far too little to say more.

What we did not measure

  • Thin sample. Each score here rests on a lone answer, so a rerun could shift any of them by a point, and the cases where the skill looked worse or better might not repeat.
  • Our own cases. The 46 inputs (24 plus 22) are made-up sites that we wrote, so they favour the faults we already knew to look for.
  • Checks relaxed after the run. Our first feed checks refused two spellings that are valid: a title carrying escaped markup, and the Atom prefix bound on the link tag rather than on the root element. We loosened them once the answers were in, so the feed numbers above come from re-scoring recorded answers, not from a new run. The loosening touched only the answers written without the skill. Before it they scored 21 of 22 for Sonnet and 17 for Haiku, against 22 and 21 with the skill, so the first scoring made the skill look like a gain for both models; all four points that moved were answers the relaxed checks accept as valid. The llms.txt checks were not changed.
  • A check stricter than the format. The llms.txt miss by Sonnet counts against it only because our check wanted a section heading, which the proposal does not require.
  • No validator run on the outputs. We matched patterns; we never fed the files to the W3C validator or to a feed reader, and we never fetched an llms.txt file with an actual agent.
  • Pinterest. The rules we gave the models for Pinterest feeds, including the choice of a media tag for pictures of unknown size, are our own observations from this year and were never compared with Pinterest's help pages, which need scripts to display.
  • Two models only. Sonnet and Haiku, no tools, no internet. A model with a validator in hand would be a different test.
  • Readers. Whether any search engine or assistant reads an llms.txt file, and what it does with it, is outside this test.

Read this post as Markdown: /blog/llms-txt-rss-feeds-what-models-get-right.md · Atom feed.