# Robots.txt Policy Writer Robots.txt Policy Writer is a tested SKILL.md that turns a crawler policy written in words into a correct robots.txt, answered as the file only; an agent buys it once for $0.03 over x402. - Page: https://aiskills402.com/skills/robots-txt-policy - Category: SEO & Content (https://aiskills402.com/categories/seo) - Price: $0.03 once, USD-priced, paid in USDC on Base over x402. Price as loaded on this page. The 402 response your agent receives is authoritative. - Version: 1.0.0 - Card (JSON): https://api.aiskills402.com/v1/skills/robots-txt-policy ## Use it when Turns a crawler policy written in words into a correct robots.txt, answered as the file only. Knows the mistakes that make a file do the opposite of what was meant - a crawler obeys only the one group that names it, so a GPTBot group silently drops every rule written for the star group and the private paths must be repeated; longest match wins and Allow wins a tie; Google ignores Crawl-delay; Host is a Yandex-only line; Sitemap must be an absolute address; Google-Extended is a control token for Gemini training and does not touch Google Search; training, search and user-triggered AI bots are separate tokens. Says in a NOTE comment what robots.txt cannot do, such as removing a page from Google. Use when asked to write, check or fix a robots.txt, to allow or block search engines, AI crawlers or AI training bots, or to keep private paths out of crawlers. ## Not for Checking a live site's file or its indexing in Search Console, removing pages from Google, or blocking bad bots by IP. It writes the file from the policy you describe; it cannot see what the site serves today. ## Tested, honestly Tested 2026-10-08. - Strong model (claude-sonnet-5-5 (Claude Code alias "sonnet")): Wrote a correct robots.txt in all 22 policies: the private paths repeated in every group that names a bot, GPTBot blocked while OAI-SearchBot stays allowed, Google-Extended blocked without touching Googlebot, no Host line, absolute sitemaps, longest-match exceptions for a search folder, PDFs and WordPress, and a comment instead of a pretend rule when the policy asked to remove a page from Google, to slow Google down, or to cover a staging host and a Cloudflare setting. - Weak model (claude-haiku-5-5 (Claude Code alias "haiku")): Got 20 of 22 right. It wrote a Host line when the policy asked for a preferred host, and for a page the policy asked to block but keep out of Google results it left the page open and explained noindex in a comment instead of writing the Disallow. The group rule, the AI bot tokens, the sitemaps and the pattern cases were right. Note: Twenty-two crawler policies written by us, all in English: about two thirds with a trap (a bot group that must repeat the private paths, training blocked while AI search stays open, Google-Extended against Googlebot, a Host request, a relative sitemap, a request to remove a page from Google, a staging host, longest-match exceptions) and the rest plain policies where the obvious file is right. Each answer was parsed as a robots.txt by the RFC 9309 rules (the most specific group replaces the star group, longest match wins, Allow wins a tie, * and $) and every named bot was asked whether it may fetch the stated paths; sitemaps had to be absolute, a Host line was refused, and cases that ask for something robots.txt cannot do also needed a comment that says so. The rules and bot names in the skill were read on the vendors' own pages on 8 October 2026. One run per model and policy. ### With and without the skill Tested 2026-10-08. - Files that do what the policy says (22 policies): Sonnet 22/22 with, 21/22 without; Haiku 20/22 with, 18/22 without. The same request on both sides, asking for the robots.txt only; a fence around the answer is removed first, and the comment check looks for a comment that says the thing, not a fixed label. Sonnet already writes nearly every policy right alone, group rule included, so its measurable gain is one line: without the skill it wrote a Host line in the one case that asked for a preferred host. Haiku without the skill also wrote Host, blocked a page the policy wanted removed from Google, opened the private folder to the user-triggered agents, and gave no noindex comment. With the skill the first two were fixed and the comment was written, yet it wrote Host again and left one page open that the policy asked to block. A strong model gains little here; the smaller one gains more. Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise. Full summary: https://aiskills402.com/skills/robots-txt-policy/tests ## Example ### English (claude-sonnet-5-5) Before: > Every crawler may crawl the whole site except /admin/ and /account/. In addition GPTBot must stay out of /pricing/. After: > User-agent: * Disallow: /admin/ Disallow: /account/ User-agent: GPTBot Disallow: /admin/ Disallow: /account/ Disallow: /pricing/ ## What is in the file - The answer - The rules that decide what a file means - AI crawlers: three purposes, separate tokens - A setting outside the file - Work in this order - Short examples ## Languages Any language ## How to buy Agent (HTTP): 1. GET https://api.aiskills402.com/v1/skills/robots-txt-policy/file without a payment header. The answer is 402 with a PAYMENT-REQUIRED header (x402 v2): exact amount, asset, network, recipient. 2. Sign `accepts[0]` with an x402 client (for example @x402/core + @x402/evm). 3. Repeat the GET with the signature in the PAYMENT-SIGNATURE header. The answer is 200 with the file, its sha256 and a re-download token. Agent (MCP): https://mcp.aiskills402.com/mcp — free tools search_skills, get_skill, redownload_skill. Buying itself is over HTTP. Full flow: https://aiskills402.com/docs ## The file - Version: 1.0.0 - Size: 8.0 KB (8188 bytes) - SHA-256: 53cad17f2c6a5ec28656fcac5ae8e11c5a80e99920ef5b364aeeb7aa61ff4bdf - Updated: 2026-10-08 - New versions are free through your re-download token. ## Versions ### 1.0.0 (2026-10-08) First release: turns a crawler policy in words into a robots.txt, answered as the file only. Covers the group rule (a crawler obeys only the group that names it, so private paths are repeated in every named group), longest match with Allow winning ties, wildcards, Google ignoring Crawl-delay, no Host line, absolute Sitemap addresses, one file per host, the split between blocking crawling and indexing (a NOTE comment instead of a fake fix), AI crawlers by purpose (training, search, user-triggered) with Google-Extended as a control token, and a check of Cloudflare's managed robots.txt setting. ## License Perpetual, non-exclusive; use and modify for yourself incl. paid work; no resale or republishing. Holder: Georgi Kalchev, aiskills402.com. Terms: https://aiskills402.com/docs#license ## FAQ ### Is the reply only the file, or do I get an explanation too? Only the robots.txt, ready to save at the root of the host. Where your policy asks for something a robots.txt cannot do, such as taking a page out of Google, the file carries a NOTE comment that says so and names the right tool. ### What is the most common mistake it prevents? Giving one bot its own group. A crawler follows only the group that names it, so a new GPTBot group quietly cancels every rule written for all crawlers, and the private folders become open to that bot. The skill repeats the private paths in each named group. ### Can I block AI training but stay in AI search? Yes. Training, search and user-triggered bots use separate names, so GPTBot can be blocked while OAI-SearchBot stays allowed. Google-Extended is only a control token for Gemini training and does not change Google Search; blocking Googlebot would. ### Is it worth buying for a strong model? Only a little. Across 22 policies Sonnet got 21 files right unaided and 22 with the skill; the gain was one Host line. Haiku went from 18 to 20 and still wrote a Host line once. The skill helps most a smaller model, or an agent that should explain in comments what robots.txt cannot do. ## Related skills - [SEO Meta Writer](https://aiskills402.com/skills/seo-meta.md): $0.03 once - [Workers Pitfalls Review: D1, OpenNext, Fetch](https://aiskills402.com/skills/workers-pitfalls-review.md): $0.05 once - [Cron Schedule for Cloudflare Workers](https://aiskills402.com/skills/cron-schedule.md): $0.06 once ## Measurement limits - Models other than the two named above were not run. - Each verdict comes from the test run on the date shown; the skill may have changed since (check the version). - Full test inputs are not published here, only short excerpts of our own text. - Results on your own texts, languages and domains can differ. Offer note: Paid in USDC (USD-pegged) over x402 by an AI agent; one-time.