Turns a crawler policy written in words into a correct robots.txt, answered as the file only. Knows the mistakes that make a file do the opposite of what was meant - a crawler obeys only the one group that names it, so a GPTBot group silently drops every rule written for the star group and the private paths must be repeated; longest match wins and Allow wins a tie; Google ignores Crawl-delay; Host is a Yandex-only line; Sitemap must be an absolute address; Google-Extended is a control token for Gemini training and does not touch Google Search; training, search and user-triggered AI bots are separate tokens. Says in a NOTE comment what robots.txt cannot do, such as removing a page from Google. Use when asked to write, check or fix a robots.txt, to allow or block search engines, AI crawlers or AI training bots, or to keep private paths out of crawlers.
Robots.txt Policy Writer is a tested SKILL.md that turns a crawler policy written in words into a correct robots.txt, answered as the file only; an agent buys it once for $0.03 over x402.
Checking a live site's file or its indexing in Search Console, removing pages from Google, or blocking bad bots by IP. It writes the file from the policy you describe; it cannot see what the site serves today.
Tested 2026-10-08 with a strong and a weak model.
| Sonnet | Haiku | |||
|---|---|---|---|---|
| with | without | with | without | |
| Files that do what the policy says (22 policies) | 22/22 | 21/22 | 20/22 | 18/22 |
The same request on both sides, asking for the robots.txt only; a fence around the answer is removed first, and the comment check looks for a comment that says the thing, not a fixed label. Sonnet already writes nearly every policy right alone, group rule included, so its measurable gain is one line: without the skill it wrote a Host line in the one case that asked for a preferred host. Haiku without the skill also wrote Host, blocked a page the policy wanted removed from Google, opened the private folder to the user-triggered agents, and gave no noindex comment. With the skill the first two were fixed and the comment was written, yet it wrote Host again and left one page open that the policy asked to block. A strong model gains little here; the smaller one gains more.
Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.
Our own test text, before and after the skill ran. Excerpts only.
Before
Every crawler may crawl the whole site except /admin/ and /account/. In addition GPTBot must stay out of /pricing/.
After
User-agent: * Disallow: /admin/ Disallow: /account/ User-agent: GPTBot Disallow: /admin/ Disallow: /account/ Disallow: /pricing/
Any language. Tried in: English.
Perpetual, non-exclusive; use and modify for yourself incl. paid work; no resale or republishing. Holder: Georgi Kalchev, aiskills402.com. Full terms.
Current version 1.0.0, updated 2026-10-08. Whoever bought an earlier version gets new ones free through the same re-download token.
v1.0.0 · 2026-10-08
First release: turns a crawler policy in words into a robots.txt, answered as the file only. Covers the group rule (a crawler obeys only the group that names it, so private paths are repeated in every named group), longest match with Allow winning ties, wildcards, Google ignoring Crawl-delay, no Host line, absolute Sitemap addresses, one file per host, the split between blocking crawling and indexing (a NOTE comment instead of a fake fix), AI crawlers by purpose (training, search, user-triggered) with Google-Extended as a control token, and a check of Cloudflare's managed robots.txt setting.
Only the robots.txt, ready to save at the root of the host. Where your policy asks for something a robots.txt cannot do, such as taking a page out of Google, the file carries a NOTE comment that says so and names the right tool.
Giving one bot its own group. A crawler follows only the group that names it, so a new GPTBot group quietly cancels every rule written for all crawlers, and the private folders become open to that bot. The skill repeats the private paths in each named group.
Yes. Training, search and user-triggered bots use separate names, so GPTBot can be blocked while OAI-SearchBot stays allowed. Google-Extended is only a control token for Gemini training and does not change Google Search; blocking Googlebot would.
Only a little. Across 22 policies Sonnet got 21 files right unaided and 22 with the skill; the gain was one Host line. Haiku went from 18 to 20 and still wrote a Host line once. The skill helps most a smaller model, or an agent that should explain in comments what robots.txt cannot do.
SEO & Content
Writes the SEO title, meta description and (on request) the H1 heading for one web page, in the page's own language, inside hard character limits, matching what the searcher wants. Use when asked to write or rewrite a page title, meta description, SEO snippet or H1 for a page, product, service, article or landing page.
Code & Engineering
Reviews pasted Cloudflare Workers code and configuration (a Worker, a Next.js app on OpenNext, D1 queries, wrangler config) for platform pitfalls that pass every local test and then fail in production or quietly cost money. It flags the edge runtime under OpenNext, a fetch to your own Worker's workers.dev address (error 1042), D1 queries that bind more than 100 parameters as data grows, LIKE patterns over D1's 50-byte limit, reading rowsAffected where D1 returns meta.changes, interactive transactions D1 does not have, secrets kept in plain vars, outbound fetch code that treats only a thrown error as failure, a browser User-Agent that bot protection challenges, client hop-by-hop headers forwarded to fetch, and unbounded queries on the request path. Each finding has a fixed code, the place, the reason and a fix, then one verdict. Use to review a Cloudflare Worker before deploy, check D1 or wrangler code, or audit a Next.js app running on Cloudflare.
Code & Engineering
Turns a schedule described in words into cron expressions for Cloudflare Workers Cron Triggers, answered as JSON with the crons and a note. Knows where Cloudflare differs from classic cron, measured on a live Worker - day-of-week 1 is Sunday, so 1-5 runs Sunday to Thursday and 7 is Saturday, while 0 and ? are rejected; times are UTC only. Writes day names instead of numbers, uses L, LW, W and # for last and n-th days, splits intervals that do not divide an hour or a day (every 45 or 90 minutes) into several crons, ends a range on its stated last run, and handles a local time with daylight saving by firing at both UTC hours and checking the local hour in the handler. Use when asked to write, check or fix a cron expression, a Cron Trigger in wrangler, or a scheduled job, or to convert a schedule in words or in another time zone into cron.