The aiskills402-read bot

The full user agent string, which we never change or hide:

aiskills402-read/1.0 (+https://aiskills402.com/docs/web-to-markdown)

• One GET of /robots.txt, then one GET of the page address it was given. Nothing else: no POST, no crawling of other pages, no image or script downloads.
• Two headers only: user-agent and accept. Nothing from our customer is passed on: no cookies, no headers, no credentials. The requests are built from scratch every time.
• Every request carries this name. If your server answers 403, 401 or a challenge, the bot gives up and the customer is not charged; it never retries without its name and never works around a block or a CAPTCHA.
• It does not run JavaScript. A page that only shows content after scripts run returns nothing readable, and that read is not charged either.

The bot looks for a group for aiskills402-read first, then for *, and follows the longest matching rule, as in RFC 9309. If robots.txt forbids the page, the bot does not fetch it. If robots.txt cannot be read (a server error), it does not fetch the page either. To keep it out of your whole site:

User-agent: aiskills402-read
Disallow: /

• At most 120 pages per hour from the same host, however many customers ask. A host reached through a redirect counts too.
• At most 5 redirects are followed; each new address is vetted again, and a redirect to another host reads that host's robots.txt first.
• Pages over 2 MB are refused, not truncated. Each request is cut off after 10 seconds, the whole read after 15.
• Only https on port 443 and DNS names. IP addresses, private names and our own hosts are refused before anything is sent.
• Only HTML and plain text come back (also JSON, XML and feeds as raw text); PDFs, images and other files are refused.

The requests come from Cloudflare's network, the same shared addresses every Cloudflare Worker uses. Blocking those addresses blocks far more than this bot; block the user agent instead.

The text goes to the customer who paid for the read. We keep one log line per read with your host name, the status code and the number of bytes. We do not keep the path, the query, the page content or your headers.

• Add the robots.txt group above, or block the user agent aiskills402-read in your firewall or WAF. The bot records the refusal and does not try another way in.
• Or write to hello@aiskills402.com with your host name, and we will exclude it.

We return the text of someone else's page exactly as it is. A web page can contain text written to manipulate an AI agent (prompt injection): hidden instructions, fake system messages, requests to send data somewhere. We do not follow them, because there is no language model in this service, only code that converts HTML. Your agent must treat the returned content as untrusted data, never as instructions: quote it, summarise it, but do not obey it, and do not let it change which tools your agent calls or where it sends anything.

web-to-markdown is a pay-per-call API for AI agents: $0.005 USDC per page, paid over x402, and you pay only when the page text is returned. Refusals (robots.txt, a block, a page too large, no readable text) cost nothing. The free example, with its input schema and a recorded result, is at https://api.aiskills402.com/v1/run/web-to-markdown/example. Paying for calls in general: pay-per-call APIs.