robots.txt builder
Say who may crawl your site and where. The file prints itself as you decide, and every line is explained. Nothing you type here goes anywhere. Not even to us.
Search yes, scrapers no.
Eleven AI training crawlers are turned away. Everyone else may fetch everything.
Paste a robots.txt on the right and every line gets read back to you.
The file you have
Paste the contents of a robots.txt. It is read here, in your browser, and explained one line at a time: what each directive does, who it applies to, and the things that are usually wrong with these files.
3 GROUPS 7 RULES 2 SITEMAPS 184 BYTES
How a crawler reads this file
A crawler fetches /robots.txt before anything else, then looks for the group whose User-agent matches its own name most closely. Only that one group applies; if there is none, it uses the * group, and if there is none of those either, everything is allowed. Inside the group, the rule with the longest path that matches the URL wins, and when an Allow and a Disallow are exactly as long, Allow wins. That is why Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/ for that one file.
What it is not
It is not access control. A blocked path is still served to anyone who asks; robots.txt is a request that well-behaved crawlers choose to honour. It does not keep a page out of search results either: Google can still list a blocked URL, with no snippet, if other pages link to it. To keep a page out of the index, let it be crawled and put <meta name="robots" content="noindex"> on it, or return an X-Robots-Tag: noindex header. For anything private, use a password.
Wildcards and the dollar sign
Paths match from the start of the URL path, so /admin also covers /administrator and /admin/anything. A * stands for any run of characters and a $ pins the end of the URL: /*.pdf$ means every URL that ends in .pdf, /*? means every URL with a query string. Paths are case-sensitive, so /Admin/ and /admin/ are different doors.
Blocking AI training without losing search
The training crawlers and the search crawlers use different names, so you can turn one away and welcome the other. GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent and Amazonbot collect pages for model training; OAI-SearchBot and Googlebot fetch them for search results; ChatGPT-User and Claude-User fetch a page because someone asked about it in a chat. Google-Extended is the odd one: it is not a crawler, it is a switch that tells Google whether the copy Googlebot already has may train Gemini. Blocking it costs you nothing in Search.
The crawlers, by name
Names are matched case-insensitively and by product token, so googlebot and Googlebot are the same, and a group for Googlebot also covers Googlebot-News unless that one has a group of its own. This is who is behind each name on the list, and what turning it away actually does.
| Name | Who it is | Blocking it means |
|---|
What stays here
Everything on this page is plain JavaScript in your browser. There is no server building the file, so there is nothing to send it to. The rules are stored in this browser only, and a shared link carries the whole file in its address. The only thing this page records is that a button was pressed, and which one. Never your rules or your domain.
Sources
The rules of reading are from RFC 9309, the Robots Exclusion Protocol (2022), which is what the notes above mean by "longest match wins". Google's behaviour, including which lines it ignores and the 500 KiB limit, is from Google's robots.txt specification and its introduction to robots.txt. Crawler names and what they are for come from the vendors' own pages, which change; check the newest one if a name matters to you.