How to Control AI Crawlers With robots.txt

robots.txt is a plain text file that tells crawlers where they may go on your site. For classic search engines, that has mostly been the whole story. AI crawlers add two complications a lot of guides on this subject skip: what the file actually stops, which is narrower than most people assume, and which crawlers actually follow it, which is fewer than the ones that publish a name for themselves. This guide covers the file itself, in order: what it controls, how the underlying standard works, how to write and place one, which AI-crawler tokens are currently confirmed against each vendor's own documentation, which of those tokens are not crawlers at all, and how to check the result yourself once it is live. None of this covers whether an AI engine chooses to mention you once it can reach your pages; that broader, separate question is the subject of our guide to improving AI-search visibility.

1. Start with what robots.txt actually controls

Google states this plainly, and it is worth reading before writing a single line: a page disallowed in robots.txt "can still be indexed if linked to from other sites." Google's own robots.txt introduction goes further, calling the file "not a mechanism for keeping a web page out of Google" and warning directly against using it "as a means to hide your web pages from Google Search results." When that happens, the bare URL, and sometimes the wording of the links pointing to it, can still appear in results, just without a description.

Search Console has a name for the resulting state: "Indexed, though blocked by robots.txt." Google's own explanation is direct: "Google always respects robots.txt, but this doesn't necessarily prevent indexing if someone else links to your page," because "Google won't request and crawl the page, but we can still index it, using the information from the page that links to your blocked page."

The fix is not a robots.txt change. It is the noindex directive, delivered as either a meta tag or an X-Robots-Tag header, and it carries a requirement that trips people up. Google's own guidance on blocking indexing states it directly: "For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file...the crawler will never see the noindex rule, and the page can still appear in search results." A crawler has to fetch a page to read a tag written on it. Disallowing a page in robots.txt and adding noindex to that same page is a contradiction, not a stronger block, because the second instruction never gets read.

The action: decide, per page, which of two outcomes you actually want. If a page should stay out of results entirely, remove any robots.txt block on it and add noindex instead. If a bare URL appearing without a snippet is acceptable, robots.txt alone does that job and nothing more needs to change.

2. Learn the rules that decide what a crawler actually does with your file

RFC 9309 is the actual standard behind robots.txt, formalized by the IETF in 2022 from decades of informal practice, and Google publishes its own specification alongside it. A short list of rules from both decides almost everything that goes wrong in a hand-written file.

One detail worth a sentence of its own: a user-agent name, what the standard calls a product token, can only contain letters, underscores and hyphens. If you're copying a token from somewhere and it picks up a stray character, the rule silently stops matching.

The file-location row above has a consequence people miss. https://www.example.com/robots.txt says nothing about http://www.example.com, about https://example.com without the www, or about https://shop.example.com. A site split across a subdomain, an old http version still resolving, or a staging environment on a different port needs its own file, or consistent rules repeated across each one.

3. Create the file and put it where crawlers expect it

A robots.txt file is nothing more than plain text, so any text editor works; there's no special software and no build step. What matters is the name and the location. Per both RFC 9309 and Google's specification, the file has to be named exactly robots.txt, all lowercase, and it has to sit at the top level of the site: yourdomain.com/robots.txt, never inside a folder such as yourdomain.com/files/robots.txt. A crawler that doesn't find it at that exact path treats the site as having no rules at all, per the 4xx behavior in the table above.

If the site is managed through a host's file manager, an FTP client, or a static-site deploy pipeline, uploading a robots.txt to the root directory is the entire task. Generator tools for this file are advertised widely, and for a large, multi-section file they can save time, but a typical AI-crawler policy is rarely more than a few dozen lines. Typing it directly is usually faster than configuring a generic tool and leaves nothing to that tool's assumptions about what a given site needs. If a CMS is involved, check first whether it already generates a file automatically; WordPress does, and it's documented well enough to say exactly how that works and where it commonly causes confusion.

4. Handle WordPress's virtual robots.txt correctly

WordPress ships its own robots.txt logic, and it catches people who go looking for a file to edit and don't find one. When no physical robots.txt file exists in a site's root directory, WordPress's do_robots() function generates one on the fly, from memory, the moment anything requests /robots.txt. WordPress's own documentation confirms the condition directly: this virtual behavior applies only "if WordPress is installed in the site root and no robots.txt exists." There is nothing on disk to open in an FTP client, because the file being served does not exist as a file. Its default output disallows the admin path while explicitly allowing admin-ajax.php inside it, a narrow default unrelated to AI crawlers either way.

Two practical consequences follow. Uploading a real robots.txt file to the root directory fully replaces the virtual one; WordPress does not merge the two, so the uploaded file becomes the entire ruleset from that point on. And for anyone who would rather not manage a physical file, WordPress exposes a robots_txt filter hook specifically so a theme or plugin can add rules programmatically without one ever existing on disk.

One more WordPress-specific correction is worth stating plainly, because articles on this exact subject still get it wrong regularly. The "Discourage search engines from indexing this site" checkbox under Settings, then Reading, does not edit robots.txt, and hasn't since WordPress version 5.3. Checking it today adds a noindex, nofollow meta tag to pages, a page-level instruction, not a robots.txt rule. The robots.txt-editing version of that same checkbox is WordPress's own documented behavior, explicitly marked as retired "till version 5.2." A tutorial describing that checkbox as changing a site's robots.txt file is describing a WordPress version several years out of date.

The action: search the root directory for a physical robots.txt file before assuming WordPress's defaults apply, and if that Reading-settings checkbox is in use, treat it as the noindex control it currently is, not a robots.txt control.

5. Add rules for the AI crawlers that publish a token

Every major AI company that documents a crawler splits it by purpose, and the split matters more than the company name does. A token that gathers material to train a future model is a different line in the file from a token that fetches one page the moment a live user asks about it, which is different again from a token that crawls to build a search index. Blocking one doesn't block the others; a blanket User-agent: * / Disallow: / written years ago for an unrelated reason blocks all of them at once, often by accident.

The table below lists every AI-related token confirmed directly against a vendor's own current documentation, read on August 28, 2026, its stated purpose, and whether that vendor states it honors robots.txt.

Two rows need a flag before they're read as ordinary crawlers: Google-Extended and Applebot-Extended. Neither makes an HTTP request of its own. Google states this about its token directly: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings." Apple's wording for its equivalent token is just as direct: "Applebot-Extended does not crawl webpages...Applebot-Extended is only used to determine how to use the data crawled by the Applebot user agent." Both are usage-control tokens layered onto a crawl that already happened under a different, real crawler's name. Naming Google-Extended in a robots.txt file changes what Google may do with content Googlebot already fetched; it does not turn away a single request.

Two names that come up constantly in this subject publish nothing to put in the table above. Cohere has no reachable, official documentation describing a crawler, a token, or a robots.txt policy; if one exists, it hasn't been stated anywhere that could be confirmed here. ByteDance is different: a crawler commonly called Bytespider is widely reported in third-party sources, but ByteDance's own documentation page for it is not reachable, and no vendor-confirmed user-agent string exists to print here with confidence. Printing an unverified string for either company would mean a rule that might not match the real crawler's actual name, silently doing nothing. The honest entry for both is that there's nothing confirmed to add, not a guess dressed up as one.

One correction worth flagging directly: a token called FacebookBot circulates in some published AI-bot lists as a current Meta AI agent. It does not appear anywhere on Meta's own current crawler documentation. The five Meta tokens in the table above are the complete list that page names.

6. Know which of those crawlers don't fully promise to honor the file

Three of the rows above already say so themselves, and this deserves to be read as carefully as the crawling-versus-indexing point in step 1. OpenAI states directly that "because these actions are initiated by a user, robots.txt rules may not apply" to ChatGPT-User. Perplexity's own documentation says its equivalent, Perplexity-User, "generally ignores robots.txt rules," for the same reason: the fetch happens because a person asked a live question, not because a standing crawl decided to visit. Meta's documentation uses softer but still explicit language for two of its own tokens, FacebookExternalHit and Meta-ExternalFetcher: both "may bypass" robots.txt, the first for security and integrity checks, the second when fetching a link an agent is acting on at a user's request. None of this is a secret or a violation; each company put it in writing.

A second, separate category is disputed, not merely disclosed, and the distinction matters. Perplexity's documented, named crawler, PerplexityBot, is not what's in question here. Cloudflare's own blog, by Reid Tatoris and published August 4, 2025, describes a separate, undeclared crawler tied to Perplexity, using a generic browser-like user-agent and rotating through IP ranges outside Perplexity's published list, which Cloudflare says it observed specifically "in response to the restrictive robots.txt policy" on sites that had blocked the declared bot. Cloudflare removed Perplexity from its Verified Bots program over the finding. A year earlier, Wired reported a similar pattern, in a piece by Dhruv Mehrotra and Tim Marchman published June 19, 2024: a crawler tied to Perplexity fetching pages that had disallowed it, under an undeclared identity. Perplexity's own response, given to TechCrunch, called the Cloudflare post "a sales pitch" and said the specific bot Cloudflare named "isn't even ours," disputing the attribution of that traffic, not the pattern itself.

Bytespider carries its own dated reporting. Fortune reported in October 2024 that research it covered found the crawler "does not respect robots.txt." A TollBit industry report covering the first half of 2026, as described in Search Engine Journal's coverage by Matt G. Southern on August 14, 2026, found that Bytespider and a separate crawler called Youbot each reached pages disallowed against them on close to half of the European sites that had specifically listed them by name, a materially higher rate than the same report found for most other named crawlers. The same report's ChatGPT-User figure isn't a new finding on top of this; it lines up with what OpenAI already discloses about that token in the table above.

The action: for any crawler this guide discloses may not honor a block, understand robots.txt as the extent of what a text file can do, a stated policy, not an enforcement mechanism. Stopping traffic that has a specific, credible reason to be suspected of misrepresenting its own identity means a server or CDN-level rule keyed on IP ranges or behavior, not a more strongly worded robots.txt line.

7. Put the confirmed tokens into one file

Everything above is enough to write a real file. The example below blocks the crawlers whose stated purpose is training a model, allows the crawlers whose stated purpose is answering a live question or building a search index, and leaves the two regular search crawlers, Googlebot and Applebot, untouched. It uses only tokens confirmed in the table above.

# Regular web search: untouched by the rules below
User-agent: Googlebot
Allow: /

User-agent: Applebot
Allow: /

# Crawlers whose stated purpose is training a model
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: MistralAI-Training
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Crawlers whose stated purpose is answering a live question or building a search index
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: MistralAI-Index
Allow: /

User-agent: MistralAI-User
Allow: /

User-agent: DuckAssistBot
Allow: /

ChatGPT-User and Perplexity-User are left out on purpose. Both fire only when a live user session asks about a specific page, and both vendors say directly that a standing robots.txt rule may not govern that kind of request anyway, unlike Claude-User, which Anthropic documents as honoring the file the same way its other two agents do. Meta's five tokens are left out too: only two of them state a compliance posture at all, and both of those are a qualified "may bypass," which makes a confident Allow or Disallow line more certain than the vendor's own documentation is. Treat this file as a starting policy, not a finished one. Change any Disallow to Allow for a training crawler that's acceptable to feed, and revisit step 6 before assuming a Disallow line alone has settled anything for a crawler with reported non-compliance.

8. Verify the result yourself

Every check below runs on tools the vendors themselves publish, and none of them asks for trust in this page.

Start with the plainest possible check:

curl -s https://yourdomain.com/robots.txt

This prints the exact bytes a crawler receives, bypassing any browser cache or rendering. Add a declared identity to test how the server treats a specific bot:

curl -A 'GPTBot' -I https://yourdomain.com/robots.txt

This sends the request as that user-agent, which catches something a plain fetch can't: a CDN or firewall rule serving different content, or blocking the request outright, based on the user-agent string alone. Keep the limit of this check in view, especially after step 6: curl only proves what a server does when a client is honest about its identity. It says nothing about whether the real crawler is, since some of the crawlers discussed above have been reported switching to an undeclared identity specifically to get past a block like this one.

For a check closer to what Googlebot itself runs, Google publishes its actual parsing code as an open-source library, described on its own repository page as "Google's robots.txt parser and matcher," the real logic behind the crawler, not a reimplementation of it. It ships a small command-line tool: give it a robots.txt file, a user-agent, and a URL, and it returns ALLOWED or DISALLOWED, the same way Googlebot's own code decides it. No neutral, standards-body-run validator for RFC 9309 turned up in this research; this is the closest thing to an official one, published by the company that helped write the standard.

Search Console's robots.txt report shows which file Google found for a site, when it was last fetched, and any warnings, for up to the top 20 hosts under a property. One thing worth knowing before going looking for it: Google retired the older, interactive robots.txt Tester, the version that let a person test one URL against one user-agent directly, when this report replaced it. Trade-press coverage at the time dated the change to December 2023; that exact date could not be independently confirmed against a Google-owned page, but the broader fact holds on a live check today: the interactive, per-URL tester is gone, and the report is what remains.

One more layer, specific to the disputed-compliance question in step 6: most of the vendors in the step 5 table publish their crawlers' real IP ranges as JSON, among them OpenAI's gptbot.json and three sibling files, Anthropic's bots.json, and per-token files from Perplexity and Mistral. Matching a request in server logs against the published range confirms it actually came from the vendor's real infrastructure, not just a user-agent string that anyone can type.

For a faster read than any of the above, several tools run the same checks and hand back a result. Is My Brand In AI gives a fast, curated, no-signup read of the crawler families that matter most. AIclicks validates against the same open-source Google parser referenced above. BrandCited sweeps the widest list, roughly 64 agents, for anyone who wants the long tail. Hyperleap pairs the same access check with a real syntax validator that flags a malformed file, not just a blocked bot. xSeek turns the result into a single weighted readiness score. And for checking hundreds of domains on a schedule instead of one site by hand, the Apify robots.txt Actor runs the same audit as a programmable, batch job with change detection built in. None of the six edits a file automatically, and none of them can see a firewall or CDN rule sitting in front of a server; they read exactly what steps 1 through 7 just covered how to write.

<!-- Source materials for this guide (implementer reference, not for the reader-facing page): - Data/seo/briefs/robots-txt-for-ai-crawlers-facts.md — every factual claim above traces to this file, read 2026-08-28, with confidence markers honored (MEDIUM claims hedged in body + duplicated into unconfirmedSteps where the brief instructed it). - Data/seo/briefs/robots-txt-for-ai-crawlers-brief.md — structure, demand-family weighting, and house rules. - content/guides/improve-brand-visibility-in-ai-search.md — sibling guide in this pillar, already links forward to this page's slug; read in full to keep sentence-level phrasing distinct on overlapping topics (crawling vs indexing, Google-Extended) and to place one reciprocal link back to it. - content/reviews/{ismybrandinai,aiclicks,brandcited,hyperleap,xseek,apify-robots-checker}.md — the six bot-checker reviews linked in step 8, each with a distinct, non-decorative reason for the link; also cross-checked against the facts brief's Section 7 findings (stale anthropic-ai/Claude-Web and FacebookBot tokens) so this guide doesn't repeat those errors. - site/lib/guides.ts and site/app/_pending/guides/[slug]/page.tsx — confirmed the exact frontmatter contract (no author/authorKey field is read for guides; byline is hardcoded to mucahit-kaya in the template) and that the body renders with no separate H1 or TL;DR block. -->