Apify Robots.txt AI Checker Review (2026): Programmable Bot Audits, Pricing & the One-Off Trap

What the Apify Robots.txt AI Checker is

The Apify Robots.txt AI Checker is not a website you visit and a box you paste a URL into. It is an Apify Actor, a packaged program that runs on Apify's automation-and-scraping platform. You give it a list of domains, it fetches each site's robots.txt, parses the rules, and reports whether the major AI crawlers are allowed or blocked, returning the result as structured JSON.

That framing is the single most important thing to understand before judging it. Most AI bot checkers are instant, browser-based, no-signup pages built for a human checking one site. This one is built for a developer checking many sites, on a schedule, inside a pipeline. It's the same underlying question (is this AI crawler allowed in robots.txt?) answered with very different machinery.

The trade-off is real and goes both ways. As an Actor it gains things a browser tool structurally cannot offer: batch runs across hundreds of domains, scheduled monitoring, API access from Python/Node.js/cURL, and stateful change detection that flags when a site's policy shifts between runs. It also loses the thing a browser tool does best: an instant, free, zero-friction answer for one URL.

Which AI crawlers it covers

This is where the Actor is strong. It checks the AI user-agents that matter for generative-engine visibility, including:

• OpenAI: GPTBot, ChatGPT-User, OAI-SearchBot
• Anthropic: ClaudeBot, Claude-Web, anthropic-ai

Two of those three Anthropic tokens are not on Anthropic's own crawler page, which we read on August 28, 2026: it documents ClaudeBot, Claude-User and Claude-SearchBot, and neither anthropic-ai nor Claude-Web appears on it. Anthropic publishes no retirement notice for them, so absence from the page it maintains is the whole of the evidence. The practical consequence is the same either way: an Actor run that reports your Claude coverage as complete is checking two rules current Claude traffic does not match, and is not checking the two agents that fetch pages when a user asks Claude a question.

• Google: Google-Extended (the AI-training opt-out token, separate from Googlebot)
• Common Crawl: CCBot (the corpus many models train on)
• ByteDance: Bytespider
• Perplexity: PerplexityBot

One honesty note worth keeping: the Actor's own primary specification names GPTBot, ClaudeBot, anthropic-ai and PerplexityBot explicitly, and describes an "all-AI-bot policy snapshot" capability for the rest. The broader list above reflects the listing's documented coverage rather than something we confirmed agent-by-agent on a live run. So if a specific bot is decision-critical for you, verify it against your actual run output. Either way, the coverage spans both the crawling agents (GPTBot, ClaudeBot, CCBot) and the training-permission tokens (Google-Extended, anthropic-ai), which is the distinction that actually governs whether your content can feed and surface in AI answers.

How it actually runs

You run the Actor one of two ways: from the Apify Console (a web UI where you fill in inputs and click Start) or via the Apify API (the developer path: call it from a script, a cron job, or a CI step). Inputs include the domain list (the listing caps a run at 500 domains, and a demo mode limits you to one for kicking the tires), concurrency settings, and a delivery method (Apify dataset or a webhook).

Output is per-domain JSON: each entry carries the domain, a status, a summary block (counts of blocked / allowed / partially blocked bots), an aiPolicies array with each bot's disposition, any detected changes since the last run, and a timestamp. That structure is the point: it's designed to be parsed and acted on by other software, not just read by a person.

The change-detection feature deserves a callout because none of the browser-based checkers we cover has it. Because an Actor run can compare against a stored baseline, it can tell you a site that used to allow GPTBot now blocks it, a genuinely useful signal if you're monitoring publishers, competitors, or your own properties over time, and the kind of thing you'd otherwise have to build yourself.

Disclosure

AI Tools Police earns affiliate commissions when readers sign up for some tools we cover. That never changes a score, a capability claim, or whether we surface a limitation. We sell no bot checker and no AI-visibility product of our own, which is exactly why an independent reference like this can call a capable developer tool "the wrong fit for a non-technical one-off" without flinching.

How we reviewed this

This review is built on the Actor's public Apify listing, its documented inputs, outputs and pricing, and corroborating search results for the same Actor. We did not run a hands-on test, fabricate a metric, or produce a screenshot, and we did not execute a paid run to verify per-bot behavior. Coverage and pricing figures are presented as documented on the listing and labeled as such. Re-verify them against the live page and your own run before relying on them, since usage-priced platform tools and community-maintained Actors can change.

Pricing: usage-based, not free

Here's the honest pricing picture, re-checked against the live listing and Apify's own pricing pages on August 27, 2026. This is not a free browser tool, and framing it as one would be misleading. The Actor now bills under Apify's pay-per-event model at $11.00 per 1,000 robots.txt AI-policy rows, or $0.011 per delivered result, a change from the roughly $0.003-per-result-plus-start-fee structure the listing showed when this review first ran in June 2026. Same "from $11.00 / 1,000" headline, different mechanics underneath: you're charged for the rows a run actually delivers.

That per-delivered-row detail has one genuinely elegant consequence for monitoring. The Actor offers two dataset modes (all emits every result, changes_only emits only what shifted since the last run), and the listing states that a changes_only run finding nothing new incurs "zero rows and zero charges." A scheduled watch over a stable set of domains therefore costs money only in the weeks something actually moves, which is exactly the billing shape you want for a tripwire.

The platform side frames the entry cost. Every Apify account, including the no-card free plan, carries a $5 monthly platform credit, which at $0.011 a row covers a few hundred checked domains before you owe anything; paid platform plans start at $29/month (Starter), with Scale at $199 and Business at $999 for heavier automation. So a developer already living on Apify can trial this Actor at no marginal cost, while at real scale the per-row price stays trivial next to building the same monitoring yourself.

One more platform mechanic the browser-based checkers reviewed here never make you think about: your runs persist. The domain lists you feed in and the JSON that comes out live in Apify storage under your account; on the free plan, "Your 10 most recent runs are retained for 4 months," per Apify's storage docs. The Actor also keeps its change-detection baseline in a named key-value store (robotstxt-ai-checker-state), which Apify exempts from automatic deletion. That persistence is the feature: it's what makes week-over-week diffs possible at all. It also means a bot-policy audit here is stored account data rather than an ephemeral page view, worth knowing if the domain list itself is client-confidential.

Checking one site, then, you're paying setup friction and platform overhead for an answer a free browser tool hands you instantly. Checking a thousand on a schedule, the economics invert completely.

Where it stops being the right tool

The wall isn't capability; it's fit. For a non-technical user who wants to know whether their site blocks GPTBot, the Actor model is pure overhead: create an Apify account, find the Actor, configure inputs, run it, read JSON. That's minutes of setup and a small cost for a question an instant browser tool answers in one screen, free, with no signup. There's also the maintenance reality: this is a community-maintained Actor by an individual developer (taroyamada), not a first-party Apify product, so support responsiveness and long-term upkeep rest on one author rather than a company.

So the boundary is clear. If you're a developer who wants to automate, batch, or monitor AI-crawler policies (across many domains, on a schedule, wired into a pipeline), this Actor is a genuinely good fit and does things browser tools can't. If you're checking one site once, you want the opposite kind of tool.

GEO note: necessary, not sufficient

Run this Actor across 500 domains and the output sorts them into who admits the AI crawlers and who turns them away, and neither column predicts who gets cited. That's the caveat to keep bolted to every batch report. A blocking rule guarantees a site can't be cited, so finding one is decisive; an open file merely permits crawling without causing any of what follows: the fetch, the indexing, the answer that names the site. In pipeline terms, this check belongs at the gate of the funnel, and a green gate measures nothing about flow. Wire it into CI to catch regressions; look elsewhere for evidence that the open door ever got used.

Verdict

The Apify Robots.txt AI Checker is a capable, honestly powerful tool aimed at the wrong audience for a casual visit. As a programmable, batch-first, change-aware Actor it does things no instant browser tool can: audit hundreds of domains, run on a schedule, surface policy shifts over time, and drop structured JSON into your automation. Buy into it if you're a developer who wants AI-crawler checks living inside a pipeline or a monitoring job. Skip it if you just want to know whether your one site blocks the AI bots right now. For that, an instant, no-signup browser tool like Is My Brand In AI answers the same question in seconds and at no cost. The decisive variable isn't the feature list; it's whether you're checking one site or a thousand.