Apify Robots.txt AI Checker Review (2026): Programmable Bot Audits, Pricing & the One-Off Trap

What the Apify Robots.txt AI Checker is

The Apify Robots.txt AI Checker is not a website you visit and a box you paste a URL into. It is an Apify Actor — a packaged program that runs on Apify's automation-and-scraping platform. You give it a list of domains, it fetches each site's robots.txt, parses the rules, and reports whether the major AI crawlers are allowed or blocked, returning the result as structured JSON.

That framing is the single most important thing to understand before judging it. Most tools in the "AI bot checker" cluster are instant, browser-based, no-signup pages built for a human checking one site. This one is built for a developer checking many sites, on a schedule, inside a pipeline. It's the same underlying question — is this AI crawler allowed in robots.txt? — answered with very different machinery.

The trade-off is real and goes both ways. As an Actor it gains things a browser tool structurally cannot offer: batch runs across hundreds of domains, scheduled monitoring, API access from Python/Node.js/cURL, and stateful change detection that flags when a site's policy shifts between runs. It also loses the thing a browser tool does best: an instant, free, zero-friction answer for one URL.

Which AI crawlers it covers

This is where the Actor is strong. It checks the AI user-agents that matter for generative-engine visibility, including:

• OpenAI — GPTBot, ChatGPT-User, OAI-SearchBot
• Anthropic — ClaudeBot, Claude-Web, anthropic-ai
• Google — Google-Extended (the AI-training opt-out token, separate from Googlebot)
• Common Crawl — CCBot (the corpus many models train on)
• ByteDance — Bytespider
• Perplexity — PerplexityBot

One honesty note worth keeping: the Actor's own primary specification names GPTBot, ClaudeBot, anthropic-ai and PerplexityBot explicitly, and describes an "all-AI-bot policy snapshot" capability for the rest. The broader list above reflects the listing's documented coverage rather than something we confirmed agent-by-agent on a live run — so if a specific bot is decision-critical for you, verify it against your actual run output. Either way, the coverage spans both the crawling agents (GPTBot, ClaudeBot, CCBot) and the training-permission tokens (Google-Extended, anthropic-ai), which is the distinction that actually governs whether your content can feed and surface in AI answers.

How it actually runs

You run the Actor one of two ways: from the Apify Console (a web UI where you fill in inputs and click Start) or via the Apify API (the developer path — call it from a script, a cron job, or a CI step). Inputs include the domain list (the listing references up to several hundred domains per run), concurrency settings, a delivery method (Apify dataset or a webhook), and a demo/dry-run toggle for testing.

Output is per-domain JSON: each entry carries the domain, a status, a summary block (counts of blocked / allowed / partially blocked bots), an aiPolicies array with each bot's disposition, any detected changes since the last run, and a timestamp. That structure is the point — it's designed to be parsed and acted on by other software, not just read by a person.

The change-detection feature deserves a callout because nothing in the browser-tool side of this cluster has it. Because an Actor run can compare against a stored baseline, it can tell you a site that used to allow GPTBot now blocks it — a genuinely useful signal if you're monitoring publishers, competitors, or your own properties over time, and the kind of thing you'd otherwise have to build yourself.

Disclosure

AI Tools Police earns affiliate commissions when readers sign up for some tools we cover. That never changes a score, a capability claim, or whether we surface a limitation. We sell no bot checker and no AI-visibility product of our own, which is exactly why an independent reference like this can call a capable developer tool "the wrong fit for a non-technical one-off" without flinching.

How we reviewed this

This review is built on the Actor's public Apify listing, its documented inputs, outputs and pricing, and corroborating search results for the same Actor. We did not run a hands-on test, fabricate a metric, or produce a screenshot, and we did not execute a paid run to verify per-bot behavior. Coverage and pricing figures are presented as documented on the listing and labeled as such — re-verify them against the live page and your own run before relying on them, since usage-priced platform tools and community-maintained Actors can change.

Pricing: usage-based, not free

Here's the honest pricing picture. This is not a free browser tool, and framing it as one would be misleading. It runs on Apify's platform under usage-based pricing: roughly a small per-run start fee (about $0.01) plus a per-result charge (about $0.003 per domain checked), which the listing summarizes as "from ~$11.00 / 1,000 results." Apify's general free monthly platform credit can absorb small or trial runs, so a developer already on Apify may check a handful of domains at effectively no marginal cost — but you still need an Apify account, and at any real scale you are paying per result.

That math reframes the value proposition cleanly. Checking one site, you're paying setup friction and platform overhead for an answer a free browser tool gives you instantly. Checking a thousand sites on a schedule, the per-result cost is trivial relative to doing it any other way. The price isn't high — it's just attached to a model that only makes sense at batch or monitoring scale.

Where it stops being the right tool

The wall isn't capability — it's fit. For a non-technical user who wants to know whether their site blocks GPTBot, the Actor model is pure overhead: create an Apify account, find the Actor, configure inputs, run it, read JSON. That's minutes of setup and a small cost for a question an instant browser tool answers in one screen, free, with no signup. There's also the maintenance reality: this is a community-maintained Actor by an individual developer (taroyamada), not a first-party Apify product, so support responsiveness and long-term upkeep rest on one author rather than a company.

So the boundary is clear. If you're a developer who wants to automate, batch, or monitor AI-crawler policies — across many domains, on a schedule, wired into a pipeline — this Actor is a genuinely good fit and does things browser tools can't. If you're checking one site once, you want the opposite kind of tool.

GEO note: necessary, not sufficient

Knowing whether GPTBot, ClaudeBot, CCBot and the rest are allowed in your robots.txt is a real generative-engine-optimization input — if you're blocking the crawlers, you can't be cited in AI answers, full stop. But it's a necessary, not sufficient condition. An open robots.txt doesn't guarantee crawling, indexing, or citation; it only removes one obvious blocker. Treat this Actor (or any bot checker) as a gate-check, not a visibility strategy — it confirms the door is unlocked, not that anyone walked through it.

Verdict

The Apify Robots.txt AI Checker is a capable, honestly powerful tool aimed at the wrong audience for a casual visit. As a programmable, batch-first, change-aware Actor it does things no instant browser tool can — audit hundreds of domains, run on a schedule, surface policy shifts over time, and drop structured JSON into your automation. Buy into it if you're a developer who wants AI-crawler checks living inside a pipeline or a monitoring job. Skip it if you just want to know whether your one site blocks the AI bots right now — for that, an instant, no-signup browser tool like Is My Brand In AI answers the same question in seconds and at no cost. The decisive variable isn't the feature list; it's whether you're checking one site or a thousand.