AI Crawler Checker: Can AI Bots Read Your Site?

What the AI crawler checker looks at

01
robots.txt, bot by bot

We read your robots.txt the way crawlers do (RFC 9309) and figure out, for each of 14 bots, whether it’s allowed and which rule decides it. A bot with its own group ignores the rules under User-agent: *.
02
Server and firewall blocks

We request the page as a normal browser and as seven AI bots. If the browser gets in and a bot gets a 403, 429, 503 or a challenge page, something on your server or CDN is likely blocking it.
03
noindex and nosnippet

We read the meta robots tag and the X-Robots-Tag header for noindex, nosnippet and max-snippet:0, which keep a page out of answers or limit what can be quoted.
04
Content without JavaScript

We check whether your H1 is in the raw HTML and count the words visible without JavaScript, because that HTML is all most AI crawlers read.
05
Structured data

We list the JSON-LD types on the page, flag any that fail to parse, and check for an Organization or LocalBusiness block that links your profiles with sameAs.
06
llms.txt

Whether the file exists, is plain text rather than an HTML page and opens with a # title, how many links it lists, and whether they work (we test up to five).
07
Sitemap

Whether you declare one in robots.txt or have one at /sitemap.xml, how many URLs it lists, and the date of its latest lastmod.
08
hreflang

Which languages and regions you declare (es-ES, es-MX, es-419 and so on), plus the self-reference and x-default. This matters most for Spanish-language sites that sell in several countries.

AI search bots vs. AI training bots

AI companies run more than one crawler, and each does a different job. Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch pages so they can show and cite them in answers. User bots (ChatGPT-User, Claude-User, Perplexity-User) open a page when someone asks the assistant to. Training bots (GPTBot, ClaudeBot, CCBot and others) collect content to train future models.

Each bot has its own user agent, so you can allow or block them one by one. OpenAI, for example, documents that you can block GPTBot and still allow OAI-SearchBot. Blocking training bots is a legitimate business call. Blocking search and user bots takes you out of those answers.

Google works differently. Googlebot crawls for Search, which includes AI Overviews and AI Mode, while Google-Extended only controls Gemini training and grounding in the Gemini app and Vertex AI. To limit how your pages show up in its AI features, Google points site owners to nosnippet, max-snippet and noindex instead.

The 14 bots we check: who runs each one, what it does, and what happens if you block it.
Bot	Company	What it does
`OAI-SearchBot` AI search	OpenAI	ChatGPT search results Your pages can’t show up as sources in ChatGPT search answers.
`ChatGPT-User` User request	OpenAI	Pages a ChatGPT user asks for ChatGPT can’t open your page when a user asks it to.
`GPTBot` Training	OpenAI	Training OpenAI’s models OpenAI won’t train on your content, but ChatGPT search can still show you if OAI-SearchBot is allowed.
`Claude-SearchBot` AI search	Anthropic	Claude search results Claude’s search can’t use your pages, so they’re less likely to appear in its answers.
`Claude-User` User request	Anthropic	Pages a Claude user asks for Claude can’t fetch your page when a user asks it to.
`ClaudeBot` Training	Anthropic	Training Anthropic’s models Anthropic won’t train on your content, but Claude’s search and user requests run on other bots.
`PerplexityBot` AI search	Perplexity	Perplexity answers and citations Perplexity can’t index your site, so it won’t show you or link to you in its answers.
`Perplexity-User` User request	Perplexity	Pages a Perplexity user asks for Perplexity can’t open your page when a user asks it to.
`Googlebot` AI search	Google	Google Search, AI Overviews, AI Mode You drop out of Google Search, including AI Overviews and AI Mode.
`Google-Extended` Mixed use	Google	Gemini training and grounding Google won’t use your content for Gemini training or grounding, but Google Search and AI Overviews are unaffected.
`Bingbot` AI search	Microsoft	Bing search and Microsoft Copilot You drop out of Bing, which also feeds Microsoft Copilot.
`Applebot-Extended` Training	Apple	Training Apple’s models Apple won’t train its models on your content, but Siri and Spotlight, which use Applebot, are unaffected.
`CCBot` Training	Common Crawl	Common Crawl’s open web archive You stay out of Common Crawl, an archive many AI models are trained on.
`Meta-ExternalAgent` Training	Meta	Meta AI training and products Meta won’t use your content for AI training or in Meta AI products.

How to set up robots.txt for AI bots

Give every bot you care about its own group. Under RFC 9309, a bot that finds a group with its name follows only that group and ignores the rules under User-agent: *. So a blanket Disallow: / aimed at scrapers also blocks OAI-SearchBot, unless you give OAI-SearchBot a group of its own.

The example below lets the search and user bots in and keeps the training bots out. If you’re fine with your content being used for training, delete the second half. Either way, put these groups at the top of the file and run the checker again to confirm.

One more thing to watch: if robots.txt returns a 5xx error or times out, crawlers treat the entire site as off-limits. A 404, on the other hand, means everything is allowed.

# AI search and user bots: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# AI training bots: blocked (optional)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yourbrand.com/sitemap.xml

If your site uses Cloudflare

Since July 1, 2025, Cloudflare has turned on AI crawler blocking by default for new domains. Many site owners don’t know it’s on, so their robots.txt welcomes AI bots while the firewall turns them away.

You manage it in the Cloudflare dashboard (AI Crawl Control and the bot settings), one bot at a time. If our server test shows AI bots getting a 403 or a challenge page while a browser gets through, start there. Because the test runs from cloud servers rather than the bots’ real IPs, confirm what you see in the dashboard before changing anything.

Can AI crawlers see your content without JavaScript?

Research published by Vercel and MERJ in December 2024 found that GPTBot, ClaudeBot and PerplexityBot don’t execute JavaScript. They read the HTML your server sends and stop there. Googlebot does render JavaScript, which is how a site can look fine in Google and nearly blank to AI crawlers.

Single-page apps that build everything in the browser are the typical case. The checker looks for your H1 in the raw HTML and counts the words visible without JavaScript. If that count is close to zero, move your main content to server-side rendering or static HTML.

llms.txt: what it is and why we give it little weight

llms.txt is a Markdown file at the root of your site that lists your key pages for language models. Jeremy Howard of Answer.AI proposed it in September 2024 as a short table of contents for machines.

No major AI search engine has confirmed that it uses llms.txt. It’s cheap and harmless, so the checker looks for it and validates the format, but we give it far less weight than crawler access and server-rendered content. If you publish one, serve it as plain text and open it with a # title.

ChatGPT Ads calculator →

Now find out whether AI names you or a competitor.

See where you show up today in ChatGPT, Claude, Gemini, Perplexity and Google AI Overviews, which competitors they name instead, and 3 to 5 questions you can win first. Free, by email.

Get your free AI visibility diagnosis

FAQ

Questions about AI bots and robots.txt

Can ChatGPT crawl my site?

Only if three things line up: your robots.txt allows OAI-SearchBot and ChatGPT-User, your firewall lets them through, and your content is in the HTML rather than loaded by JavaScript. If any of them fails, ChatGPT can’t use the page. The checker above tests all three.

Should I block GPTBot?

That’s a business decision. GPTBot only collects content to train OpenAI’s models, and OpenAI documents it as separate from OAI-SearchBot, so blocking it doesn’t take you out of ChatGPT search. If you block it, make sure OAI-SearchBot and ChatGPT-User stay allowed.

How do I check if my robots.txt blocks AI bots?

Paste your URL into the AI crawler checker above. It reads your robots.txt the way the bots do (RFC 9309) and shows, for each of 14 bots, whether it’s allowed and which rule applies. It also tests whether your server lets seven of them in.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended only controls Gemini training and grounding in the Gemini app and Vertex AI. AI Overviews and AI Mode run on Googlebot, and Google points to nosnippet, max-snippet and noindex if you want to limit how your pages appear there.

Do I need an llms.txt file?

No major AI search engine has confirmed using it. It’s cheap and harmless, so add one if you like, but fix robots.txt, server blocks and server-side rendering first.

My robots.txt allows AI bots. Why does the checker say they’re blocked?

Because the block is happening at your server or firewall, not in robots.txt. Cloudflare, for example, blocks AI crawlers by default on domains added since July 1, 2025. Our test runs from cloud servers rather than the bots’ real IPs, so treat the result as a strong signal and confirm it in your firewall or CDN settings.