FLOXOLAB / AI CRAWLER ACCESS CHECKER
AI crawler access checker:
can AI crawlers read this page?
Enter a URL. The checker reads its robots.txt, requests the page with the user agent of 14 AI crawlers, and compares each answer with a normal browser request. You get a verdict per crawler and the line to change.
robots.txt says what a crawler may fetch.
Your CDN decides what it actually gets.
Check a URL
REPORT
Crawler access for
Site checks
Per crawler
| Crawler | Purpose | robots.txt | HTTP | Verdict |
|---|
01 / METHOD
What the check does
- Browser baseline. Requests the page with an ordinary desktop browser user agent. If this request is refused too, the site blocks automated requests in general, and crawler-specific results are marked unknown.
- robots.txt, per crawler and per URL. Reads
/robots.txtfrom the host that serves the page and applies RFC 9309: the group naming the crawler wins overUser-agent: *, the longest matching rule wins, andAllowwins a tie. A robots.txt that answers 5xx or 429 counts as “disallow everything”; a 404 counts as “no rules”. CloudflareContent-Signallines are reported. - One request per crawler user agent. Requests the page again with each crawler's user agent, reads only the status code and headers, and compares them with the browser answer. Google-Extended and Applebot-Extended are robots.txt tokens only, so the request uses Googlebot and Applebot, which fetch for them.
- Page directives. Reads
meta name="robots", crawler-named meta tags andX-Robots-Tagheaders, including ones scoped to one crawler, such asGPTBot: noindex. - Text without JavaScript. Counts words in the raw HTML after removing scripts, styles and templates. Many AI crawlers fetch HTML without running JavaScript, so a client-rendered page can look empty to them.
- Discovery files. Checks
/llms.txt, the sitemap declared in robots.txt (or/sitemap.xml) and the canonical URL.
Allowed robots.txt permits the URL and the crawler's user agent gets the page.
Partial it can fetch the page, but something limits use: noindex, nosnippet, rate limiting, almost no text without JavaScript, or a robots.txt rule a user-initiated agent may not follow.
Blocked robots.txt disallows the URL, robots.txt is unreachable, or the server refuses that user agent while serving a browser.
Unknown our request can't settle it, for example when the site refuses every request from our server or verifies crawler IP addresses.
02 / LIMITS
What this checker can't tell you
- Requests come from FloxoLab's server, not from OpenAI, Anthropic or Google. Some CDNs check whether a request claiming to be a crawler comes from that vendor's published IP ranges. Such a CDN may refuse our request and still admit the real crawler, or the opposite. When a site even refuses our Googlebot-UA request, the report flags this and marks user-agent blocks as unknown.
- Crawler requests read headers only. A bot challenge that answers a crawler user agent with HTTP 200 and a challenge page is not detected; one that answers 403, 429 or 503 is.
- One request, one moment. Rate limits, geo rules and A/B bot rules can change between requests. Server logs remain the proof of what a real crawler received.
- Access is not inclusion. An allowed crawler may still never visit, index, cite or train on the page.
- Some user agents are not published. Anthropic documents its robots.txt tokens but not full user-agent strings, and ByteDance has no official Bytespider page we could verify. For those, the request uses the format seen in server logs; the report says so.
- User-initiated agents. OpenAI and Perplexity say ChatGPT-User and Perplexity-User may not follow robots.txt, because a person asked for the page. A robots.txt rule therefore only makes them “partial”.
- noai / noimageai. These meta values are a community convention. None of the vendors below documents them as a control, so the report lists them without counting them as a block.
- No JavaScript rendering, no login. The checker reads raw HTML up to 3 MB and does not follow pages behind authentication. Only ports 80 and 443 are checked; private, local and internal network addresses are refused.
03 / CRAWLERS
The 14 crawlers tested
Tokens, purposes and user agents checked against each vendor's documentation on .
| robots.txt token | Vendor | Purpose | Follows robots.txt | Source |
|---|---|---|---|---|
GPTBot | OpenAI | Model training | Yes | OpenAI bots |
OAI-SearchBot | OpenAI | ChatGPT search | Yes | OpenAI bots |
ChatGPT-User | OpenAI | User-initiated fetch | May not apply | OpenAI bots |
ClaudeBot | Anthropic | Model training | Yes | Anthropic help |
Claude-SearchBot | Anthropic | Search quality | Yes | Anthropic help |
Claude-User | Anthropic | User-initiated fetch | Yes | Anthropic help |
PerplexityBot | Perplexity | Search index | Yes | Perplexity bots |
Perplexity-User | Perplexity | User-initiated fetch | Generally ignores | Perplexity bots |
Google-Extended | Gemini training and grounding; token only, fetched by Googlebot | Yes | Google crawlers | |
Applebot-Extended | Apple | Generative AI training; token only, fetched by Applebot | Yes | About Applebot |
CCBot | Common Crawl | Open web archive used for training | Yes | CCBot |
Bytespider | ByteDance | Model training | No official statement found | No official page verified |
Meta-ExternalAgent | Meta | Training and indexing | Yes | Meta crawlers |
Amazonbot | Amazon | Products and AI training | Yes | Amazonbot |
04 / QUESTIONS
Questions
Robots.txt allows GPTBot, but the report says blocked. Why?
robots.txt is a request to the crawler; your CDN, firewall or hosting can still refuse the request. Cloudflare's AI crawler controls, managed bot rules and some hosting “bad bot” filters block by user agent. The report shows the status code each user agent received next to the browser's.
Should I block training crawlers but allow search crawlers?
That is a business choice. Vendors now split the jobs: GPTBot versus OAI-SearchBot, ClaudeBot versus Claude-SearchBot, Google-Extended versus Googlebot. Google states that Google-Extended does not affect inclusion in Google Search, and OpenAI lists GPTBot and OAI-SearchBot as separate settings. Check each vendor's page linked above before you split rules.
Does llms.txt make AI crawlers read my site?
No major vendor in this list documents llms.txt as a crawling or ranking input. The checker reports it because some tools and agents read it; see do you need llms.txt.
Is my URL or report stored?
No. The check runs on request and the report goes back to your browser only; the checker keeps no database of URLs or results. Hosting access logs may record the request like any page view.