GPTBot, ClaudeBot, PerplexityBot: the complete AI crawler list (and how to verify them)

Short answer: The AI crawlers most likely to hit your site in 2026 are OpenAI's GPTBot, OAI-SearchBot, and ChatGPT-User; Anthropic's ClaudeBot, Claude-User, and Claude-SearchBot; Perplexity's PerplexityBot and Perplexity-User; plus Apple, Amazon, Meta, ByteDance, and Common Crawl. Google's AI use is controlled by the Google-Extended robots token, which is not a separate crawler. The table below lists each one with its operator's official IP-range source so you can verify it — because the user-agent string itself is trivially forged and should never be trusted on its own. This page is reviewed monthly; it was last verified 2026-06-11.

The list

A few sample user-agent strings, so you know what to grep your logs for:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot
Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)
Mozilla/5.0 (compatible; ClaudeBot/1.0; [email protected])
Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://docs.perplexity.ai/guides/bots)

(Version numbers drift; match on the bot name token, not the full string.)

The two-token operators: Google and Apple

Google-Extended and Applebot-Extended are the most misunderstood entries, so they get their own note. Neither is a crawler with its own request user-agent. They are robots.txt control tokens. Google reuses Googlebot to fetch pages, and the Google-Extended token only decides whether that already-crawled content may be used to train Gemini and ground Google's AI products. Per Google's own documentation, blocking Google-Extended does not remove you from Google Search and is not a ranking signal. The catch: there is no clean way to appear in Google Search while opting out of AI Overviews grounding, because the same Googlebot fetch feeds both. Applebot-Extended works the same way relative to Applebot and Apple Intelligence.

Why you can't trust the user-agent string

A user-agent header is a plain string the client chooses. Anyone — a scraper, a competitor, a botnet — can set User-Agent: GPTBot and walk straight through a UA-based allow-list. The inverse also happens: real crawlers occasionally show up with unexpected strings. So a UA match is a hint, never proof.

There are two reliable ways to verify a crawler is who it claims to be:

1. Match against the operator's published IP ranges. OpenAI and Perplexity publish JSON files of their crawler CIDR blocks (linked in the table). Fetch the current list, and confirm the request's source IP falls inside it. Cache the list and refresh it on a schedule — the ranges change.

# Is a request IP inside OpenAI's published SearchBot ranges?
curl -s https://openai.com/searchbot.json | grep -o '"[0-9.]*/[0-9]*"'
# → compare the request's source IP against these CIDRs in your code

2. Reverse-then-forward DNS, where the operator supports it. For crawlers that don't publish IP lists (notably Anthropic, which explicitly does not publish ranges), do a reverse DNS lookup on the source IP and confirm it resolves to the operator's domain, then forward-resolve that hostname back to the same IP. A spoofer controls the UA string but not the operator's DNS.

Diagram — the verification gate: A flowchart. An inbound request arrives claiming User-Agent: GPTBot. Branch 1: does the source IP fall inside OpenAI's published CIDR ranges? If yes → verified, allow. If no → Branch 2: does reverse-then-forward DNS resolve to the operator's domain? If yes → verified. If no → it's a spoofer wearing the bot's name → rate-limit or block. The visual point: the UA string is the claim; the IP/DNS check is the proof.

A blunt warning to go with the gate: do not allow-list a bot purely by its IP ranges and then skip the UA check, and do not block AI bots by blanket-banning cloud IP ranges. Anthropic notes that banning broad cloud ranges can stop its bots from even reading your robots.txt, which defeats a clean opt-out. Combine UA + verified IP/DNS, and express your actual intent in robots.txt.

A note on robots.txt vs. these tokens

Listing a token in robots.txt only works if the operator honors it — and the reputable ones (OpenAI, Anthropic, Perplexity, Google, Apple) state that they do. Less reputable scrapers ignore robots.txt entirely, which is exactly why edge-level verification matters for anything you truly need to gate. robots.txt is a published request; IP/DNS verification is enforcement.

Why we keep this dated

AI crawler names, versions, and IP ranges change often — new bots launch, operators split one crawler into three (as Anthropic did), and CIDR blocks rotate. A reference like this is only useful if it carries a verification date and gets re-checked. We review this table monthly and update the updated date at the top when anything changes. If you spot a stale entry, that's a bug — treat the "last verified" date as the trust signal.

Knowing who is crawling is step one. Whether those crawlers — and the agents built on them — can actually complete a task on your site is a separate question, and a more interesting one: see can an AI agent actually buy from your website. And if you only want to confirm the big one, here's how to check if ChatGPT can read your website. Keeping this current by hand is tedious; EchoRanked verifies live crawler access against the published ranges as part of every scan.

Frequently asked questions

Is Google-Extended a crawler I can see in my logs?

No. Google-Extended is a robots.txt control token, not a separate crawler — it has no distinct request user-agent. Google fetches pages with Googlebot; the token only governs whether that content may be used to train Gemini and ground Google's AI features. Blocking it doesn't affect Google Search.

How do I verify a request really came from GPTBot and not a spoofer?

Match the request's source IP against OpenAI's published ranges at openai.com/gptbot.json (and the equivalent files for SearchBot and ChatGPT-User). The user-agent string alone is forgeable. For operators that don't publish IP lists, such as Anthropic, use reverse-then-forward DNS to confirm the IP resolves to the operator's domain.

Does Anthropic publish IP ranges for ClaudeBot?

No. Anthropic does not publish IP ranges and recommends controlling its crawlers via robots.txt user-agent rules (ClaudeBot, Claude-User, Claude-SearchBot) rather than IP blocking — partly because banning broad cloud ranges can prevent its bots from reading your robots.txt at all. Verify Anthropic traffic with reverse DNS instead.

Sources

• OpenAI — Overview of OpenAI crawlers
• Anthropic — Does Anthropic crawl the web, and how to block it
• Perplexity — Perplexity crawlers
• Google — Google crawlers overview
• Common Crawl — CCBot