How to check if ChatGPT can read your website

Short answer: ChatGPT can read your site only if three things are true at once — your robots.txt allows OpenAI's crawlers, your content exists in the raw HTML before JavaScript runs, and no bot-protection layer is serving the crawler a challenge page instead of your content. You can verify all three by hand in about ten minutes using curl. This guide walks each check in order, because they fail in that order: access first, then rendering, then content quality. Fixing the wrong one wastes a day.

First, a distinction that trips people up: "ChatGPT" is not one crawler. OpenAI runs three, and they're governed independently.

OpenAI's three agents

These are documented in OpenAI's bots overview. The practical upshot: if your goal is to be cited in ChatGPT, the one to watch is OAI-SearchBot. People routinely block all three by accident and then wonder why they're invisible. (For the full cross-engine roster, see the complete AI crawler list.)

Check 1 — Read your live robots.txt

Not the one in your repo. The one your CDN is actually serving:

curl -s https://yoursite.com/robots.txt

Read it as a crawler would. Look for:

• A blanket User-agent: * followed by Disallow: / with no explicit allow for the AI bots. That blocks everything, including OAI-SearchBot.
• Named blocks like User-agent: GPTBot / Disallow: /. Someone may have added these deliberately during an "AI scraping" panic.
• A CDN or WAF bot-protection preset. Many ship with AI user agents denied by default; the rule may not even be in your file — it's injected at the edge.

If you want to be read, the safe explicit form is:

User-agent: GPTBot
Allow: /
 
User-agent: OAI-SearchBot
Allow: /
 
User-agent: ChatGPT-User
Allow: /

Remember that robots.txt matching is by user-agent token and is case-insensitive on the token, but the rule groups are independent. A Disallow: / under * does not apply to a bot that has its own named group — but a bot with no named group falls back to *. When in doubt, name the bots explicitly.

Check 2 — The JavaScript-rendering test

Most AI crawlers fetch your HTML and leave. They do not run JavaScript, and they do not wait for client-side hydration the way a browser does. So the question is: does your content exist in the raw HTML response, or only after a framework boots?

Pick a sentence you care about — a price, a product claim, a key definition — and grep the raw response for it:

curl -s https://yoursite.com/pricing | grep -i "29 per month"

If grep finds it, a crawler can too. If it returns nothing, your content is client-rendered and effectively invisible to bots that don't execute JS. The tell-tale shape of a client-only page is a near-empty body:

curl -s https://yoursite.com/pricing | grep -c "<div id=\"root\">"

A page that reduces to <div id="root"></div> plus a script bundle is a shell. The fix is server-side rendering or static generation for anything you want a model to know. In Next.js terms: keep the content in Server Components or statically generated pages, not in a client component that fetches after mount.

Diagram — what the crawler sees vs. what you see: Two side-by-side browser frames of the same URL. Left ("Your browser"): the fully rendered pricing page, prices and copy visible. Right ("GPTBot / OAI-SearchBot"): the same URL as raw HTML — a header, an empty #root div, and a bundle.js reference, with the price absent. An arrow labeled "no JS execution" connects them. The point: the gap between these two frames is exactly what you're testing for.

Check 3 — The markdown-emptiness test

A page can pass Checks 1 and 2 and still give a model nothing to quote. The test: strip the page down to its text and ask whether anything answer-shaped survives.

A quick approximation — fetch the page and look at the visible text density:

curl -s https://yoursite.com/ \
  | sed -e 's/<[^>]*>//g' \
  | tr -s ' \n' ' \n' \
  | grep -iE 'price|plan|how it works|what is' || echo "no answer-shaped text found"

That's crude, but the principle is what matters: if the readable text on your homepage is three taglines of brand atmosphere and a "Book a demo" button, a model has nothing to lift. The pages that get quoted state the answer plainly in the first hundred words. A cleaner version of this test is to paste the page into any markdown-conversion tool and see what's left — if the markdown is nearly empty, so is the page from a model's perspective.

Two reliable fixes: put the answer in the opening paragraph (let the atmosphere follow), and publish a markdown version of key pages — the convention behind llms.txt and the .md companion files.

Check 4 — Rule out a WAF challenge page

This is the sneakiest failure because it looks fine in your browser. WAFs and bot-managers (Cloudflare, Akamai, DataDome, and others) sometimes serve crawlers a JavaScript challenge, a CAPTCHA interstitial, or an HTTP 403 — while serving humans the real page. A crawler that gets a challenge page indexes the challenge page.

Simulate a crawler by sending its user-agent string:

curl -s -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  -o /dev/null -w "%{http_code}\n" https://yoursite.com/

A 200 with real content is good. A 403, a 503, or a 200 whose body is a "Checking your browser…" interstitial means your protection layer is eating the bot. The right fix is to allow-list the verified crawlers at the edge — and "verified" is the operative word, because user-agent strings are trivially spoofed. Allow-list by the crawler's published IP ranges, not the UA string alone. We cover exactly how to verify each one in the complete AI crawler list.

Putting it together

Run the four checks in order and stop at the first failure — each one gates the next:

1. Access — robots.txt and the edge allow the bot.
2. Rendering — the content is in the raw HTML.
3. Content — there's an answer in the first hundred words.
4. Delivery — no challenge page replaces your content for bots.

Most "ChatGPT can't see us" problems are check 1 or check 2 — plumbing, not strategy. That's the good news: plumbing is fixable in an afternoon. If you'd rather not run these by hand on every page, EchoRanked runs all four as part of its readiness scan and shows you the exact request and response at the point each one fails.

Frequently asked questions

Does ChatGPT execute JavaScript when it reads my site?

No. OpenAI's crawlers (GPTBot, OAI-SearchBot, ChatGPT-User) primarily fetch raw HTML and do not run client-side JavaScript or wait for hydration. Content that only appears after a framework mounts is effectively invisible to them. Server-render or statically generate anything you want a model to read.

Which OpenAI agent should I allow if I only care about being cited in ChatGPT?

OAI-SearchBot — it's the crawler that indexes content for ChatGPT's search citations. GPTBot is for training and ChatGPT-User is for on-demand fetches when a user pastes your link. Blocking OAI-SearchBot is the most common reason a site never shows up as a ChatGPT source.

My site loads fine in my browser — why would a crawler see something different?

Two common reasons. First, your content may be client-rendered, so the browser builds it with JavaScript that the crawler never runs. Second, a WAF or bot-manager may serve crawlers a challenge or 403 while serving humans the real page. Test with curl and a bot user-agent to see the crawler's actual view.

Where to go next

Once you know ChatGPT can reach you, find out who else is crawling and how to verify them in the complete AI crawler list — and curate which pages they should prioritize with a good llms.txt.