OAI-SearchBot vs GPTBot: Which One Controls Your ChatGPT Visibility?

What is OAI-SearchBot?

OAI-SearchBot is the crawler OpenAI documents for search. Its job is discovery: fetching publicly available pages so they can be found and surfaced when ChatGPT answers a question with links to sources.

This is the crawler that matters when someone asks ChatGPT for a recommendation and you want your page to be a candidate answer. If OAI-SearchBot cannot fetch your pages, they are simply not in the pool of pages that could be surfaced.

So what is GPTBot?

GPTBot is a different user agent with a different purpose. OpenAI documents it as the crawler used to collect publicly available web content for model development.

That is a legitimate thing to have an opinion about. Some publishers want their content used, some do not. The important part for this article is narrower: that decision is not the same decision as whether your pages can be discovered for ChatGPT Search.

Blocking GPTBot is a content-usage choice. Blocking OAI-SearchBot is a visibility choice. They are frequently confused, and generic "block all AI bots" snippets confuse them by design.

OAI-SearchBot and GPTBot are not interchangeable

Two separate user agents, two separate robots.txt rules, two separate outcomes. Here is the practical difference:

Can you allow ChatGPT Search but block GPTBot?

Yes. Because they are distinct user agents, robots.txt can address them individually. A site that wants to be findable in ChatGPT Search but does not want its content crawled for model development can say exactly that:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

robots.txt is only the first layer

robots.txt states an intention. It does not describe what actually happens when a crawler sends a request to your server. In practice we regularly see sites whose robots.txt allows a crawler while the CDN, WAF or bot-protection layer answers the very same request with 403, a challenge page or a login screen.

From the crawler's point of view the result is the same as a block: no usable content. And because robots.txt looks fine, nobody notices.

This is what you should really check

1. 1

   robots.txt permission

   Check whether OAI-SearchBot is explicitly or indirectly blocked.

2. 2

   Actual HTTP response

   The crawler should receive the real page, not a 403, login screen or challenge page.

3. 3

   CDN and WAF rules

   Bot protection can override what robots.txt says.

4. 4

   Server-rendered content

   Important content should not disappear when JavaScript is unavailable.

5. 5

   Machine-readable meaning

   Make company, product and service information explicit and understandable.

6. 6

   Actual information value

   Technical accessibility does not make a weak page worth citing.

What happens if you block OAI-SearchBot?

You remove yourself from the discovery step. Your pages are not fetched for search purposes, so they cannot be selected as a source when ChatGPT answers with links. Nothing dramatic happens on your site; you simply stop being a candidate.

This is sometimes intentional — a private tool, a staging environment, a members-only area. It is rarely intentional on a marketing site or a webshop, and it is a common accident when someone copy-pastes an aggressive robots.txt from a forum thread.

What happens if you block GPTBot?

You state that your public content should not be crawled for model-development purposes. Per OpenAI's own documentation this is a separate control from search, so it does not remove you from ChatGPT Search discovery as long as OAI-SearchBot is still allowed.

Whether to do it is a business decision, not a technical error. Publishers, marketplaces and agencies land on different answers for good reasons. What matters is that you make the choice deliberately, and that you do not lose search visibility as a side effect of it.

Don't confuse crawlability with AI optimization

Access and value are two different problems. Access is binary and testable. Value is editorial: does the page actually answer the question a person asked?

Fixing access is usually a one-hour infrastructure task. Becoming the page worth citing is the longer job — clear answers, specific facts, prices, delivery terms, who you are.

What about structured data?

JSON-LD does not make a machine like your page. It removes ambiguity. It states plainly that this is an organisation, this is a product, this is the price, this is the availability, this is who is behind the site — instead of leaving it to be inferred from layout.

Think of structured data as clarification, not a ranking hack.

That is why we report structured data as a measurable signal rather than a score booster: it is either present and valid, or it is not.

Your website can look fine while being difficult for crawlers

A modern front end can render a rich, complete page in the browser while the server's initial HTML contains little more than an empty container and a script tag. Visitors never notice. A crawler that reads the raw response may see almost nothing.

The fix is not "stop using JavaScript". It is making sure the content that carries meaning — the headline, the description, the prices, the key facts — exists in the server response.

A practical robots.txt starting point

If you want search discovery but not model-development crawling, this is a reasonable baseline. Adapt it to your own policy:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Do not blindly overwrite an existing robots.txt file. Existing crawler and private-path rules may still be required.

How do you know whether your site has a problem?

You can do a first pass by hand in two minutes:

The last line is the important one. robots.txt tells you the stated rule; only an actual request sent with the crawler's user agent tells you what the server really answers.

Frequently asked questions

Continue with AI visibility