How to Check Which AI Bots Can Access Your Website

Choose the bots and pages to test

“AI bots” describes several different jobs. Start with the purpose you want to allow.

These roles come from the providers’ documentation: OpenAI crawlers, Perplexity crawlers, and Anthropic crawlers.

For Google AI Overviews and AI Mode, check Googlebot access and normal Search eligibility. Google-Extended is a separate control; it is not the crawler switch for those Search features. Google’s AI features guidance

If you need the background first, read OAI-SearchBot vs GPTBot.

Choose a small, useful sample: your homepage, one service or category page, one product page if applicable, one article, and a contact or delivery-information page. Include the exact hostname you use publicly. A successful homepage check does not establish access to every other URL.

Step 1: Check robots.txt

Open the file at the root of the relevant site:

https://www.example.com/robots.txt

Inspect the returned text. Make sure it is actually a robots.txt file rather than a login screen, error document or security challenge.

Look for the bot-specific group and the path you want it to crawl. For example:

User-agent: OAI-SearchBot
Disallow: /products/

This discourages that crawler from fetching paths under /products/. It does not mean the whole site is blocked.

A group using User-agent: * supplies fallback rules when there is no matching bot-specific group. Do not assume those fallback restrictions are automatically inherited by a separately named group. Evaluate the applicable group and the most specific matching path rule, rather than searching the file for the word “Disallow.” Robots Exclusion Protocol

How to fix an unintended robots.txt restriction

First decide which public sections should be available. Then modify the relevant rules while preserving exclusions you actually want.

For a simple site, this illustrative configuration allows public search crawling while retaining two example exclusions and disallowing GPTBot:

User-agent: *
Disallow: /account/
Disallow: /checkout/

User-agent: OAI-SearchBot
Allow: /
Disallow: /account/
Disallow: /checkout/

User-agent: GPTBot
Disallow: /

The paths are examples, not a replacement for your existing file. Named groups need their own applicable restrictions. OpenAI documents the search and training controls as independent. OpenAI crawler controls

Keep these distinctions clear:

• robots.txt communicates crawl preferences; it does not authenticate or physically stop requests.
• A missing file is not automatically a sitewide block. Failed requests, server errors and redirects require separate interpretation. Google’s robots.txt handling
• Sensitive pages need actual access controls, not just a robots.txt exclusion.

After a change, fetch the public file again. Verify the deployed result instead of relying on what the CMS editor displays.

Step 2: Test the server response with a bot User-Agent

The User-Agent is a request header: a label sent by the client. Changing it lets you test how your site responds to that label. It does not turn your computer into the provider’s crawler.

The following examples use curl in a Bash-compatible terminal. On Windows, use WSL or ask your developer to run them. Replace the URL with a public page you own, and run tests from a new working folder because the commands write local files.

Save a baseline response

curl --silent --show-error --location --max-redirs 5 \
  --connect-timeout 10 --max-time 30 --compressed \
  --dump-header baseline-headers.txt \
  --output baseline.html \
  --write-out 'Final status: %{http_code}\nFinal URL: %{url_effective}\n' \
  'https://www.example.com/product/example-product'

This is a normal curl request, not a browser test. It gives you a baseline to compare with the same request using a bot-style header.

Repeat with an OAI-SearchBot User-Agent

curl --silent --show-error --location --max-redirs 5 \
  --connect-timeout 10 --max-time 30 --compressed \
  --user-agent 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot' \
  --dump-header oai-headers.txt \
  --output oai-page.html \
  --write-out 'Final status: %{http_code}\nFinal URL: %{url_effective}\n' \
  'https://www.example.com/product/example-product'

The User-Agent above follows OpenAI’s documented example at the time of writing; versions can change. Use each provider’s current documented string when repeating the test for another bot. Use different output filenames to preserve your evidence.

These commands make GET requests and save both the response headers and body. A headers-only HEAD request cannot tell you whether the returned document contains your product information or just a challenge screen. See the curl manual for the options used here.

Read the response, not just the status

An HTTP result belongs to that request, URL and time. For example, record:

“The request using the OAI-SearchBot User-Agent returned HTTP 200 and contained the expected product description from our test location.”

Do not shorten that to “OpenAI has full access.” The real crawler has a different source IP and may have a different network fingerprint or request history.

Also, curl does not apply robots.txt rules in these examples. A 200 response cannot override a robots.txt disallow in your final assessment.

Step 3: Review CDN and WAF rules

A CDN sits between visitors and your server. A WAF—web application firewall—can inspect and block requests before your application sees them.

This is why an allowed robots.txt rule can coexist with a blocked request.

How to check

Open your provider’s security events or request logs. Match the test using its timestamp, URL and source IP, or a request identifier when available. Check:

• AI crawler policies and bot-management decisions;
• managed challenges and custom firewall rules;
• rate limits and country or network restrictions;
• the action taken and the rule responsible.

If you use Cloudflare, inspect the actual AI bot policy applied to your zone. Its documentation distinguishes Search, Agent and Training behavior; an old instruction to toggle one generic “AI bots” setting may not match the current configuration. Cloudflare AI bot policies

How to fix a confirmed false positive

Identify the rule causing the block before changing anything. Then create the narrowest exception your provider supports for the verified crawler and intended public content. Keep authentication and unrelated security protections in place.

Do not bypass the firewall merely because a request contains OAI-SearchBot or GPTBot. A client can send that text itself—as your test just did.

Use the provider’s supported verification method. Where official crawler IP lists are published, validate the source against the current list and keep that validation updated. Perplexity’s documentation, for example, describes combining source-IP and User-Agent conditions. Perplexity WAF guidance

On a managed ecommerce platform, send support the exact URL, timestamp, response and request identifier. Ask which layer rejected the request and whether a scoped exception is available. You may not control that layer from the store admin.

Step 4: Inspect what your website looks like to an AI crawler

Open the saved oai-page.html as text. Search for a distinctive sentence from the page—not just the brand name in the navigation.

For a product page, check the product name, description, key specifications, visible price where applicable, and availability. For a service page, check what you offer, where you operate and how a customer can enquire.

A quick text search can help:

grep -nFi 'a distinctive sentence from your page' oai-page.html

A missing match is a prompt to inspect further, not proof that the sentence is absent. HTML tags, entities and whitespace can split text. Use an HTML parser when the source is difficult to read.

Compare the response with the rendered page

Your browser can execute JavaScript after receiving the document. The final page may therefore contain information absent from the initial HTML.

Compare the saved response with the browser’s rendered page, including product tabs and parameter tables. Check whether the important information arrives only after a click, API request or consent action.

Google documents a JavaScript rendering process. That does not establish what another provider’s crawler can execute, or when it will do so. Google’s JavaScript crawling and rendering documentation

For AI readable websites, a useful practical goal is to make essential public facts available in the initial HTML wherever feasible. Server-side rendering or static generation may help when those facts currently appear only after client-side execution.

Keep delivery separate from indexability

Also inspect the response headers and HTML for indexing directives, and check the canonical URL. A page can be retrievable while carrying noindex or pointing search systems toward another canonical page.

Step 5: Verify real crawler activity in your logs

Your test answers, “What happened when I sent this request?” Logs can provide stronger evidence of an actual provider visit.

For GPTBot detection—or detection of another named crawler—start by filtering CDN or server logs for its User-Agent. Treat those matches as candidates, then verify their identity with the provider’s documented method. OpenAI publishes separate IP lists for its crawlers. OpenAI crawler verification information

Record the requested URL, time, verified source, status code and security action. If possible, connect the request to the served representation or cached response. A logged 200 alone still does not show the page body was useful.

If the CDN served the page from cache, the request may not appear in origin-server logs. Conversely, no matching request during your observation period does not prove a block: the crawler may simply not have visited.

Successful verified retrieval supports a narrow conclusion: that crawler reached that URL at that time. It does not prove that the provider indexed the page, retained it or will cite it.

Record the result without overstating it

Keep separate fields instead of one “AI accessible” tick:

The example above is illustrative. It shows why “not yet verified” is a valid result, not an error to hide.

Save the original evidence before changing rules. Repeat the same checks afterwards so your developer can show what improved.

When not to change your configuration

Not every restriction is a defect.

• Keep private account areas and non-public information protected.
• Keep intentional training restrictions when they reflect your content policy.
• Keep restrictions on staging environments and other pages you do not want discovered.
• Investigate harmful or excessive traffic before loosening rate limits.
• Do not change robots.txt to fix a firewall block, or rewrite content to fix a delivery failure.

User-triggered fetches also need separate treatment. OpenAI says robots.txt may not apply to ChatGPT-User actions. Perplexity says its user-triggered fetcher generally ignores robots.txt. Anthropic documents robots.txt compliance for its bots, including its user-directed agent. Avoid one universal rule for all three providers.

Your AI search visibility checklist

Before calling the access check complete, confirm that you have:

• selected the relevant search, training or user-directed bot;
• tested representative URLs on the correct hostname;
• evaluated the applicable robots.txt group and path;
• saved the final HTTP response and inspected its body;
• reviewed security decisions where logs are available;
• checked whether essential content is present without client-side execution;
• separated claimed bot identity from verified provider traffic;
• recorded what remains unverified.

For the wider technical and content picture, continue with our AI Visibility Checklist: 12 Things to Check on Your Website.

Frequently asked questions

Technical references

• OpenAI – Overview of OpenAI Crawlers
• Anthropic – Web crawler documentation
• Perplexity – Perplexity Crawlers
• Cloudflare – Block AI bots
• Google Search Central – AI features and your website
• RFC 9309 – Robots Exclusion Protocol

Go deeper