Why AI Crawlers May Not See Your JavaScript Content

Start by separating access from rendering

Before investigating JavaScript, make sure your test received the intended page. A firewall challenge, login screen or redirect is a delivery problem. Server rendering will not fix it.

If you have not checked crawl permissions and HTTP responses, start with our AI bot access guide.

The DOM is the browser’s in-memory representation of the document. A screenshot shows its visual result, not the original response. A local browser test also does not prove identical processing by every AI service.

Why the original HTML can be almost empty

With client-side rendering, the server may return a shell and leave the browser to load data and construct the page.

<main id="product">
  <p>Loading product details...</p>
</main>
<script src="/assets/product-app.js" defer></script>

After the script runs, the customer might see a complete product page. A basic HTML fetch still sees the shell. Compare that with an illustrative server-rendered response:

<main id="product">
  <h1>Oak desk with two drawers</h1>
  <p>A compact oak desk for home offices.</p>
  <dl>
    <dt>Width</dt>
    <dd>120 cm</dd>
    <dt>Delivery</dt>
    <dd>See current delivery options at checkout.</dd>
  </dl>
  <a href="/delivery">Delivery information</a>
</main>
<script src="/assets/product-interactions.js" defer></script>

Here, core facts arrive as HTML while JavaScript can still power galleries, variant selectors and other interactions. These examples are illustrative, not complete production templates.

Server rendering and hydration are different steps

Server-side rendering generates HTML on the server. Static generation creates HTML ahead of the request. Hydration attaches client-side behavior to existing HTML.

React describes hydration as connecting React logic to server-generated content and warns that the initial client output should match the server output. A mismatch is a bug to investigate, not evidence that React prevents discovery. React hydration documentation

A framework name tells you less than the response itself. Inspect the deployed URL rather than guessing from the technology label.

Can AI crawlers execute JavaScript?

There is no single documented answer for every search crawler, user-triggered fetcher and browser agent. Their purposes and implementations differ, and evidence about one does not establish the capability of another.

Google documents a rendering process that executes JavaScript and also notes that not all bots can do so. That makes Google’s renderer useful evidence about Google, not a universal model for AI retrieval. Google’s JavaScript SEO basics

Independent network-traffic research has found major non-Google AI crawlers fetching script files without executing them. Providers do not publish a permanent, complete rendering specification, so treat this as current evidence—not a guarantee—and test the response your site actually delivers. Vercel and MERJ crawler research

“The product description is absent from the original response and appears after a browser executes JavaScript.”

That is a measurable dependency. It is more useful than declaring the whole website invisible to AI. Google provides inspection evidence for its own systems; other major AI providers do not offer an equivalent site-owner rendering test.

Step 1: Choose pages and facts worth testing

Test representative pages. For a shop, include a product, category and delivery-information page. For a service business, include a service page, relevant location page and contact page.

For each URL, write down three to five public facts:

• the product or service name;
• the main description and specifications;
• the public price, if one is published;
• delivery, availability or service-area information;
• a relevant internal link or contact method.

Use one distinctive sentence or exact fact as a search target. A brand name may appear in navigation even when the main content is missing. Open the URL directly in a fresh session as well as through site navigation.

Step 2: Save the original response

Ask your developer to run this command, or use a Bash-compatible terminal. Replace the example with a public URL you own and run it in a new folder because it creates local files.

curl --silent --show-error --location --max-redirs 5 \
  --connect-timeout 10 --max-time 30 --compressed \
  --dump-header response-headers.txt \
  --output response.html \
  --write-out 'Status: %{http_code}\nFinal URL: %{url_effective}\n' \
  'https://www.example.com/products/oak-desk'

The request follows redirects and saves the response without running the page’s JavaScript. Inspect the final URL, headers and body before interpreting missing text.

In Chrome DevTools, select Network, reload, choose the main document request and inspect its Response. The Elements panel shows the DOM, which scripts may already have changed. Chrome Network reference

Look for actual content, not just a matching string

A phrase can be split by tags, encoded as entities or present only inside a JavaScript data object. A description in application-state JSON is not the same as ordinary page content; record it as “present in embedded data” rather than calling it rendered HTML or completely absent.

Step 3: Compare JavaScript on and off

In Chrome DevTools, open the Command Menu, choose Disable JavaScript and reload. Re-enable JavaScript afterwards. Chrome’s instructions

Check whether your selected facts and key links remain readable. Then reload with JavaScript enabled, keeping the URL, language and public user state consistent.

This is a dependency check, not a crawler simulator. It does not reproduce another service’s network, cookies, parsing rules or security treatment.

Check content after interaction separately

• already in the original HTML;
• inserted automatically after scripts run;
• fetched only after a user interaction.

A collapsed section may already contain its full text, while a visually identical section may request text only when clicked. Google warns against relying on clicks or scrolling to load content for crawling and recommends discoverable paginated URLs alongside infinite scrolling. Google lazy-loading guidance

Step 4: Find the dependency that causes the gap

Use the browser’s Network and Console panels to identify what happens between the original response and the complete page.

Do not make a private API public just to satisfy a crawler. Deliver facts already intended to be public through an appropriate public page.

Step 5: Use Google Search Console for Google evidence

For a verified property, use URL Inspection and its live test to examine the tested page and available rendered output. Keep the live test separate from indexed-page information: a successful live fetch is not proof the current version is indexed. Search Console URL Inspection documentation

For AI Overviews and AI Mode, Google says normal Search eligibility applies, including indexing and snippet eligibility. You do not need a special AI rendering format. Google AI features guidance

Do not use a successful Google test to certify what OAI-SearchBot, PerplexityBot or Claude-SearchBot processed. Record Google evidence as Google evidence.

How to fix missing server-side content

Choose the smallest change that makes important public facts reliably available. A gallery can remain interactive while the product name, description and dimensions are already readable.

These are options, not universal rankings. For stock and pricing, define freshness rules so server HTML and post-JavaScript content do not conflict.

Keep internal navigation discoverable

<a href="/services/custom-furniture">Custom furniture services</a>

A button that only invokes a routing function is not equivalent to a normal link for discovery. Google crawlable-link guidance

Treat bot-specific dynamic rendering as a workaround

A special rendered version for selected crawlers adds another delivery path to maintain. Google describes dynamic rendering as a workaround and recommends server-side rendering, static rendering or hydration instead. Google dynamic-rendering guidance

When not to fix or rebuild the page

A large JavaScript bundle is not, by itself, proof that important content is missing. Do not change architecture solely because:

• an audit counted many script tags;
• a detector identified React, Vue or another library;
• an optional widget needs JavaScript;
• a private dashboard is unreadable without login;
• one extractor missed text clearly present in the response.

JavaScript dependence is not the same as a Core Web Vitals failure. If core facts already arrive in HTML and relevant checks pass, investigate the next demonstrated issue instead of pursuing an abstract “AI friendly” label.

A developer handoff that can be verified

Give your developer the URL, observed response, missing facts and conditions needed to reproduce the problem. Define the expected result without prescribing an unnecessary rebuild.

“On the public product URL, the original response contains the product name but not the description or dimensions. Those fields appear after the details API completes. Please make those public fields available as page HTML on direct requests, keep them consistent after hydration, and verify cache freshness after an update.”

• direct access to each tested URL works;
• selected public facts appear in the original response;
• JavaScript does not replace them with conflicting information;
• login-dependent data stays protected;
• important links and metadata remain appropriate;
• cached responses follow the agreed freshness rules.

Save before-and-after evidence. “The site now uses SSR” is an implementation statement; “these missing facts now appear in the response” is a testable outcome.

Frequently asked questions

Technical references

• Google Search Central – JavaScript SEO basics
• Google Search Central – Fix JavaScript search issues
• Google Search Central – Dynamic rendering
• Google Search Central – AI features and your website
• Chrome DevTools – Network reference
• React – hydrateRoot
• Vercel and MERJ – The rise of the AI crawler

Go deeper