What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Most AI crawlers that Vercel and MERJ measured in December 2024 did not execute JavaScript. If your article text, product details, or key links appear only after client-side scripts run, those crawlers may never see them. Google Search is different: it documents a rendering stage that does execute JavaScript for eligible pages. The gap between the two is the reason raw HTML and rendered HTML need to be checked separately.
Two versions of the same page
Every URL can produce two different documents, and the difference matters for anyone trying to predict what a crawler reads.
As an Amazon Associate I earn from qualifying purchases.
- Raw HTML (the initial response): the HTML the server sends back to the crawler’s request, before any client-side JavaScript has changed the document.
- Rendered HTML: the document state after a rendering environment loads page resources and runs the JavaScript. It can contain text, links, and metadata that were not in the initial response.
Google’s JavaScript SEO documentation describes the two common architectures that produce this gap. In client-side rendering, the browser receives a relatively sparse app shell, and JavaScript builds or fetches the main content after load. In server-side rendering or prerendering, the server or build process places meaningful content into the initial HTML, so it is available even to clients that never run scripts. Google recommends server-side rendering or prerendering because not all bots can run JavaScript. (Google JavaScript SEO documentation)
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minute| Question | Raw HTML | Rendered HTML |
|---|---|---|
| Where it comes from | Server response to the first request | Output after scripts run in a rendering environment |
| Typical content in a client-rendered site | An app shell, navigation stubs, and script tags | Article body, product data, internal links built by JavaScript |
| Visible in “View Source” | Yes | No, unless the page is also server-rendered |
| Visible in browser DevTools Elements panel | Not directly | Yes, because the Elements panel shows the live DOM after scripts run |
| What a non-rendering crawler can use | Everything in this response | Nothing that depends on script execution |
A browser inspector therefore tells you what a user’s browser built, not what a crawler that skipped rendering received.
#1 Best Overall
What Google documents about rendering
Google describes JavaScript processing in three stages: crawling, rendering, and indexing. After the initial fetch, eligible pages enter a rendering queue. A headless Chromium renderer executes the JavaScript, and Google indexes the rendered HTML. Pages can wait in that queue, so JavaScript-generated content may be indexed later than server-delivered content. Google also says rendering may be skipped in some cases, such as when the server returns a non-200 response. (Google JavaScript SEO documentation)
Rendering depends on access. Google warns that blocking the page or its JavaScript resources, for example in robots.txt, can prevent the renderer from producing the final content.
Google’s Googlebot reference adds two constraints. Search primarily indexes the mobile version of most sites. Googlebot also fetches the first 2 MB of a supported file type, and referenced resources such as CSS and JavaScript are fetched separately under their own size limits. These are Google’s documented limits for Googlebot, not rules for every AI crawler. (What Is Googlebot)
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →What Vercel and MERJ measured in December 2024
The most direct public measurement of AI crawler rendering that this article relies on is Vercel’s report published December 17, 2024, written with MERJ. The team monitored traffic on nextjs.org and Vercel’s network, and checked its findings against a Next.js job board and a site built on a custom monolithic framework.
In that sample, the major AI crawlers measured did not render JavaScript. The crawlers named were OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, Meta-ExternalAgent, Bytespider, and PerplexityBot. The authors reported that ChatGPT and Claude fetched JavaScript files but did not execute them. They also noted that content present in the initial response, such as JSON or delayed React Server Components, could still be interpreted. (Vercel/MERJ study)
The traffic figures in that report
The report also published request counts and content-type shares. Read them as descriptions of that sample, not as industry totals:
- Over the study’s “past month,” Vercel’s network recorded 569 million GPTBot requests, 370 million Claude requests, and 4.5 billion Googlebot requests. These are counts across Vercel’s network as of the December 2024 post, not global crawler volumes.
- JavaScript files made up 11.50% of ChatGPT fetches and 23.84% of Claude fetches in the measured data. Fetching these files did not mean the crawlers ran them.
- HTML accounted for 57.70% of ChatGPT’s fetches in that data, and Claude fetched images in 35.17% of its fetches.
The shares show what each crawler requested. They do not show whether a given page was indexed, cited, or used.
How current is this evidence?
The Vercel/MERJ findings are almost two years old as of October 2026. Crawlers change, and no equivalent public measurement covering the same set of crawlers has been located for a later date. Treat the December 2024 result as a documented observation for that sample and period. Do not treat it as a permanent property of ChatGPT, Claude, or any other system. Retest before making architecture decisions based on it.
What OpenAI and Anthropic document about their crawlers
OpenAI and Anthropic each publish crawler documentation, but these pages describe purpose and access control. Neither page establishes JavaScript rendering behavior, so keep those categories separate.
OpenAI describes OAI-SearchBot as serving ChatGPT search features, and GPTBot as collecting content that may be used to improve its foundation models. It states that the robots.txt settings for these two bots are independent. It describes ChatGPT-User as user-initiated access rather than automatic web crawling. The page does not promise that any of these agents execute JavaScript, and purpose alone is not evidence of rendering. (OpenAI crawler documentation)
Anthropic’s help article, dated April 7, 2026, distinguishes ClaudeBot (potential training-data collection), Claude-SearchBot (search), and Claude-User (user-directed requests). It says Anthropic’s bots respect standard robots.txt directives. It does not specify whether they render JavaScript. (Anthropic crawler guidance)
| Crawler | Purpose stated by the vendor | Documented JavaScript rendering | Observed in the Vercel/MERJ sample (December 2024) |
|---|---|---|---|
| Googlebot (Google Search) | Search crawling and indexing | Yes, via a rendering queue with headless Chromium | Not part of the non-rendering finding |
| OAI-SearchBot | ChatGPT search features | Not stated by OpenAI | Did not render JavaScript |
| GPTBot | Collecting content that may be used to improve foundation models | Not stated by OpenAI | Did not render JavaScript |
| ChatGPT-User | User-initiated access | Not stated by OpenAI | Did not render JavaScript; fetched JavaScript files without executing them |
| ClaudeBot | Potential training-data collection | Not stated by Anthropic | Did not render JavaScript; fetched JavaScript files without executing them |
| Claude-SearchBot | Search | Not stated by Anthropic | Not stated (not among the agents named in the report) |
| Claude-User | User-directed requests | Not stated by Anthropic | Not stated (not among the agents named in the report) |
Meta-ExternalAgent, Bytespider, and PerplexityBot were also named in the Vercel/MERJ sample as not rendering JavaScript. Their vendor documentation was not reviewed for this article, so no purpose is stated for them here.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to check what a crawler can read on your page
You can separate the two versions with a few checks. None of them proves what every crawler does, but they show whether critical content depends on scripts.
- Fetch the raw response. Run
curl -s https://www.example.com/your-page/ -o raw.html, then search the file for a sentence from the main body, the page title, the meta description, and an internal link. If they are missing, the content is not in the initial response. - Check View Source. In Chrome, right-click the page and choose View page source. This shows the same initial HTML in a browser, not the DOM after scripts run.
- Compare with the live DOM. In Chrome DevTools, open the Elements panel. Content that appears here but was missing in step 1 is produced by JavaScript.
- Check blocking rules. Open your robots.txt file and confirm that the scripts, stylesheets, and API paths used to build the page are not disallowed for the crawlers you care about.
- Check the status code. A non-200 response can cause Google to skip rendering. Confirm that the page returns a 200 status for the URL you want indexed.
If step 1 shows the article text, title, and links, the core content is already in the initial response. If it does not, the page depends on JavaScript for that content.
Choosing an architecture for content that must be found
Server-rendered or prerendered HTML is the safer default for content that matters to discovery. That includes article bodies, product titles and descriptions, metadata, and crawlable internal links. Vercel and MERJ recommended server-side rendering, incremental static regeneration, or static site generation for important content, based on their observations. Google’s documentation supports server-side rendering or prerendering for the same reason: not all bots can run scripts.
Client-side rendering can still be useful for nonessential behavior, such as filters, personalization, interactive widgets, or enhancements that users can do without. The rule is to keep the content that defines the page in the initial response, and to let JavaScript improve the experience around it.
Server-side rendering does not guarantee ranking, citations, or inclusion in any AI answer. It removes one dependency. Crawl access, page quality, and each platform’s own retrieval decisions still determine what happens next.
Limits of what is established
- The direct evidence for AI crawler rendering is one observational study with a stated sample and period. A globally representative figure for how many AI crawlers render JavaScript has not been established.
- OpenAI and Anthropic’s crawler pages define purposes and controls. They do not describe JavaScript rendering.
- Google’s documentation covers Google Search crawling and indexing. It does not cover every Google product or every AI answer path.
When a crawler’s behavior matters for a decision, check its current documentation and test your own pages against the initial response.
Quick Recap
The Bottom Line
“”
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
