I wanted to know which AI crawlers actually fetch my site—and, in the question I kept coming back to, “did any of them come?” A small server-side counter showed requests from indexing, answer-time, and training agents, but the most important discovery was what those agents received: my most-fetched URL was a JavaScript application shell with no numbers in its HTML. That changed what I built next.
Three kinds of AI-agent requests answer different questions
I grouped requests by apparent purpose rather than treating every automated visit as one “bot hit.” These labels are useful for interpreting a counter, but identifying a crawler from its user-agent string is only a practical approximation, not a durable or complete verification method.
As an Amazon Associate I earn from qualifying purchases.
| Role | Examples in my counter | What a request suggests | What it does not establish |
|---|---|---|---|
| Index | OAI-SearchBot, PerplexityBot, Claude-SearchBot | An agent is crawling on its own schedule to build a searchable corpus. | That a person asked a question, clicked a citation, or saw the page. |
| On-behalf | ChatGPT-User, Perplexity-User, Claude-User | An assistant fetched the page after a person asked it something. | That the person opened the page or that the request itself was a click. |
| Training | GPTBot, ClaudeBot, Google-Extended | A request categorized as collecting material for model training. | That the content was used to train a model or that the fetch relates to search. |
I tracked referral click-throughs separately. In my example, ChatGPT adds utm_source=chatgpt.com to rendered links; I treated visits with that attribution as people choosing to open a citation. An on-behalf fetch is a separate event, not proof of a human visit.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →How I counted requests without relying on browser analytics
I added roughly a hundred lines of server-side counting. Because it runs when the server receives a request, it can count a page fetch even when no browser analytics script executes. Client-side analytics, by contrast, can observe activity only when its code runs in a client; it is not a substitute for counting requests that never execute that code. Neither method alone answers every question: server counts show requests and paths, while attributed visits can help distinguish a citation click from an assistant fetch.
My illustrative matcher classifies user agents by substring. The order matters: I check a specific string such as chatgpt-user before a broader one such as gptbot, so a request is not swallowed by an earlier, less specific match.
function classify(userAgent) {
const ua = userAgent.toLowerCase();
if (ua.includes("chatgpt-user")) return "on-behalf";
if (ua.includes("gptbot")) return "training";
// Add other specific matchers before broader ones.
return null;
}This is a sketch of the matching idea, not a complete crawler directory or proof that a request claiming a particular user agent came from that organization. Keep the categories aligned with the question you want the counter to answer.
What the first half-hour showed—and what it cannot tell us
Roughly thirty minutes after I shipped the counter, I saw this snapshot on one small site:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
| Category | Requests or visits observed |
|---|---|
| Index requests | 9 |
| On-behalf fetches | 15, all attributed in my example to ChatGPT-User |
| Training requests | 0 |
| Click-throughs | 0 |
As I wrote on DEV Community on August 25, 2026, “The honest caveat on all of this: it is roughly thirty minutes of data from a single site in a niche, and some of those fetches were me testing. It is not a study.” These are my reported observations, not an industry benchmark or evidence that assistants generally fetch pages at any particular rate. Read the original account on DEV Community.
The most-fetched URL was reachable but did not contain the answer
The most-fetched URL was my JavaScript application shell. Its HTML contained no numbers, so a request reaching that URL did not mean the useful reading was present in the response. A new page I had linked internally had not appeared in the fetches I saw.
In the search mode I observed, I inferred that the assistant opened returned search results rather than following links from a page it had just read. That is my interpretation of this small snapshot, not a rule for every assistant, search mode, or site. The practical check is more direct: inspect the response body for the URLs that receive requests. A URL can exist and be internally linked yet still return a shell that does not expose the content you want fetched.
Rank #4
Why I put the current reading in a no-JavaScript fallback
I moved the current reading into a <noscript> block in the HTML shell. That way, the same content is served to everyone, and the block is also a genuine fallback for people whose browsers do not run JavaScript; without it, that page would otherwise be blank. The change addressed what the fetched HTML contained, rather than merely adding another internal link.
Keeping a small counter narrow and bounded
My implementation aggregates counts by day and path in memory, then persists them on a debounced flush. It also bounds the number of distinct paths recorded per agent per day. If mirroring the counter to a paid key-value store, I would persist the accumulated day rather than write on every request.
Best Value
I record the matched agent, path, and date. I do not retain IP addresses, accounts, sessions, verbatim user-agent strings, or query-string values other than the attribution parameter. Counts expire after sixty days. Those are choices for my implementation, not a standard retention period or a privacy rule; adding the tracking also meant updating my site’s privacy policy.
Quick Recap
Two shutdown and loading bugs to avoid
- Flush ordering: An earlier shutdown handler that calls
process.exit(0)can prevent a later-registered handler from flushing pending counts. Ensure persistence completes before the process exits. - Missing data file: When the loader cannot find its file, clear in-memory state rather than inheriting old counts. A test caught this load-or-reset problem in my implementation.
What I would check before drawing conclusions
- Separate indexing, on-behalf fetching, and training categories; a single total conceals different purposes.
- Count attributed citation visits separately from assistant fetches.
- Inspect the actual HTML returned by fetched URLs, especially when the page is a JavaScript application.
- Keep a short, single-site snapshot in perspective; it cannot establish general crawler behavior.
- Limit stored fields and choose an explicit retention period that fits your own privacy obligations.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
