The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →GPTBot can appear in WordPress access logs because OpenAI identifies it with an HTTP user-agent. Google-Extended is not a separate crawler identity: Google says it has no separate HTTP user-agent string, so you should not expect to see “Google-Extended” as a visitor in request logs. It is a robots.txt control token for certain uses of content Google crawls.
Why GPTBot can appear in a WordPress log
An HTTP user-agent is request metadata that identifies the software making a request. OpenAI documents GPTBot as a crawler with a distinct user-agent identity; its example includes GPTBot/1.4. A request carrying that text may therefore be recorded in a WordPress or server access log, depending on what that logging setup captures. The example version is not a promise that every request will use that exact string.
As an Amazon Associate I earn from qualifying purchases.
OpenAI says GPTBot crawls content that may be used to train its generative AI foundation models. It is separate from OAI-SearchBot, which OpenAI uses to surface websites in ChatGPT search. OpenAI says the settings for GPTBot and OAI-SearchBot are independent, so allowing or disallowing one should not be treated as controlling the other.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsWhy Google-Extended does not appear as a visitor
Google-Extended is a robots.txt control token, not a separate HTTP crawler user-agent. Google states, “Google-Extended doesn’t have a separate HTTP request user agent string.” Google crawls with its existing user-agent identities; Google-Extended is used in robots.txt to control certain uses of content Google crawls.
#1 Best Overall
Google says the token can control whether crawled content is used to train future Gemini generations and for grounding in Gemini Apps and Vertex AI. Google also says Google-Extended does not affect whether a page is included in Google Search and is not a Search ranking signal. In other words, robots.txt policy and access-log identity answer different questions: the token expresses a content-use preference, while the request log records HTTP requests and their metadata.
What to look for in logs and robots.txt
- To identify a claimed GPTBot request: inspect the request’s user-agent field for GPTBot. A matching string is a claimed identity, not proof that OpenAI originated the request.
- To understand Google crawling: inspect the actual Google user-agent in the request, rather than searching for Google-Extended. Google says a Googlebot subtype can be identified from the HTTP user-agent header.
- To check Google-Extended policy: inspect the site’s robots.txt rules for the Google-Extended token. Its presence there does not imply there should be a matching visitor in access logs.
- To distinguish OpenAI traffic: do not conflate GPTBot with OAI-SearchBot or ChatGPT-User. OpenAI describes ChatGPT-User as being used for certain user-triggered page visits, not automatic web crawling, and says robots.txt rules may not apply to those visits.
How to verify a claimed crawler
User-agent text can be spoofed. Google warns, “The HTTP user agent string can be spoofed.” If reliable attribution matters, compare the source IP with the provider’s published crawler IP information and follow the provider’s verification guidance. Google recommends reverse DNS lookup or checking the IP against its published crawler ranges. OpenAI also publishes GPTBot IP addresses. Check current vendor documentation and IP data before using them to verify or block traffic, because examples and published addresses can change.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What a missing log entry does—and does not—prove
The absence of a Google-Extended line is expected because Google-Extended is not a separate request user-agent. The absence of a GPTBot line, by itself, does not establish that GPTBot never fetched a page. What appears in a particular site’s records depends on its logging configuration and infrastructure: a WordPress plugin, origin server, reverse proxy, CDN, retention period, or filtering rule may capture different requests or fields. To explain a specific site’s records, check the actual request and infrastructure logs as well as its robots.txt file.
Google’s relevant documentation is Google’s common crawlers and its Googlebot reference. OpenAI’s Overview of OpenAI Crawlers describes its crawler identities and published IP information.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
