Google has utterly reorganized its crawlers and user-triggered fetchers documentation. It used to be all on one web page and now it’s in a number of pages. Many of the modifications have been simply transferring content material round however Google did add sections for what product every crawler impacts, and added a robots.txt snippet for every crawler to display tips on how to use the person agent tokens.
Google wrote, “The documentation grew very lengthy which restricted our skill to increase the content material about our crawlers and user-triggered fetchers,” so that’s the reason they redid it.
Should you dig into every crawl, for instance Googlebot you will note two new sections:
(1) Affected merchandise
(2) Instance robots.txt group
Here’s a pattern screenshot however what I highlighted in crimson was added for each crawler:
Right here is how every crawler impacts merchandise:
- Googlebot: Crawling preferences addressed to the Googlebot person agent have an effect on Google Search (together with Uncover and all Google Search options), in addition to different merchandise equivalent to Google Photographs, Google Video, Google Information, and Uncover.
- Googlebot Picture: Crawling preferences addressed to the Googlebot-Picture person agent have an effect on Google Photographs, Uncover, Google Video, and all options in Google Search the place pictures, logos, and favicons are offered.
- Googlebot Video: Crawling preferences addressed to the Googlebot-Video person agent have an effect on video-related Google Search options and different merchandise depending on movies.
- Googlebot Information: Crawling preferences addressed to the Googlebot-Information person agent have an effect on all surfaces of Google Information (for instance, the Information tab in Google Search and the Google Information app).
- Google StoreBot: Crawling preferences addressed to the Storebot-Google person agent have an effect on all surfaces of Google Procuring (for instance, the Procuring tab in Google Search and Google Procuring).
- Google-InspectionTool: Crawling preferences addressed to the Storebot-Google person agent have an effect on Search testing instruments such because the Wealthy Outcome Check and URL inspection in Search Console. It has no impact on Google Search or different merchandise.
- GoogleOther: Crawling preferences addressed to the GoogleOther person agent do not have an effect on any particular product. GoogleOther is the generic crawler that could be utilized by numerous product groups for fetching publicly accessible content material from websites. For instance, it might be used for one-off crawls for inner analysis and improvement. It has no impact on Google Search or different merchandise.
- GoogleOther-Picture: Crawling preferences addressed to the GoogleOther-Picture person agent do not have an effect on any particular product, just like GoogleOther. GoogleOther-Picture is the model of GoogleOther optimized for fetching publicly accessible picture URLs.
- GoogleOther-Video: Crawling preferences addressed to the GoogleOther-Video person agent do not have an effect on any particular product, just like GoogleOther. GoogleOther-Video is the model of GoogleOther optimized for fetching publicly accessible video URLs.
- Google-CloudVertexBot: Crawling preferences addressed to the Google-CloudVertexBot person agent have an effect on crawls requested by the positioning house owners’ for constructing Vertex AI Brokers. It has no impact on Google Search or different merchandise.
- Google-Prolonged: Google-Prolonged is a standalone product token that net publishers can use to handle whether or not their websites assist enhance Gemini Apps and Vertex AI generative APIs, together with future generations of fashions that energy these merchandise. Google-Prolonged doesn’t affect a web site’s inclusion or rating in Google Search.
- APIs-Google: Crawling preferences addressed to the APIs-Google person agent have an effect on the supply of push notification messages by Google APIs.
- AdsBot Cellular Net: Crawling preferences addressed to the AdsBot-Google-Cellular person agent have an effect on Google Adverts’ skill to examine net web page advert high quality.
- AdsBot: Crawling preferences addressed to the AdsBot-Google person agent have an effect on Google Adverts’ skill to examine net web page advert high quality.
- AdSense: Crawling preferences addressed to the Mediapartners-Google person agent have an effect on Google AdSense. The AdSense crawler visits collaborating websites with a view to present them with related adverts.
- Google-Security: The Google-Security person agent handles abuse-specific crawling, equivalent to malware discovery for publicly posted hyperlinks on Google properties. As such it is unaffected by crawling preferences.
- Feedfetcher: Feedfetcher is used for crawling RSS or Atom feeds for Google Information and PubSubHubbub.
- Google Writer Middle: Google Writer Middle fetches and processes feeds that publishers explicitly provided to be used in Google Information touchdown pages.
- Google Learn Aloud: Upon person request, Google Learn Aloud fetches and reads out net pages utilizing text-to-speech (TTS).
- Google Website Verifier: Google Website Verifier fetches Search Console verification tokens.
Google wrote:
Reorganized the documentation for Google’s crawlers and user-triggered fetchers. We additionally added express notes about what product every crawler impacts, and added a robots.txt snippet for every crawler to display tips on how to use the person agent tokens. There have been no significant modifications to the content material in any other case.
Google additionally added details about the content material encodings (compressions) supported by Google’s crawlers and user-triggered fetchers. That is simply documentation change, no change in conduct. Google additionally up to date the URL within the GoogleProducer HTTP person agent string within the documentation for Google’s user-triggered fetchers to match the worth utilized by the precise fetcher.
Discussion board dialogue at X.
Picture credit score Lizzi Sassman
