Google updates crawlers and user-triggered fetchers documentation

Google made a sequence of updates to its crawlers and user-triggered fetchers documentation, principally breaking out the single-page doc into a number of pages and paperwork. However Google additionally expanded what product every crawler impacts with a brand new part subsequent to every crawler and added a robots.txt snippet for every crawler to display how one can use the consumer agent tokens by every crawler.

What Google mentioned. Google posted about these adjustments saying:

“Reorganized the documentation for Google’s crawlers and user-triggered fetchers. We additionally added specific notes about what product every crawler impacts, and added a robots.txt snippet for every crawler to display how one can use the consumer agent tokens. There have been no significant adjustments to the content material in any other case.”

“The documentation grew very lengthy which restricted our skill to increase the content material about our crawlers and user-triggered fetchers.”

What’s new. In addition to for simply transferring plenty of content material round, to prepare the content material. Google additionally added the “affected merchandise” sections and in addition the “Instance robots.txt group” sections. Here’s a screenshot of this utilized to the Googlebot crawler, nevertheless it was added to every particular person crawler:

I pulled out each for you:

• Googlebot: Crawling preferences addressed to the Googlebot consumer agent have an effect on Google Search (together with Uncover and all Google Search options), in addition to different merchandise corresponding to Google Pictures, Google Video, Google Information, and Uncover.
• Googlebot Picture: Crawling preferences addressed to the Googlebot-Picture consumer agent have an effect on Google Pictures, Uncover, Google Video, and all options in Google Search the place photographs, logos, and favicons are offered.
• Googlebot Video: Crawling preferences addressed to the Googlebot-Video consumer agent have an effect on video-related Google Search options and different merchandise depending on movies.
• Googlebot Information: Crawling preferences addressed to the Googlebot-Information consumer agent have an effect on all surfaces of Google Information (for instance, the Information tab in Google Search and the Google Information app).
• Google StoreBot: Crawling preferences addressed to the Storebot-Google consumer agent have an effect on all surfaces of Google Procuring (for instance, the Procuring tab in Google Search and Google Procuring).
• Google-InspectionTool: Crawling preferences addressed to the Storebot-Google consumer agent have an effect on Search testing instruments such because the Wealthy End result Take a look at and URL inspection in Search Console. It has no impact on Google Search or different merchandise.
• GoogleOther: Crawling preferences addressed to the GoogleOther consumer agent don’t have an effect on any particular product. GoogleOther is the generic crawler that could be utilized by numerous product groups for fetching publicly accessible content material from websites. For instance, it could be used for one-off crawls for inner analysis and growth. It has no impact on Google Search or different merchandise.
• GoogleOther-Picture: Crawling preferences addressed to the GoogleOther-Picture consumer agent don’t have an effect on any particular product, much like GoogleOther. GoogleOther-Picture is the model of GoogleOther optimized for fetching publicly accessible picture URLs.
• GoogleOther-Video: Crawling preferences addressed to the GoogleOther-Video consumer agent don’t have an effect on any particular product, much like GoogleOther. GoogleOther-Video is the model of GoogleOther optimized for fetching publicly accessible video URLs.
• Google-CloudVertexBot: Crawling preferences addressed to the Google-CloudVertexBot consumer agent have an effect on crawls requested by the positioning house owners’ for constructing Vertex AI Brokers. It has no impact on Google Search or different merchandise.
• Google-Prolonged: Google-Prolonged is a standalone product token that internet publishers can use to handle whether or not their websites assist enhance Gemini Apps and Vertex AI generative APIs, together with future generations of fashions that energy these merchandise. Google-Prolonged doesn’t impression a web site’s inclusion or rating in Google Search.
• APIs-Google: Crawling preferences addressed to the APIs-Google consumer agent have an effect on the supply of push notification messages by Google APIs.
• AdsBot Cellular Net: Crawling preferences addressed to the AdsBot-Google-Cellular consumer agent have an effect on Google Adverts’ skill to examine internet web page advert high quality.
• AdsBot: Crawling preferences addressed to the AdsBot-Google consumer agent have an effect on Google Adverts’ skill to examine internet web page advert high quality.
• AdSense: Crawling preferences addressed to the Mediapartners-Google consumer agent have an effect on Google AdSense. The AdSense crawler visits collaborating websites with a view to present them with related adverts.
• Google-Security: The Google-Security consumer agent handles abuse-specific crawling, corresponding to malware discovery for publicly posted hyperlinks on Google properties. As such it’s unaffected by crawling preferences.
• Feedfetcher: Feedfetcher is used for crawling RSS or Atom feeds for Google Information and PubSubHubbub.
• Google Writer Middle: Google Writer Middle fetches and processes feeds that publishers explicitly provided to be used in Google Information touchdown pages.
• Google Learn Aloud: Upon consumer request, Google Learn Aloud fetches and reads out internet pages utilizing text-to-speech (TTS).
• Google Website Verifier: Google Website Verifier fetches Search Console verification tokens.

Why we care. Studying by these affected product sections might assist you higher perceive how every crawler impacts numerous facets of Google. Some don’t impression Google Search in any respect, whereas others are elementary to how Google Search works.

Additionally the brand new robots.txt examples could also be very helpful to you and your growth groups.