Google quietly added a brand new bot to their crawler documentation that crawls on behalf of economic shoppers of their Vertex AI product. It seems that the brand new crawler might solely crawl websites managed by the positioning house owners, however the documentation isn’t fully clear on that time.
Vertex AI Brokers
Google-CloudVertexBot, the brand new crawler, ingests web site content material for Vertex AI shoppers, not like different bots listed within the Search Central documentation which can be tied to Google Search or promoting.
The official Google Cloud documentation provides the next data:
“In Vertex AI Agent Builder, there are numerous varieties of knowledge shops. A knowledge retailer can comprise just one sort of knowledge.”
It goes on to checklist six kinds of knowledge, one among which is public web site knowledge. On crawling the documentation says that there are two sorts of web site crawling with limitations particular to every sort.
- Fundamental web site indexing
- Superior web site indexing
Documentation Is Complicated
The documentation explains web site knowledge:
“A knowledge retailer with web site knowledge makes use of knowledge listed from public web sites. You possibly can present a set of domains and arrange search or suggestions over knowledge crawled from the domains. This knowledge contains textual content and pictures tagged with metadata.”
The above description doesn’t say something about verifying domains. The outline of Fundamental web site indexing doesn’t say something about web site proprietor verification both.
However the documentation for Superior web site indexing does say that area verification is required and in addition imposes indexing quotas.
Nevertheless, the documentation for the crawler itself says that the brand new crawler crawls on the “web site house owners’ request” so it could be that it gained’t come crawling public websites.
Now right here’s the complicated half, the Changelog notation for this new crawler signifies that the brand new crawler might come to scrape your web site.
Right here’s what the changelog says:
“The brand new crawler was launched to assist web site house owners establish the brand new crawler visitors.”
New Google Crawler
The brand new crawler is known as Google-CloudVertexBot.
That is the brand new data on it:
“Google-CloudVertexBot crawls websites on the positioning house owners’ request when constructing Vertex AI Brokers.
Person agent tokens
- Google-CloudVertexBot
- Googlebot”
Person agent substring
Google-CloudVertexBot
Unclear Documentation
The documentation appears to point that the brand new crawler doesn’t index public websites however the changelog signifies that it was added in order that web site house owners can establish visitors from the brand new crawler. Do you have to block the brand new crawler with a robots.txt simply in case? It’s not unreasonable to contemplate provided that the documentation is pretty unclear on whether or not it solely crawls domains which can be verified to be underneath the management of the entity initiating the crawl.
Learn Google’s new documentation:
Featured Picture by Shutterstock/ShotPrime Studio
