Gary Illyes from Google posted on LinkedIn that his mission this yr is to “determine easy methods to crawl even much less, and have fewer bytes on wire.” He added that Googlebot ought to “be extra clever about caching and inner cache sharing amongst consumer brokers, and we must always have fewer bytes on wire.”
He added, “Lowering crawling with out sacrificing crawl-quality would profit everybody.”
Immediately, Gary added, that Google is crawling as a lot because it did earlier than – regardless of some of us pondering Google is crawling much less. He mentioned “Within the grand scheme of issues that is simply not the case; we’re crawling roughly as a lot as earlier than.”
What Google is best at that earlier than is scheduling. “Nonetheless scheduling obtained extra clever and we’re focusing extra on URLs that extra prone to deserve crawling,” he defined.
It appears Microsoft Bing, particularly Fabrice Canel from Microsoft and Gary Illyes from Google have the identical objectives. Microsoft is tackling it by encouraging web site homeowners to make use of IndexNow. Google mentioned in November 2021 that Google might consider adopting IndexNow however that got here and went…
John Mueller from Google commented on the submit suggesting, “We might simply crawl RSS feeds and create some form of Reader.” A joke about Google Reader…
Anyway – we’ll see what Google finally ends up doing right here. Right here is his full submit:
My mission this yr is to determine easy methods to crawl even much less, and have fewer bytes on wire.
Just a few days in the past there was a submit on a Reddit group about how, within the OC’s notion, Google is crawling lower than earlier years. Within the grand scheme of issues that is simply not the case; we’re crawling roughly as a lot as earlier than, nevertheless scheduling obtained extra clever and we’re focusing extra on URLs that extra prone to deserve crawling.
Nonetheless, we must always, actually, crawl much less. We should always, for instance, be extra clever about caching and inner cache sharing amongst consumer brokers, and we must always have fewer bytes on wire.
When you’ve seen an attention-grabbing IETF (or different requirements physique) web draft that would assist with this effort, or an precise normal I’d’ve missed, ship it my means. Lowering crawling with out sacrificing crawl-quality would profit everybody.
Discussion board dialogue at LinkedIn.
