Decoding Googlebot crawl stats data in Google Search Console

Tucked away within the Settings part of Google Search Console is a report few SEO professionals focus on, however I like to watch.

These stories are often called Crawl Stats.

Right here, you’ll discover an fascinating set of metrics on Googlebot crawl exercise. These metrics are particularly helpful for web sites with 1000’s or tens of millions of pages.

Googlebot ‘Crawl stats’ 

Way back, Google Search Console had easy-to-find metrics on Googlebot crawl exercise. Then, it seemingly disappeared.

In actuality, it was tucked away within the Settings part.

The way to entry the crawl stats stories:

• Click on on Settings on the backside of the left navigation.
• Go to the Crawl stats part.
• Click on Open report.

In regards to the crawl stats knowledge

As Googlebot crawls your web site, Google tracks and stories on numerous features of Googlebot’s exercise and stories on it in Google crawl stats.

That is the place you’ll discover high-level statistics about Google’s crawling historical past in your web site. 

Google says this knowledge is for superior customers

The Googlebot Crawl Stats knowledge is just not for the technical SEO rookies.

Google particularly says this knowledge is geared toward “superior customers” with 1000’s of pages on their web site, which can be why it’s positioned in such an uncommon location, unseen by many within the search engine optimisation group.

One motive Google could understand this as a complicated report is that so many issues can affect these metrics, together with community points and cloud supply providers equivalent to Akamai.

Who will discover crawl stats most helpful?

I discover the Crawl Stats stories much less of an “superior” set of stories however one thing that’s extra helpful to enterprise SEOs with out crawler monitoring instruments equivalent to Lumar and Botify. 

When doing search engine optimisation on an enterprise web site with 1000’s or tens of millions of pages, crawler optimization is a crucial process, and crawler exercise metrics present essential perception for outlining motion gadgets.

Smaller websites possible don’t want to fret an excessive amount of about crawler exercise as a result of there may be in all probability sufficient crawl funds allotted to your web site to crawl at an applicable tempo. 

Alternatively, enterprise websites are likely to have much more pages that should be crawled, found, and/or refreshed than Google crawls their web site every day.

Because of this, they have to optimize for crawler exercise, which is a instrument to assist information subsequent steps.

What to search for on this knowledge

After years of reviewing this knowledge throughout many websites, I’ve one main rule:

• Don’t spend quite a lot of time right here except you see fluctuations and correlations.

Usually these stories are fascinating however not actionable.

Instance fluctuations that I have a tendency to research: 

• HTML requests decreased (or spiked) on the similar time Bytes of JavaScript downloaded elevated (or spiked).
• Common response time elevated (or spiked) on the similar time the variety of HTML requests went down (or sudden fall).
• Complete obtain dimension elevated (or spiked), however the variety of HTML requests didn’t change.
• The % of requests for discovery (to find new URLs) will increase and the % of requests for refresh goes down; nonetheless, you didn’t launch new URLs on the positioning.

When to take a look at this crawl stats

Crawl stats are good to peruse (and log) at the least as soon as a month.

They’re particularly good to watch after main releases, equivalent to a platform migration or main redesign. This can provide help to shortly perceive how Google is responding to your newly launched adjustments.

Keep in mind: When you have a bot monitoring instrument equivalent to Lumar or Botify, then this knowledge isn’t as helpful as you’ll discover within the bot monitoring knowledge offered by these instruments.

Caveats concerning the crawl stats knowledge

Many issues can affect Google’s crawl stats metrics past a traditional dev launch.

This implies the search engine optimisation workforce, product supervisor(s) and developer(s) should assume exterior the field when evaluating the fluctuations.

You have to think about what might have prompted a rise or lower in Googlebot crawl exercise, not solely in your launch but in addition throughout the community and tech stack.

Adjustments to one thing equivalent to Akamai might probably impression this knowledge. 

Log the crawl stats knowledge in a spreadsheet

That is knowledge I prefer to archive as a result of Google stories such a small window of time.

An ideal instance of it is a problem I’m dealing with now with a consumer. What’s reported in GSC proper now appears to be like like issues are enhancing:

Nonetheless, as a result of I’ve metrics from six months in the past, I can say that these metrics are 40% larger than they have been six months in the past.

Whereas they’re trending down, they’re nonetheless worse than they have been prior to now. The consumer’s problem is that growth has no concept why that is occurring (sadly, fixing that drawback is past the scope of this text).

It’s possible you’ll assume to only seize a screenshot. Nonetheless, it makes it very onerous to match over time.

Discover there isn’t any left axis within the chart. You actually can not inform what the strains replicate. (Notice: Numbers do seem on the left/proper axis if you find yourself solely viewing two metrics within the chart)

As a substitute, drop this knowledge right into a spreadsheet. Then, you’ve got precise knowledge that may be charted over time, calculated and used to match with different metrics, equivalent to visits. 

Having the historic knowledge in a single place is commonly helpful when discussing main adjustments with growth to indicate how a lot better the metrics have been 4-6+ months in the past. 

Keep in mind, growth likes onerous, particular knowledge, so charts with precise numbers on the left/proper axis (or worse, no numbers on the x-axis in any respect) will probably be extra helpful to you than charts with various numbers on the x-axis.

Keep in mind, the stories packing containers are paginated

Although a very powerful metrics you’ll want are possible seen within the default view, lots of the report sections are paginated – they usually’re simple to overlook! 

---

Which metrics to watch and why

Let’s get into the first metrics to look (in a short time) every month, together with a couple of suggestions to remove motion gadgets from the info:

Complete crawl requests

• View this report in Google Search Console (positioned within the high chart).
• Google definition: “The full variety of crawl requests issued for URLs in your web site, whether or not profitable or not.”
• If this metric goes up or down, examine it with common response time and complete obtain dimension (bytes).
  • An apparent motive for this metric might go up should you change quite a lot of code or launch quite a lot of new pages. Nonetheless, that’s not at all the one trigger.

Complete obtain dimension (byte)

• View this report in Google Search Console (positioned within the high chart).
• Google definition: “Complete variety of bytes downloaded out of your web site throughout crawling, for the desired time interval.”
• If this metric goes up or down, examine it with common response time
• An apparent trigger for this metric to extend is including quite a lot of code throughout 1000’s of pages or launching quite a lot of new pages. Nonetheless, that’s not at all the one trigger.

Common response time (ms)

• Google Search Console Report (positioned within the high chart).
• Google definition: “Common response time for all sources fetched out of your web site throughout the specified time interval.”
• If this metric goes up or down, examine with with complete crawl requests and complete obtain dimension (bytes).

Crawl requests breakdown by response 

• View this report in Google Search Console (positioned under the highest chart).
• Google definition: “This desk exhibits the responses that Google obtained when crawling your web site, grouped by response kind, as a share of all crawl responses…”
• Widespread responses:
  • OK (200).
  • Moved completely (302).
  • Server error (5xx).
  • Different consumer error (4xx).
  • Not discovered (404).
  • Not modified (304).
  • Web page timeout.
  • Robots.txt not out there.
  • Redirect error.

Crawl requests breakdown by file kind

• View this report in Google Search Console. 
• Google definition: “The file kind returned by the request. Proportion worth for every kind is the proportion of responses of that kind, not the proportion of of bytes retrieved of that kind.”
• Widespread responses:
  • JSON.
  • HTML.
  • JavaScript.
  • Picture.
  • PDF.
  • CSS.
  • Syndication.
  • Different XML.
  • Video.
  • Different file kind.
  • Unknown (failed requests).

Crawl requests breakdown by crawl goal

• View this report in Google Search Console.
• Two purposes:
• That is an fascinating metric for shows; nonetheless, it solely has a couple of helpful use instances. For instance:
  • If the % of Googlebot exercise that’s for Discovery all of the sudden will increase, however we’re not including URLs to the positioning, then you’ve got an motion merchandise to determine what’s being crawled that shouldn’t be crawled. 
  • If the % of Googlebot exercise that’s for Refresh decreases considerably, however you didn’t take away pages from the positioning, then you’ve got an motion merchandise to determine why fewer present pages are being crawled.

Crawl requests breakdown by Googlebot kind

• View this report in Google Search Console.
• Google definition: “The kind of person agent used to make the crawl request. Google has plenty of person brokers that crawl for various causes and have completely different behaviors.”
• It’s an fascinating metric, however not very helpful. It simply exhibits Google remains to be utilizing their desktop crawler. Actually, I normally ignore these metrics.

You may click on into every metric for extra knowledge

Usually whenever you current any search engine optimisation concern to product managers and builders, they usually wish to see instance URLs. You may click on on any of the metrics listed on this report and get instance URLs.

An fascinating metric to take a look at is “different file varieties” as a result of it’s not clear what’s within the “different file varieties” class (usually it’s font information).

The screenshot under exhibits the examples report for “different file kind.” Each file listed is a font file (blurred out for confidentiality causes).

On this report of examples, every row displays one crawl request. This implies if a web page is crawled a number of instances it may very well be listed greater than as soon as within the “examples.” 

As with all Google Search Console stories, it is a knowledge pattern and never each request from Googlebot.

Do you share these metrics with builders and product managers?

These metrics will usually generate considered one of two ideas: 

• “There’s nothing to take a look at right here.” 
• “What might have prompted that?” 

In my expertise, the solutions to “what prompted that” are likely to require the enter of product managers and/or builders.

When presenting the info and your questions on potential causes for points, keep in mind to obviously clarify that these metrics will not be person exercise and solely characterize Googlebot’s exercise and expertise on the web site.

I discover product managers and builders usually get a bit confused when discussing this knowledge, particularly if it doesn’t match up with different metrics they’ve seen or details they know concerning the web site. 

By the way in which, this usually occurs for many Google Search Console knowledge conversations.

If there are not any Crawl Stats fluctuations or correlations to be involved about, don’t carry it as much as growth, nor product administration. It simply turns into noise and prevents them from specializing in extra essential metrics.

What’s subsequent? 

Take a look at your crawl stats to verify there are not any spikes or correlations which are regarding. 

Then, decide how usually you wish to have a look at these and arrange techniques that immediate you to test these and different Google Search Console metrics in a scientific, analytical technique every month.

When you try your Googlebot Crawl Stats, I’ll write Half 4 on this sequence that can speak about the way to know which URLs you must deal with for technical search engine optimisation enhancements and specifically, Core Internet Vitals metrics.

Dig deeper

That is the third article in a sequence recapping my SMX Superior presentation on the way to flip search engine optimisation metrics into motion gadgets. Beneath are hyperlinks to the primary two articles: