Google added eight or so phrases between brackets to make clear that fields that are not explicitly listed within the robots.txt documentation aren’t supported. Why did they do that? Google said as a result of typically Google will get questions on it.
The phrases Google added had been added to the syntax section of the robots.txt documentation and it reads:
(different fields corresponding to crawl-delay aren’t supported)
Right here is the earlier than and after, the place the addition is highlighted in blue:
Here’s what it appears to be like like now:
As Jaskaran Singh, Silver Product Skilled at Google in Webmaster/Search Console Group, wrote on LinkedIn, “In case you’re utilizing unsupported fields, like crawl-delay, Google will ignore them. You should definitely comply with these supported directives to make sure your web site is crawled as meant.” He added as “Professional Tip: At all times use feedback (#) for higher readability, and guarantee paths begin with / to keep away from errors.”
Discussion board dialogue at LinkedIn.
