Do you utilize a CDN for some or all your web site and also you wish to handle only one robots.txt file, as an alternative of each the CDN’s robots.txt file and your foremost web site’s robots.txt file? Gary Illyes from Google mentioned that each one it is advisable to do is redirect your foremost robots.txt to the CDN and management it on the CDN.
Gary wrote on LinkedIn:
You’ve got a CDN and you’ve got your foremost web site. You’ve got two robots.txt information, one at https∶//cdn․instance․com/robots.txt and one at https∶//www․instance․com/robots.txt. You might have only one central robots.txt with all the foundations, say in your CDN, which could allow you to with holding observe of all guidelines it is advisable to handle. All you need to do is to redirect https∶//www․instance․com/robots.txt to https∶//cdn․instance․com/robots.txt and crawlers that comply to RFC9309 will simply use the redirect goal because the robotstxt file of https∶//www․instance․com/. Bizarre or what.
Now, this isn’t an web optimization rating tip. It’s only a technique to preserve all the things beneath one robots.txt and never must handle a number of. This isn’t going that will help you rank higher or something like that.
I’m now ready to see adverts for web optimization providers to maneuver your robots.txt file to a CDN. Once more, this doesn’t provide you with any web optimization profit or rating profit, that is simply to make it simpler for you whenever you wish to keep one file. After all, in case you ever depart that CDN, then it is advisable to take care of that, however it is not a lot totally different than transferring to a brand new host.
Discussion board dialogue at LinkedIn.
