In the present day being July 4th, I usually would keep away from writing tales right here however there are quite a lot of rumors on the market that Reddit is obstructing Google Search from crawling and indexing its content material. It isn’t true – Reddit shouldn’t be blocking Google Search.
I imply, at first look, it does seem like Reddit’s robots.txt file is obstructing all search engines like google, together with Google Search:
However that’s simply what you and I see.
When you entry it from the Google Rich Results test, as if you’re actually accessing it from Google’s IP ranges, you see Google is certainly allowed:
Plus, simply attempt to see if that device can entry the Reddit house web page – it could and is allowed:
I imply, to assume that Reddit is obstructing Google is form of wild. For Reddit to really do that’s much more insane. Reddit is getting paid by Google for that content material and Reddit will get an insane quantity of site visitors from Google Search.
Good investigation work Pedro Dias on X and Ryan Siddle over here.
That is what they’re displaying to Google:
# Our robots.txt is for search engines like google
# 80legs
Consumer-agent: 008
Disallow: /# 80legs’ new crawler
Consumer-agent: voltron
Disallow: /Consumer-Agent: bender
Disallow: /my_shiny_metal_assConsumer-Agent: Gort
Disallow: /earthConsumer-agent: MJ12bot…
— Pedro Dias (@pedrodias) July 4, 2024
So you’ll be able to all calm down, Reddit will nonetheless rank above you for all of your content material in Google Search…
Discussion board dialogue at X.
