Reddit has up to date its robots.txt file, stopping Bing and plenty of different search engines like google and yahoo from crawling the location. “Bing stopped crawling Reddit after they applied their up to date robots.txt file on July 1, which prohibits all crawling of their website,” a Microsoft consultant informed Search Engine Land.
What occurred. On July 1, 2024, Reddit up to date its robots.txt file stopping many search engines like google and yahoo and AI instruments from crawling the location. Reddit did not prevent Google from crawling the location, regardless of what some folks could have thought earlier this month. However Reddit did stop most different crawlers from crawling the location.
Earlier this morning, Mark Williams-Cook dinner notified me that Reddit outcomes have been dropping out of the Bing Search index. Then several media outlets started overlaying the information. I wished to verify that Bing’s crawlers have been certainly blocked, as a result of Reddit was utilizing IP detection to point out search engines like google and yahoo one model of its robots.txt file and people one other model of the robots.txt file – as I explained earlier this month.
Bing has thus stopped crawling new content material on Reddit, which is why if you filter Reddit leads to Bing Seek for the final week, you see nothing:
Microsoft confirmed. A Microsoft spokesperson informed Search Engine Land:
“We respect the robots.txt normal. Bing stopped crawling Reddit after they applied their up to date robots.txt file on July 1, which prohibits all crawling of their website.”
Reddit assertion. Reddit spokesperson Tim Rathschmidt says in an announcement to The Verge:
“This isn’t in any respect associated to our latest partnership with Google. We have now been in discussions with a number of search engines like google and yahoo. We have now been unable to achieve agreements with all of them, since some are unable or unwilling to make enforceable guarantees concerning their use of Reddit content material, together with their use for AI.”
Why we care. With Reddit securing a licensing deal with Google, Reddit is ready to play hardball with different search engines like google and yahoo and AI instruments. So Reddit has blocked most different search engines like google and yahoo from crawling its content material. In the meantime, Google is driving insane traffic to Reddit nowadays, together with testing special treatment in its search outcomes for them.
It makes you surprise if different giant web sites can attempt to go down this route and the place that may depart smaller publishers and content material producers.
In the meantime, don’t anticipate to see a lot new Reddit content material being surfaced on Bing within the close to future.
