On July 1, 2024, Reddit up to date its robots.txt file to ban most search engine and AI crawlers from accessing its content material – together with blocking Microsoft Bing. Reddit didn’t block Google Search, which we covered here a few weeks ago.
A Microsoft spokesperson advised me last night, “We respect the robots.txt commonplace. Bing stopped crawling Reddit after they applied their up to date robots.txt file on July 1, which prohibits all crawling of their web site.”
Reddit told The Verge, “Now we have been in discussions with a number of serps. Now we have been unable to achieve agreements with all of them, since some are unable or unwilling to make enforceable guarantees concerning their use of Reddit content material, together with their use for AI.”
Mark Williams-Prepare dinner confirmed me that Reddit’s new content was not arising in Bing Search, he posted about this on X after which a number of different media outlets covered it earlier than Microsoft despatched me this assertion.
Reddit additionally stated, “This isn’t in any respect associated to our latest partnership with Google.” Sure, Google is paying Reddit $60 million annually to have the ability to entry content material on its web site.
Do not forget that Google has hidden gems algorithm that helped the Reddit acquire loads of visitors from Google. Reddit is seeing loads of bad PR round Reddit dominating Google Search. I imply, Reddit is dominating Google’s search outcomes. Heck, Google even examined special treatment for Reddit not too way back.
In any occasion, Bing and lots of different search engine and AI bots usually are not permitted to crawl Reddit’s web site anymore. Google can and does however different engines don’t.
Daring transfer Reddit – however with the Google licensing deal – I suppose it is not too daring of a transfer.
Wild: Reddit now blocks all main serps *besides* Google from crawling content material
This may solely enhance Google’s monopoly energy…
… and you’ll guess regulators can be https://t.co/9uGDivBdYc
— Cyrus search engine optimisation (@CyrusShepard) July 25, 2024
Timeline:
– Google boosts Reddit in search by over 400%
– Google broadcasts a 60M take care of Reddit proper after
– Reddit broadcasts it seeks to go public 48-72 hours after
– Reddit blocks all scraping for competitors serps shortly after👏Anti 👏Aggressive👏Conduct https://t.co/Hzrw0zkjDo
— Ori Zilbershtein (@OriZilbershtein) July 24, 2024
That is a part of a disconcerting acceleration of monopolies increasing to additional block competitors and take from customers. Serps use to supply hyperlinks to related content material. Now they spam customers with advertisements intermixed with scraped and content material laundered by AI with out attribution. https://t.co/xMvzir3DPA
— Tim Sweeney (@TimSweeneyEpic) July 24, 2024
Discussion board dialogue at X.
