Reddit’s Bold Move: Blocking AI Web Crawlers for Free Content

TL;DR:

• Reddit blocks search engines from surfacing recent posts unless they pay.
• Google has a $60 million deal with Reddit for AI content access.
• Reddit’s move sets a precedent for data control and monetisation.

In a bold move, Reddit has started blocking major search engines and their AI web crawlers from accessing recent content – unless they pay up. This decision has significant implications for how we search for information online, especially for those who rely on Reddit for personal, human answers. Let’s dive into the details.

The End of Free Reddit Content for AI

Reddit, a popular online forum platform, has taken a strong stance against AI web crawlers. According to Reddit spokesperson Tim Rathschmidt, the company has been in discussions with multiple search engines. However, they have been unable to reach agreements with all of them, as some are unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI.

This means that if you’re using a search engine other than Google, you won’t see recent Reddit posts in your search results. Instead, you’ll be greeted with outdated content. This change is significant for users who often use the ‘site.com’ search or add ‘Reddit’ to their queries to find helpful insights and recommendations.

Google’s Special Deal with Reddit

Earlier this year, Google struck a deal with Reddit worth $60 million. This deal allows Google’s AI bots to access Reddit’s content, following a Reddit blackout last year that left Google without access to many subreddits. This partnership highlights the value of Reddit’s data and the lengths Google is willing to go to maintain access to it.

However, Reddit’s recent move is not related to this partnership. Instead, it’s a broader strategy to control how Reddit’s data is used and monetised.

Reddit’s Power Play in Data Scraping

Reddit’s decision to block AI web crawlers is a power play in the world of data scraping. The company has drawn firm lines regarding AI being ‘entitled’ to data for free. Last month, Reddit updated its robots.txt file to stop web crawlers from scraping its content, reinforcing the fact that Reddit data is not for free.

Caitlin Roulston, a spokesperson for Microsoft, acknowledged that Microsoft respects the robots.txt standard. This means that Microsoft’s AI models will not use content from pages that do not want their content to be used.

The Impact on Search Engines and Users

Reddit’s move has a significant impact on both search engines and users. For search engines like Bing, they will need to decide whether to pay for access to Reddit’s content or continue to surface outdated posts. For users, this means that the search engine they choose will significantly impact the relevance and timeliness of the Reddit posts they see.

The Future of Data Control and Monetisation

Reddit’s decision sets a precedent for data control and monetisation. It sends a clear message to AI companies and search engines that they cannot expect free access to valuable data. This move could inspire other platforms to follow suit, potentially reshaping the landscape of data scraping and AI content generation.

Comment and Share

What do you think about Reddit’s decision to block AI web crawlers? Do you think other platforms should follow suit? Share your thoughts in the comments below and subscribe for updates on AI and AGI developments.

You may also like:

• Defend Your Digital Turf: A One-Click Solution to Block AI Bots
• The Demise of the Web: How Google and AI are Redefining the Internet
• The Best AI Search Engines of 2024
• To learn more about Goggle’s collaboration with Reddit tap here.