Reddit blocks search engines from surfacing recent posts unless they pay.,Google has a $60 million deal with Reddit for AI content access.,Reddit's move sets a precedent for data control and monetisation.
In a bold move, Reddit has started blocking major search engines and their AI web crawlers from accessing recent content – unless they pay up. This decision has significant implications for how we search for information online, especially for those who rely on Reddit for personal, human answers. Let's dive into the details.
The End of Free Reddit Content for AI
Reddit, a popular online forum platform, has taken a strong stance against AI web crawlers. According to Reddit spokesperson Tim Rathschmidt, the company has been in discussions with multiple search engines. However, they have been unable to reach agreements with all of them, as some are unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI.
This means that if you're using a search engine other than Google, you won't see recent Reddit posts in your search results. Instead, you'll be greeted with outdated content. This change is significant for users who often use the 'site.com' search or add 'Reddit' to their queries to find helpful insights and recommendations.
Google's Special Deal with Reddit
Earlier this year, Google struck a deal with Reddit worth $60 million. This deal allows Google's AI bots to access Reddit's content, following a Reddit blackout last year that left Google without access to many subreddits. This partnership highlights the value of Reddit's data and the lengths Google is willing to go to maintain access to it.
Enjoying this? Get more in your inbox.
Weekly AI news & insights from Asia.
However, Reddit's recent move is not related to this partnership. Instead, it's a broader strategy to control how Reddit's data is used and monetised.
Reddit's Power Play in Data Scraping
Reddit's decision to block AI web crawlers is a power play in the world of data scraping. The company has drawn firm lines regarding AI being 'entitled' to data for free. Last month, Reddit updated its robots.txt file to stop web crawlers from scraping its content, reinforcing the fact that Reddit data is not for free. This comes at a time when many companies are grappling with the value of data in the age of AI.
Caitlin Roulston, a spokesperson for Microsoft, acknowledged that Microsoft respects the robots.txt standard. This means that Microsoft's AI models will not use content from pages that do not want their content to be used. For more on how AI models are trained, you might be interested in this primer on large language models.
The Impact on Search Engines and Users
Reddit's move has a significant impact on both search engines and users. For search engines like Bing, they will need to decide whether to pay for access to Reddit's content or continue to surface outdated posts. For users, this means that the search engine they choose will significantly impact the relevance and timeliness of the Reddit posts they see. This could also affect how users interact with AI-powered search features, such as Google AI Overviews.
The Future of Data Control and Monetisation
Reddit's decision sets a precedent for data control and monetisation. It sends a clear message to AI companies and search engines that they cannot expect free access to valuable data. This move could inspire other platforms to follow suit, potentially reshaping the landscape of data scraping and AI content generation. This trend aligns with discussions around AI's Secret Revolution and shifts in data ownership.
Comment and Share
What do you think about Reddit's decision to block AI web crawlers? Do you think other platforms should follow suit? Share your thoughts in the comments below and Subscribe to our newsletter for updates on AI and AGI developments.
Latest Comments (2)
Interesting to revisit this. It makes you wonder, will this move actually push more people to use their official app for browsing? I’m thinking about how we typically access information here in the Philippines, often through mobile.
Huh, interesting to see this playing out now. While Reddit's right to protect their ‘intellectual property’ is clear, I wonder if this will genuinely deter the larger tech players. They're pretty resourceful, right? Just feels like a game of cat and mouse rather than a definitive roadblock.
Leave a Comment