TL;DR:
- Cloudflare’s new one-click feature blocks AI bots, including those that scrape content dishonestly.
- Bytespider, Amazonbot, ClaudeBot, and GPTBot are the top AI crawlers in terms of request volume.
- Only 2.98% of the top one million Internet properties protected by Cloudflare block or challenge AI bot requests.
The Rise of AI Bots and the Need for Protection
In the ever-evolving digital landscape, artificial intelligence (AI) and artificial general intelligence (AGI) have taken centre stage. As the demand for content to train AI models soars, content creators face a new challenge: AI bots scraping their original work. To combat this, Cloudflare has launched a one-click solution to block AI bots, available to all customers, including those on the free tier.
AI Bot Activity: A Closer Look
Analyzing the most popular AI bots on Cloudflare’s network, Bytespider, Amazonbot, ClaudeBot, and GPTBot lead the pack in terms of request volume. Bytespider, operated by ByteDance, the Chinese company behind TikTok, is reportedly used to gather training data for its large language models. Amazonbot and ClaudeBot follow closely, with the former used to index content for Alexa’s question-answering and the latter to train the Claude chatbot.
The Response from Website Operators
Despite AI bots accessing around 39% of the top one million Internet properties using Cloudflare, only 2.98% of these properties take measures to block or challenge these requests. Higher-ranked Internet properties are more likely to be targeted by AI bots and, consequently, more likely to block such requests.
Unmasking AI Bots: How Cloudflare Does It
AI bot operators often disguise themselves as real browsers by using a spoofed user agent. However, Cloudflare’s global machine learning model can identify this activity as bot-generated, even when operators lie about their user agent. This ensures that customers stay protected from the newest waves of bot activity.
Reporting Misbehaving AI Crawlers
If you encounter an AI bot that’s not behaving, you can report it. Enterprise Bot Management customers can submit a False Negative Feedback Loop report via Bot Analytics. Alternatively, any Cloudflare customer can submit reports of an AI bot scraping their website without permission using a dedicated reporting tool.
Join the Fight for a Safer Internet
Cloudflare is committed to helping content creators thrive and maintain control over how their content is used. Visit 1.1.1.1 from any device to get started with their free app that makes your Internet faster and safer.
Have you encountered AI bots scraping your content? How do you protect your digital turf? Share your experiences below and don’t forget to subscribe for updates on AI and AGI developments.
You may also like:
