Cloudflare Launches Tool to Blacklist AI Bots

Cloudflare has unveiled a free tool to prevent AI bots from scraping websites hosted on its platform for data to train AI models. As Cloudflare points out, some AI vendors, including Google, OpenAI, and Apple, allow website owners to block data scraping bots by editing robots.txt, but not all AI scrapers respect this.

“Customers don’t want AI bots visiting their websites, especially those that do so dishonestly,” the company declared on its official blog. “We fear that some AI companies intent on circumventing rules to access content will persistently adapt to evade bot detection.”

To address the problem, Cloudflare analyzed AI bot and crawler traffic to fine-tune automatic bot detection models. The models assess if an AI bot is trying to evade detection by mimicking human web browser behavior.

“When bad actors attempt to crawl websites at scale, they generally use tools and frameworks that we are able to fingerprint,” the blog mentions. “Based on these signals, our models are able to appropriately flag traffic from evasive AI bots as bots.”

Cloudflare has set up a form for hosts to report suspected AI bots and crawlers and commits that it will continue to manually blacklist them over time.