Online infrastructure company Cloudflare is to begin applying a feature to block data-scraping tools from AI firms by default for new customers, potentially giving publishers more control over how their content is used by companies such as Google and OpenAI.

Cloudflare said it was also developing a “pay per crawl” model that would allow publishers to ask AI companies for payment to access their content.

The move could have a significant effect on how AI firms access the human-generated information needed to train their models, given that about 16 percent of global internet traffic passes through Cloudflare’s network, as the company estimated in 2023.

‘Pay per crawl’

Cloudflare chief executive Matthew Prince said the move was about putting the “power back in the hands of creators, while still helping AI companies innovate”.

Publishers have generally welcomed web crawlers from search engines indexing their content, as this results in traffic coming to them.

AI crawlers don’t follow this pattern, however, as they use the content to generate responses without directing users to the original source.

Prince said Cloudflare was trying to create a “new model that works for everyone”.

A number of publishers and content creators have sued AI firms including OpenAI and Anthropic over using their content without permission to build their tools.

Reddit last month sued Anthropic for alleged breach of contract and “unlawful and unfair business acts” over the use of material from its users.

Copyright questions

AI companies have said their use of online material falls under fair use.

That assertion was upheld last month in a ruling by a US federal judge, who found that Anthropic’s use of books to train its AI models followed fair use rules and was “transformative”.

But the case, which was brought by authors, is only one of many in the wide-ranging debate around AI and copyright.

Roger Lynch, chief executive of publisher Condé Nast, said Cloudflare’s offering was a “critical step toward creating a fair value exchange” on the internet that supports creators while holding AI companies accountable.

Cloudflare previously trialled a system for blocking AI crawlers in September.