New Delhi: Cloudflare is about to change how AI bots interact with the web. From July 2, 2025, any new site signing up with the company will automatically block AI crawlers by default. The decision follows mounting criticism from publishers and creators who say their content is being mined by AI systems without consent or payment.
The company, which handles around 16 percent of global internet traffic, is also rolling out a new system called “Pay Per Crawl”. It lets publishers charge AI companies for every piece of content accessed by their bots. That means AI firms will now need to either pay up or stay out.
AI bots will be blocked unless told otherwise
Cloudflare is taking a clear stand: AI bots will be blocked from crawling unless the website owner gives them permission. Earlier, the system only allowed a one-click block. Now, the default setting for new domains is to deny AI scrapers altogether.
“This will fundamentally change how AI companies access web content going forward,” the company said. Many AI firms rely on scraping the internet to train large language models. That model may no longer work the same way.
For publishers, this change offers tighter control. They can now choose which bots to allow, which to charge, and which to block completely. This new control comes with another layer—Cloudflare will verify the identity of bots to make sure they are who they claim to be.
Introducing ‘Pay Per Crawl’: charge for access
The company has introduced a new feature called “Pay Per Crawl,” which lets content owners put a price on their web pages. If an AI crawler tries to access a site that has this paywall, it will receive a signal asking for payment. Only if the crawler agrees to pay the set price will the content be served.
Cloudflare says it will act as the middleman in this process. Publishers can set a price per request and decide if they want to allow, block, or charge a particular bot. The model uses HTTP response code 402, an old internet standard that was rarely used before but is now being revived.
“Instead of a blanket block or uncompensated open access, we want to empower content owners to monetise their work at Internet scale,” Cloudflare said.
Media Industry support is growing
Several media companies and tech firms have already supported the move. These include TIME, BuzzFeed, Stack Overflow, The Atlantic, and Quora. The idea is to offer publishers a third option: block, allow, or charge for access.
The shift comes at a time when publishers and AI firms are facing off in court over copyright and content usage. OpenAI and others have been criticised for scraping data from news sites, blogs, and forums without permission.
Cloudflare’s move could change the power dynamic. By making payment a requirement, the balance may tip in favour of those who create the content, not just those who consume it through machines.
“What if I could charge a crawler?” Cloudflare asked in its official blog, hinting at a future where bots are treated like paying users. That future, it seems, has just begun.