Cloudflare’s new policy pushes AI companies to pay for publishers’ content

Cloudflare has given artificial intelligence companies until September 15, 2026, to draw a clear line between the web crawlers that power conventional search results and those that feed AI model training or autonomous agents, warning that any crawler blending the two roles will be shut out of many publisher websites by default.

The company detailed the new policy this week, saying that once the deadline passes, its default configuration will automatically block so-called "mixed-use" crawlers — bots that simultaneously support search indexing, AI agent browsing, and model training — from reaching any web page that carries advertising. Site owners will keep the ability to override this default and allow such crawlers if they choose to.

The restriction is not limited to a narrow slice of Cloudflare's customer base. The company says the new default will apply to anyone signing up as a new customer, to any new website added by an existing customer, and across its entire population of free-tier sites — meaning a large share of the web infrastructure Cloudflare protects will shift toward blocking undifferentiated crawlers unless publishers actively opt back in.

Cloudflare has positioned itself as something of a gatekeeper in the increasingly strained relationship between AI developers and online publishers, previously rolling out tools that let site owners detect, block, or set pricing terms for bots scraping their content. This latest change targets a specific gap in that approach: crawlers that AI companies run under a single identity for multiple purposes, which has made it difficult for publishers to allow search indexing — which sends them traffic — while withholding free access for model training or agent tasks that offer little benefit in return.

By forcing a technical split between crawler types, the policy narrows the paths AI companies have for collecting training data and letting their agents browse the open web without some form of arrangement with content owners. Because a substantial share of internet traffic already flows through Cloudflare's network, the shift could meaningfully affect how model developers source new material and how AI agents pull information from ad-supported sites once the cutoff takes effect.

Publishers who have argued that AI firms harvest journalism and other content without payment or credit are likely to welcome the change, since it gives them a default advantage instead of requiring them to configure blocking rules on their own. AI companies, by contrast, may need to rebuild their crawling infrastructure into separate, clearly labeled bots for each purpose, and some are likely to view the requirement as a new obstacle to the broad web access their products have depended on.

How strictly the separation will be enforced, and how quickly major AI developers adjust their systems before September, remains unclear, as does whether some companies could find ways around the new default. Even so, Cloudflare's move adds to a wider industry trend toward compensating publishers for AI's use of their material, paralleling other licensing and pay-per-crawl efforts already being pursued by content owners seeking a cut of the value their work generates for AI products.

Source: TechCrunch