The policy shift targets crawlers that blend search functionality with AI training and agent capabilities. By default, Cloudflare will restrict these bots unless site owners specifically adjust their settings. This change applies to all new accounts, new sites, and existing free-tier users. CEO Matthew Prince cited the rapid rise of non-human web traffic—which recently surpassed human activity—as the primary driver for this intervention, emphasizing the need for a sustainable digital ecosystem.
Cloudflare is simultaneously expanding its monetization tools for content creators. The existing "Pay Per Crawl" marketplace is evolving into a "Pay Per Use" model, allowing publishers to charge AI developers when content generates value rather than just for initial access. Early partnerships with Ceramic.ai and You.com demonstrate this framework, where publishers receive payments when their material appears in AI-driven search results or premium access requests. The company noted that over 50% of current AI crawl traffic is wasted on re-fetching unchanged pages, suggesting these new standards could also improve bandwidth efficiency for both publishers and developers.
Comments (0)
No comments yet. Be the first!