BUSINESS TECH | Cloudflare expands AI content controls with new publisher tools

Cloudflare on July 2 unveiled new tools that allow website owners to control how artificial intelligence systems access their content while introducing analytics and payment features aimed at helping publishers monetize AI use.

Announced on July 2, the updates include new AI crawler classifications, analytics dashboards and commercial partnerships intended to give publishers greater visibility into AI traffic and more options for licensing or restricting content used by AI models.

The announcement comes as AI services increasingly replace traditional search engines as a primary way people discover online information. Cloudflare said automated bots and AI agents now account for more than half of all web requests, prompting publishers to seek greater control over how their content is indexed, cited and used for AI training.

Beginning Sept. 15, Cloudflare plans to introduce new default settings for new customers and newly added websites that allow traditional search indexing while blocking AI training and AI agent access on pages supported by advertising. Existing free-tier customers that do not change their settings before the deadline will also receive the new defaults, although site owners can modify the settings at any time through the Cloudflare dashboard.

The company said the policy is designed to encourage AI developers to separate search, AI agent and AI training crawlers instead of using mixed-purpose bots, allowing publishers to choose which activities they permit.

Cloudflare also introduced Attribution Business Insights, a dashboard that shows how AI companies crawl websites and how much referral traffic they generate. The tool is intended to help publishers negotiate licensing agreements by providing data on AI usage and user referrals.

In addition, the company is promoting what it calls Answer Engine Optimization, or AEO, which focuses on improving how websites are cited and referenced in AI-generated responses rather than in conventional search engine rankings.

Cloudflare said it is also testing technology that tells AI companies when web pages have changed, reducing repeated requests for unchanged content. According to the company, more than half of AI crawler traffic consists of repeated fetching of pages that have not been updated, consuming bandwidth without improving AI-generated answers.

The company is also expanding its Pay Per Crawl initiative into a Pay Per Use model that compensates publishers when their content contributes to AI-generated answers instead of simply when it is crawled. Cloudflare said it is working with Ceramic.ai and You.com to implement the system, with additional AI companies expected to support different payment models.

The latest announcements build on Cloudflare’s AI Crawl Control and Web Bot Auth technologies introduced over the past year. The company said those technologies, together with its new analytics and payment tools, are intended to support a more transparent marketplace for AI content licensing and online publishing.

————————————————————————-
TEN YEARS OF TECHSABADO!

PLEASE LIKE our FACEBOOK PAGE and SUBSCRIBE to OUR YOUTUBE & SPOTIFY channels.

WATCH TECHSABADO ON OUR YOUTUBE CHANNEL: