Cloudflare introduces the Agentic Internet. The deal that built the modern internet is quietly unraveling, and most business leaders have not noticed yet.

For decades, publishers accepted a simple trade: let search engines crawl your content, and they will send readers back to you. Those clicks funded advertising revenue, subscriptions, and sales. The equation worked because everyone got something.

Generative AI broke it. AI answer engines now extract information, summarize it, and deliver the answer without the click. The value flows in one direction. The creator gets nothing.

Cloudflare is proposing a different deal for what it calls the Agentic Internet, one built around permission, attribution, and payment. Whether you run a media company, a brand content operation, or a knowledge business, this shift will change how your content creates value.

Cloudflare Is Drawing a Line

Starting September 15, 2026, new websites on Cloudflare's platform will allow traditional search crawling by default while blocking AI training and agent use on advertising-supported pages. Existing free customers will receive the same defaults, with the option to adjust settings through their dashboards.

The distinction matters because some crawlers serve multiple purposes simultaneously. A publisher may want an article indexed in search results while objecting to that same reporting being used to train a commercial model or power an AI-generated answer. Cloudflare is proposing to treat those as separate permissions, not a single all-or-nothing choice.

The company is also evolving its Pay Per Crawl experiment into a broader Pay Per Use model, where publishers could be paid when their content appears in an AI result or when an agent purchases premium information for a specific task. Ceramic.ai and You.com are among the first partners in the program.

An Attribution Business Insights dashboard is planned to show how AI bots access content, where that content is cited, and how much human traffic different AI platforms return. Answer Engine Optimization, the practice of making content findable by AI rather than just by search engines, would become measurable for the first time.

In a conversation with Stephanie Cohen, Chief Strategy Officer, Cloudflare, she said, "The vast majority of our customers want AI to engage with their content. However, for those who rely on advertising and subscriptions, the challenge is distinct: they want to remain discoverable without being forced to give their work away for free. As an infrastructure provider, we can help bridge that gap. That’s why we are focused on delivering the tools necessary to support all types of site owners as the agentic Internet continues to evolve."

What Leaders Should Take From This

The strongest case for Cloudflare's model is not technical. It is structural. Large media companies can negotiate licensing deals directly with AI developers. Most businesses cannot. Cloudflare's infrastructure would let smaller content owners set permissions and receive compensation without negotiating individual contracts with every AI company in the market.

There is also a practical efficiency argument. Cloudflare reports that more than half of AI crawler traffic involves retrieving pages that have not changed. Signaling freshness could reduce bandwidth costs for publishers and computing expenses for AI companies, creating a shared incentive to cooperate.

The risks, though, are real.

Cloudflare could potentially control the identification of agents, the permissions layer, the usage measurement, and the payment infrastructure. Publishers might gain leverage over AI companies while simultaneously increasing their dependence on a single intermediary.

Attribution also remains an unsolved problem. An AI-generated answer may combine dozens of sources, paraphrase an original idea, or use reporting without displaying a citation. How the value of each source gets calculated, who audits the results, and how disagreements get resolved are still open questions.

Paid access could also entrench existing advantages. Well-funded AI platforms can absorb licensing fees and technical integrations. Startups, researchers, and open-source developers may face a more expensive web.

The Market Is Already Moving

Cloudflare is not alone in sensing the opportunity, and that itself is the signal leaders should not miss. According to Adexchanger , the numbers explain why everyone is moving at once. Publishers have been losing 20, 30, even 90 percent of their traffic and revenue over the past year. Anthropic crawls 11,122 pages for every single referral it sends back. AI chatbot referrals drive roughly 96 percent less traffic than traditional search, and users click cited sources only about one percent of the time. When the math looks like that, a market forms fast.

TollBit , ProRata, and Microsoft have all moved into the content licensing space, each with a different theory of who controls the terms. Really Simple Licensing is pushing for an open standard before any single company locks in the rules. Per Press Gazette , nearly 70 percent of publishers expect AI licensing deals to generate at least some revenue over the next three years, but most see it as a minor source today. The window to shape those terms before they calcify is now, not indefinitely.

Leaders who hand this question to the IT team are misreading what it is. This is not a crawl policy.

It is a revenue model decision.

How AI systems access, use, and compensate for content will determine where value accumulates in the next phase of the digital economy, and the companies shaping those terms right now are not waiting for everyone else to catch up.

The Question Every Leader Needs to Answer

Who gets paid when an AI answer replaces a visit to your content?

The web cannot keep producing valuable reporting, research, and expertise without a sustainable economic exchange. Permissions, attribution, and compensation are moving from nice-to-have to foundational requirements of the Agentic Internet.

But the outcome depends on transparency. Publishers need to understand how usage is measured, how prices are set, and what role intermediaries play. AI companies need standards that do not lock them into a single infrastructure provider. And business leaders need to decide now whether they are building content strategies for a world that no longer exists.

So what do you do as a business leader?

  1. Audit your content as an asset, not a cost. Map what you publish, where AI systems are accessing it, and whether you have any permissions or compensation in place. Most leaders have no idea what their content is worth to the AI economy. Find out before someone else decides for you.
  2. Pick a side on visibility versus protection. Blocking AI access protects your work but reduces your presence as audiences shift to AI-driven research and recommendations. Allowing access builds visibility but may not generate revenue yet. This is a strategic call, not a technical default. Make it deliberately.
  3. Get ahead of the standards race. The companies setting the terms right now, Cloudflare, TollBit, Microsoft, ProRata, are building the infrastructure everyone else will inherit. Assign someone on your team to track which standard gains adoption and position your organization to participate early, before path dependencies lock in around you.

AI still depends on the open web. The challenge is building an economic model that rewards the people who create its value, without concentrating access to information in fewer hands than ever. Cloudflare is all in.