Cloudflare and GoDaddy Make AI Crawlers Pay Their Way

On April 7, 2026, Cloudflare and GoDaddy announced a partnership that gives GoDaddy’s more than 20 million customers direct access to Cloudflare’s AI Crawl Control, a tool that lets website owners choose whether AI bots can access their content for free, be blocked entirely, or be required to pay per request. Cloudflare reports that websites already behind its network are sending more than one billion HTTP 402 responses daily to AI crawlers. The partnership extends that enforcement capability to GoDaddy’s hosting base without requiring those customers to independently sign up for Cloudflare. Alongside the product integration, both companies are jointly advancing two open standards designed to give AI agent traffic a verifiable identity layer: GoDaddy’s Agent Name Service and Cloudflare’s Web Bot Auth.

AI Crawl Control: What It Does

AI Crawl Control reached general availability from an earlier beta, previously called AI Audit, before the GoDaddy partnership was announced. It gives website owners visibility into which AI bots are crawling their content and three enforcement options for each: allow access at no charge, block access entirely, or require payment. The payment mechanism uses the HTTP 402 status code, which was defined in 1991 as “Payment Required” but went essentially unused until AI content licensing created a practical application for it. A 402 response can include contact information for licensing discussions or a link to an API with defined pricing, leaving the commercial terms to the publisher rather than imposing a platform-level rate.

The one billion daily 402 responses figure indicates the tool is in active use at scale, not just adopted by a niche of technically sophisticated publishers. For hosting providers, the GoDaddy integration changes the distribution model: AI Crawl Control on Cloudflare’s own platform requires a paid Cloudflare account. The GoDaddy partnership embeds the capability into the hosting account itself, reaching customers who have no direct Cloudflare relationship. The specific pricing terms of the GoDaddy integration were not disclosed at launch.

Agent Name Service: The Directory Layer

GoDaddy’s Agent Name Service was announced in October 2025. It functions as a DNS-like registry for AI agents, providing consistent naming, verification, and discovery for agents across different systems. The architecture is built on infrastructure GoDaddy already operates at scale: DNS, public and private certificate authorities issuing TLS and identity certificates, DNSSEC securing the trust chain, TLSA records for certificate pinning, and Merkle tree-based transparency logs creating immutable audit trails.

Each agent enrolled in ANS receives a name following a structured format: Protocol://AgentID.Capability.Provider.vX.Y.Z.Extension. Any change to any component of that name, including a minor version increment, forces a new unique ANSName and triggers immediate certificate revocation, creating a permanent record of agent version history. The specification is designed to be protocol-agnostic, with adapters translating agent records into common framework formats to avoid vendor lock-in. ANS is available on GitHub and aligns with IETF draft standards. GoDaddy CTO of Product and AI Travis Muhlestein described the current AI agent environment as “a lot like the ‘Wild West’ of the early internet” when the project launched in October.

Web Bot Auth: The Cryptographic Identity Layer

Cloudflare introduced Web Bot Auth in May 2025 as a cryptographic framework addressing a fundamental limitation of traditional bot verification: user-agent strings and IP allowlists are trivially spoofed and change unpredictably, providing no reliable basis for enforcement or payment. Web Bot Auth uses two mechanisms. HTTP Message Signatures, defined in RFC 9421, have agents cryptographically sign outgoing HTTP requests using Ed25519 key pairs, with each signature including a validity window and a key identifier pointing to the agent’s publicly available public keys. Request mTLS provides a second mechanism using mutual TLS certificate authentication via the TLS Flags extension currently under IETF review.

Web Bot Auth is the authentication layer that makes Pay Per Crawl fraud-resistant: a crawler that claims to be a paying subscriber must present a valid cryptographic signature, not just a user-agent header that any bot can copy. Initial signed agent partners using the framework include the ChatGPT agent, Goose from Block, Browserbase, and Anchor Browser.

How the Three Technologies Work Together

ANS, Web Bot Auth, and AI Crawl Control address three different layers of the same problem. Web Bot Auth provides the identity verification layer: a crawler signs its requests with a key pair, proving it is the agent it claims to be. ANS provides the registry layer: it is the directory where those public keys and agent metadata are stored and discoverable, functioning as the lookup infrastructure for verifying a signature. AI Crawl Control provides the enforcement layer: once a crawler’s identity is verified through Web Bot Auth and ANS, the website owner decides what happens to that specific verified agent, allow, block, or charge.

Cloudflare CSO Stephanie Cohen framed the system as infrastructure for a new internet business model: “By putting tools like AI Crawl Control and open standards into the hands of website owners, we are providing essential underpinnings for a new Internet business model.” GoDaddy CSO Jared Sine cited the ANS identity layer as the prerequisite for that model: “We are working with the broader industry to ensure the agentic open web does too.” The practical question for hosting providers is whether the open standard approach gains enough crawler adoption to make the payment mechanism effective. Cloudflare’s own network covers approximately 20 percent of websites globally. GoDaddy serves over 20 million customers. Together, a crawler that wants to avoid 402 blocks at scale needs to participate in the identity and payment system the partnership is building.

What This Costs and Who Pays

The basic AI Crawl Control dashboard, which shows which AI bots are accessing a site and lets owners block them, is available to paid Cloudflare customers. The Pay Per Crawl feature, which adds the actual payment collection via HTTP 402, is currently in private beta. Cloudflare acts as the Merchant of Record: it records billing events each time a paying crawler makes an authenticated request, aggregates those events, charges the crawler, and distributes earnings to the publisher. The specific commission Cloudflare retains from those transactions has not been disclosed publicly. Publishers set their own per-request price for their site. No standard rate is defined by the platform.

For GoDaddy customers, the pricing of the integrated AI Crawl Control was not disclosed at the April 7 launch. GoDaddy has not stated whether the feature will be included in existing hosting plans, sold as an add-on, or priced separately. Until those terms are published, GoDaddy customers cannot calculate what, if anything, the protection will cost them.

What This Means for a Hosting Provider

For a hosting company, this partnership introduces a structural question that goes beyond a product feature: who captures the value when AI companies use the content that hosting customers publish?

Until now, AI crawlers have treated hosted websites the same way search engine crawlers did in the 1990s, treating access as free by default and leaving content owners with no lever other than robots.txt, which crawlers can ignore. Cloudflare’s one billion daily 402 responses show that enforcement at scale is already happening. The GoDaddy partnership attempts to extend that enforcement to a much larger population of smaller publishers who would not otherwise have the technical infrastructure to implement it.

The business case for a hosting provider offering these controls is straightforward. If a customer’s content generates crawler revenue, that revenue flows through the platform the customer is already paying for. The hosting provider does not need to monetize the crawler traffic directly. The value is in retention: a hosting platform that offers AI crawler controls is harder to leave than one that does not, particularly as the volume of AI-driven traffic grows and publishers become more aware of it.

The open question is whether major AI companies will participate in the payment system at all. As of April 2026, no large AI company has made a public commitment to support HTTP 402 pay-per-crawl. The ChatGPT-User crawler, which OpenAI updated in December 2025, removed prior language stating it complies with robots.txt, creating uncertainty about whether OpenAI’s crawlers will honor 402 responses. Anthropic has publicly stated that all three of its crawlers, ClaudeBot, Claude-User, and Claude-SearchBot, honor robots.txt. Google added its Google-Agent crawler to its official list in March 2026 and uses the Web Bot Auth protocol for identity, but has made no public statement on paying for access. The authentication infrastructure is being built. The commercial terms that would make it a revenue source for publishers are still being negotiated outside of public view.

For a hosting provider evaluating this partnership as a competitive move, the relevant fact is that Cloudflare’s network sits in front of roughly 20 percent of websites globally, and GoDaddy’s customer base adds 20 million more. A crawler that wants uninterrupted access to web content at scale eventually has to reckon with that coverage. That is the leverage the partnership is designed to create. Whether AI companies respond by paying, by finding alternative content sources, or by challenging the model legally is a question that will determine whether Pay Per Crawl becomes a real revenue line for publishers or remains a technical capability with limited commercial adoption.