Cloudflare Built a Tool That Scores Sites on AI Readiness. Only 4% Pass.

The dominant conversation about AI and websites over the past year has been about blocking: which crawlers to stop, how to enforce it, and whether HTTP 402 payment requirements will stick. Cloudflare alone is sending over one billion 402 responses daily to AI crawlers, and its April 2026 partnership with GoDaddy was built around giving 20 million hosting customers tools to control or charge AI bots. The direction of travel has been defensive.

On April 17, 2026, Cloudflare released a tool that works in the opposite direction. Is Your Site Agent-Ready? scans any website and scores it on how well it supports AI agents, not how well it blocks them. The same day, Cloudflare published adoption data from a scan of the 200,000 most visited domains. The result: 78 percent of sites have a robots.txt file, but only 4 percent have declared AI usage preferences in it. Standards for agent authentication and machine-readable APIs appear on fewer than 15 sites in the entire dataset.

The two directions are not contradictory. A site can block training crawlers and simultaneously make itself accessible to agents that help users find and buy things. But the readiness score introduces a question that blocking tools do not: if you want agents to work with your site rather than just scrape it, is your infrastructure actually ready for that? For most websites, the answer is no. For hosting providers, that gap is a platform decision waiting to be made.

What Gets Scored

The scanner checks four categories. The first two cover the basics most sites are already close to supporting. The second two are where almost every site currently fails.

Discoverability and content accessibility are the low-hanging fixes. Discoverability checks for a valid robots.txt, a sitemap, and HTTP Link response headers (RFC 8288) that surface key resources without requiring an agent to parse HTML. Content accessibility checks whether the site responds with clean Markdown when an agent requests it via an Accept: text/markdown header. Cloudflare measured up to 80 percent token reduction on pages that support this, which translates directly to faster and cheaper agent responses.

Bot access control and capabilities are where the gap is structural. Bot access control checks whether the site’s robots.txt includes Content Signals directives, which separately control AI training, AI inference use, and search indexing, and whether it supports Web Bot Auth for cryptographic bot identity. Capabilities checks for machine-readable service descriptions: an Agent Skills index, an API Catalog (RFC 9727), OAuth discovery endpoints, an MCP Server Card, and WebMCP support. These tell an agent not just that a site exists, but what it can do and how to interact with it without reading documentation. Fewer than 15 sites in the 200,000-domain dataset pass any of these checks.

A fifth category covers agentic commerce standards including x402, the Universal Commerce Protocol, and the Agentic Commerce Protocol. These are checked but do not currently count toward the score. For each failing check, the tool generates a prompt the site owner can paste directly into a coding agent to implement the fix.

Where the Web Stands Today

Cloudflare’s scan of the 200,000 most visited domains, filtered to exclude redirects, ad servers, and tunneling services, produced the following baseline figures, now tracked weekly on Cloudflare Radar:

• robots.txt present: 78 percent of sites
• AI usage preferences declared in robots.txt (Content Signals): 4 percent
• Markdown content negotiation supported: 3.9 percent
• MCP Server Cards and API Catalogs combined: fewer than 15 sites in the entire dataset

The Cloudflare blog post introducing the tool draws the comparison to Google Lighthouse, which scores websites on performance and security and has driven broad adoption of web standards by making non-compliance visible. The intent with the agent readiness score is the same: create a measurable benchmark that pushes adoption.

Most Fixes Are a Configuration Change. That Is the Platform Decision.

Hosting providers have been here before. When Google began factoring page speed into search rankings, hosting companies that offered faster infrastructure and one-click caching configurations gained an advantage. When HTTPS became a ranking signal, managed SSL certificate provisioning became a selling point. The agent readiness score introduces a similar dynamic.

The difference this time is that the scoring criteria are not controlled by a search engine but are tied to open standards published by IETF and implemented across multiple platforms. A site that scores poorly is not penalized by one company’s algorithm; it is simply less useful to any AI agent, regardless of who built it.

The practical implications break down by customer segment.

For shared hosting customers, the gap between passing and failing most checks is a configuration question, not a platform limitation. robots.txt with Content Signals directives, a sitemap, and HTTP Link headers can be enabled at the server level or via a platform plugin. Hosting providers that offer these as defaults, rather than requiring customers to find and configure them manually, will have a competitive advantage as awareness of the score grows.

For managed WordPress and application hosting, Markdown content negotiation is the most technically involved check. It requires the server to respond with clean Markdown when an agent requests it, which depends on either a plugin or server-side logic. Providers that build this into their managed stacks before customers start asking about it will avoid reactive support requests later.

For enterprise and developer-focused hosting, the Capabilities category is the relevant benchmark. Agent Skills indexes, API Catalogs, and MCP Server Cards are how an agent discovers what a site or application can do. Customers building products that need to be accessible to AI agents will look for hosting and deployment environments that make publishing these endpoints straightforward. Providers with developer portals and documentation sites of their own should also check their own scores: Cloudflare used the tool to overhaul its own developer documentation and describes it as now “the most agent-friendly documentation site.”

The Adoption Baseline Updates Every Week

The Cloudflare Radar dataset tracking adoption across the 200,000 most visited domains updates weekly. Right now, the baseline is low enough that shipping even one of these checks as a platform default puts a hosting provider ahead of the field. That window closes as awareness grows and competitors act.

The tool itself is free, the scan takes seconds, and the fix prompts it generates are designed to be pasted directly into a coding agent for implementation. For a hosting provider, the most useful starting point is running the scanner against a sample of customer sites across different plan tiers. The results will show which checks fail consistently, which maps directly to where a platform default or a new managed feature has the most impact.