Cloudflare signals fed OpenAI hundreds of millions of pages

The pages Cloudflare and OpenAI want to find are the dull ones: the manual for one exact washing machine model, the current passport fees, a local restaurant's latest menu and opening hours. Cloudflare says a research pilot drawing on about 1% of its network has now put hundreds of millions of previously unknown pages like these into OpenAI's search index.

The early results appeared on Cloudflare's blog. The pilot itself started in July 2026. According to Cloudflare's press release, it was announced on July 8 as a way to test whether insights from participating websites could help AI search engines discover and index content, with the stated focus on accurate and timely answers.

At a glance

• A participating cohort of about 1% of Cloudflare's network helped OpenAI discover and index hundreds of millions of new web pages.
• Cloudflare supplies signals about which pages exist and when they change, and OpenAI evaluates and ranks those pages on its own. Cloudflare says the work uses metadata about the web and does not share page content for model training.
• Cloudflare admits that crawling can rise at first while new pages are evaluated and indexed. Measuring what happens to sites after their pages are discovered is left for the next phase.

Cloudflare's signals tell the crawler what exists and when it has changed

A crawler that discovers the web by following links will rarely reach pages that few other sites link to, and the appliance manual is exactly that kind of page. Cloudflare sits in front of a large share of sites and sees requests to them directly. The press release puts that share at more than 20% of the web, and says OpenAI brings frontier models, a large search and answer system and real user queries to test against. According to the announcement, the signals were meant to include content freshness, traffic quality and actual page changes. Automater describes the feed as traffic patterns, request characteristics and domain behavior passed to OpenAI's search crawler.

Raw traffic is noisy, and most of what Cloudflare reports learning is about what to throw away. Bots keep requesting files that don't exist, which can make a missing page look popular. Some sites redirect almost any address to their homepage. Other addresses turn out to be copies of the same content. Spotting these patterns before the crawler fetches anything saves work for OpenAI and spares the site a request it gains nothing from.

The pilot also tests flags for potentially harmful pages. Cloudflare's example is a legitimate site with one compromised page, such as a fake login form planted by an attacker: the flag lands on that page and the rest of the domain is left alone. Early results suggest such signals can identify pages that should be kept out of an index.

The last piece is timing. Re-fetching every page more often to stay current would be expensive, and for a site owner a crawler downloading an unchanged page is pure load. Change signals tell the crawler when a page deserves another visit. Cloudflare also says this work complements GPT-6 and Intelligent UI, which OpenAI is using to make ChatGPT more visual and interactive for shopping, learning and finding answers.

Cloudflare now controls both who gets in and what gets found

Our reading is that the pilot is less a change of direction than the second half of a single position. Cloudflare appears to be making itself the layer that decides both whether an AI crawler may enter a site and which of its pages are worth that crawler's time.

The first half was built over the past year. As TNW recounts it, Cloudflare let sites block AI crawlers unless AI firms paid publishers, built a browser protocol to tell humans from bots, and ran under the slogan "your content, your rules". The new blog post still ends by pointing owners to AI Crawl Control, Cloudflare's tool for seeing and managing how AI systems access a site, and says plainly that a signal about a page doesn't grant permission to access it.

The discovery half is the new part, and the October post frames it for publishers as an opportunity. Cloudflare's pitch is that people increasingly ask AI for help, so owners who opt in should want those systems to find their work, represent it accurately and send readers to it. For OpenAI, the post says, the gain is spending more of its search resources on useful pages. Both pitches rely on the same network view.

The strongest counter-argument comes from the pilot's own limits

TNW itself calls this a notable turn, with Cloudflare now on both sides of the line, and there is a fair case that "gatekeeper" overstates it. The pilot covers a cohort of about 1% of Cloudflare's network. It is about discovery, and TNW points out that it is separate from the licensing deals OpenAI has signed to feed content into ChatGPT. Ranking stays with OpenAI. In July, according to Enterprise DNA, the companies called it a research project and gave no timeline for results or for how findings might reach production. The October post still uses the word pilot.

A metadata feed that grants no access and controls no ranking is a modest thing. On that reading, Cloudflare is simply sharing what it already sees, and the access controls are untouched.

The results leave out the number site owners need

The figure in the post measures what OpenAI gained: hundreds of millions of indexed pages. Nothing in it says how many of those pages were cited in ChatGPT answers, how much traffic came back to their owners, or how much extra crawling the cohort's sites absorbed during evaluation. Cloudflare says the next phase will improve page selection, test faster and more reliable change signals, and measure what happens after content is discovered. It also says it is working with OpenAI on how better discovery translates into business outcomes.

Our view would be wrong if the next results show the discovery signals working independently of the access tools, available on the same terms whatever a site has chosen in AI Crawl Control. If instead being findable starts to depend on those settings, the two halves are one product. Site owners in the cohort: has your crawl volume from OpenAI gone up or down since July?