Cloudflare signals fed OpenAI hundreds of millions of pages

Signals from a cohort of about 1% of Cloudflare’s network helped OpenAI find and index hundreds of millions of web pages its search crawler had never seen, Cloudflare wrote in an October 15 post on its blog. Those pages are now among the sources ChatGPT can draw on when it answers a question.
The two companies announced the research pilot on July 8. At the time, Cloudflare described its contribution as a vantage point: more than 20% of the web sits behind its network, so it sees how content changes and how traffic behaves in real time. OpenAI was to bring frontier models, a large-scale search and answer system, and real user queries to test against. Nick Ryder, OpenAI’s VP of Research, said then that the pilot would let OpenAI explore whether network-level insights could help it discover content more efficiently. The July announcement named content freshness, traffic quality and actual page changes as the kinds of signals the project would use.
What passes between the companies is metadata: traffic patterns, request characteristics, how a domain behaves. Cloudflare’s signals point OpenAI’s crawler at pages. OpenAI then evaluates and ranks those pages on its own, judging how relevant and helpful they are, and ChatGPT Search links back to the original sources. The target is the long tail of very specific pages: the manual for one exact washing machine model, the current passport requirements and fees, a local restaurant’s latest menu and opening hours. Such a page can be useful even when few other sites link to it, and a crawler that finds pages by following links can easily miss it.
Finding a URL is only the first step. Cloudflare says automated traffic can easily make a nonexistent page look interesting, and it lists the patterns it ran into. Bots repeatedly ask for files that aren’t there. Some sites redirect almost any address to their homepage. Other addresses lead to copies of the same content. Recognizing these patterns lets the system drop such addresses before a crawler ever fetches them, which saves work for the crawler and for the site being crawled.
Freshness is the other half of the problem. Fetching every page more often would be expensive for everyone, and for a site owner a crawler that keeps downloading an unchanged page adds load and brings nothing back. Change signals from Cloudflare’s network tell the crawler when a page is worth another visit and when it isn’t.
The pilot is also testing signals that flag potentially harmful pages. Cloudflare’s example is a legitimate site with one compromised page, such as a fake login form planted by an attacker. Flagging that single page instead of the whole domain keeps users away from it without penalizing the site owner. Early results suggest these signals can help pick out pages that should be kept out of an index.
Cloudflare concedes there is a trade-off. Finding new pages can increase crawling at first, while those pages are evaluated and indexed, so more discovery does not immediately mean less crawler traffic. The stated goal is to keep discovering and stay current while cutting the work involved. Cloudflare says that for its customers this should mean less unnecessary load, and for OpenAI more of its search resources spent on information that helps people.
Cloudflare places the work alongside OpenAI’s product push. With GPT-6 and Intelligent UI, OpenAI is turning ChatGPT into a place to shop, learn and find answers through more visual, interactive experiences, and the post describes the discovery pilot as a way to feed it more useful, up-to-date information from across the web.
The next phase has three parts: better selection of which pages to surface, faster and more reliable change signals, and measuring what happens after content is discovered. Cloudflare says it is continuing to work with OpenAI on how better site discovery translates into real business outcomes for the sites involved. No figure for those outcomes has been published yet.
Cloudflare also sets out what the pilot does not do. The research uses metadata about the web for search and discovery and does not share page content for model training. Site owners who want to see and manage how AI systems reach their pages are pointed to Cloudflare’s AI Crawl Control. And the post states the limit plainly: “A signal about a page doesn’t grant permission to access it.”
Up next
OpenAI blames Apple’s offboarding in trade secrets fight
OpenAI says Apple paints “routine job interviews as acts of trade secret theft.” On Wednesday a San Jose judge hears both Apple’s injunction bid and OpenAI’s motion to dismiss. When you last changed jobs, how fast was your old work access cut off?
Related stories
- Cloudflare checks AI-found code flaws against live traffic
- Cloudflare’s Rust port of TypeScript runs up to 8x faster
- Scientists tell OpenAI work takes 1.9x as long without AI
- OpenAI drops lottery for free ChatGPT access for scientists
- OpenAI opens internal models to four outside safety labs
- GPT-6 Astra co-built a candidate lattice Standard Model
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

