Skip to content

OpenAI

Cloudflare signals fed OpenAI hundreds of millions of pages

4 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: OpenAI models and products →
Image: blog.cloudflare.com

Signals from a cohort of about 1% of Cloudflare’s network helped OpenAI find and index hundreds of millions of web pages its search crawler had never seen, Cloudflare wrote in an October 15 post on its blog. Those pages are now among the sources ChatGPT can draw on when it answers a question.

The two companies announced the research pilot on July 8. At the time, Cloudflare described its contribution as a vantage point: more than 20% of the web sits behind its network, so it sees how content changes and how traffic behaves in real time. OpenAI was to bring frontier models, a large-scale search and answer system, and real user queries to test against. Nick Ryder, OpenAI’s VP of Research, said then that the pilot would let OpenAI explore whether network-level insights could help it discover content more efficiently. The July announcement named content freshness, traffic quality and actual page changes as the kinds of signals the project would use.

What passes between the companies is metadata: traffic patterns, request characteristics, how a domain behaves. Cloudflare’s signals point OpenAI’s crawler at pages. OpenAI then evaluates and ranks those pages on its own, judging how relevant and helpful they are, and ChatGPT Search links back to the original sources. The target is the long tail of very specific pages: the manual for one exact washing machine model, the current passport requirements and fees, a local restaurant’s latest menu and opening hours. Such a page can be useful even when few other sites link to it, and a crawler that finds pages by following links can easily miss it.

Finding a URL is only the first step. Cloudflare says automated traffic can easily make a nonexistent page look interesting, and it lists the patterns it ran into. Bots repeatedly ask for files that aren’t there. Some sites redirect almost any address to their homepage. Other addresses lead to copies of the same content. Recognizing these patterns lets the system drop such addresses before a crawler ever fetches them, which saves work for the crawler and for the site being crawled.

Freshness is the other half of the problem. Fetching every page more often would be expensive for everyone, and for a site owner a crawler that keeps downloading an unchanged page adds load and brings nothing back. Change signals from Cloudflare’s network tell the crawler when a page is worth another visit and when it isn’t.

The pilot is also testing signals that flag potentially harmful pages. Cloudflare’s example is a legitimate site with one compromised page, such as a fake login form planted by an attacker. Flagging that single page instead of the whole domain keeps users away from it without penalizing the site owner. Early results suggest these signals can help pick out pages that should be kept out of an index.

Cloudflare concedes there is a trade-off. Finding new pages can increase crawling at first, while those pages are evaluated and indexed, so more discovery does not immediately mean less crawler traffic. The stated goal is to keep discovering and stay current while cutting the work involved. Cloudflare says that for its customers this should mean less unnecessary load, and for OpenAI more of its search resources spent on information that helps people.

Cloudflare places the work alongside OpenAI’s product push. With GPT-6 and Intelligent UI, OpenAI is turning ChatGPT into a place to shop, learn and find answers through more visual, interactive experiences, and the post describes the discovery pilot as a way to feed it more useful, up-to-date information from across the web.

The next phase has three parts: better selection of which pages to surface, faster and more reliable change signals, and measuring what happens after content is discovered. Cloudflare says it is continuing to work with OpenAI on how better site discovery translates into real business outcomes for the sites involved. No figure for those outcomes has been published yet.

Cloudflare also sets out what the pilot does not do. The research uses metadata about the web for search and discovery and does not share page content for model training. Site owners who want to see and manage how AI systems reach their pages are pointed to Cloudflare’s AI Crawl Control. And the post states the limit plainly: “A signal about a page doesn’t grant permission to access it.”

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.