Cloudflare’s Crawl Control improves business discovery in ChatGPT, aims to optimize outcomes
The research uses network signals to improve search discovery and reduce unnecessary crawling, while leaving access decisions with website owners through tools including Cloudflare’s AI Crawl Control.
The initiative addresses a challenge for AI search: useful information is not always found on prominent pages or websites with extensive incoming links.
NEW DELHI: A research pilot between Cloudflare and OpenAI has helped OpenAI discover and index hundreds of millions of previously unknown web pages, expanding the sources available to ChatGPT Search, according to an update shared by the companies.
Launched in July 2026, the pilot used a participating cohort representing approximately 1% of Cloudflare’s total network. It examined whether signals from Cloudflare’s infrastructure could help OpenAI’s search crawler identify useful content more efficiently.
The companies said the work focuses on search and discovery using metadata about the web, rather than sharing page content for model training. Website owners retain control over whether AI systems can access their content.
Improving discovery beyond heavily linked pages
The initiative addresses a challenge for AI search: useful information is not always found on prominent pages or websites with extensive incoming links.
Product manuals, first-hand accounts, current government requirements and local business information can answer specific questions even when relatively few other sites link to them. Making those pages discoverable gives AI systems a broader set of sources from which to find relevant information.
Cloudflare provides network signals that can help OpenAI’s search crawler locate potentially useful pages. OpenAI then independently evaluates and ranks those pages according to their relevance and usefulness.
The update said the pilot had brought previously unknown pages into OpenAI’s search index, giving ChatGPT additional sources for answering questions. ChatGPT Search links to relevant sources, allowing users to read the original material and discover the publishers behind it.
The companies did not disclose figures for referral traffic, publisher revenue or other commercial outcomes arising from the pilot.
Filtering nonexistent and duplicate pages
A substantial part of the research concerns distinguishing useful pages from addresses that create unnecessary work for crawlers.
Automated traffic can make a nonexistent page appear significant when bots repeatedly request files that are not available. Some websites redirect almost any requested address to their homepage, while multiple addresses may lead to copies of the same content.
Recognising these patterns before fetching a page can help crawlers avoid wasted requests. For website operators, that could reduce server load associated with repeated visits that provide little discovery benefit.
The companies are also testing signals that identify potentially harmful pages. One example is a legitimate website containing an individual page compromised by an attacker and used to host a fake login form.
The approach seeks to flag the affected page rather than penalise the entire domain. Early findings suggest these signals can help identify pages that should remain outside a search index, according to the update.
Tracking changes without repeated downloads
The pilot also explores how crawlers can keep information current without repeatedly downloading unchanged pages.
Change signals are intended to indicate when a page merits another visit and when a further request is unlikely to add value. Combined with filtering for nonexistent pages and redirects, this could allow crawling resources to focus on new or updated information.
The companies acknowledged that discovering previously unknown pages can initially increase crawling while those pages are assessed and indexed. The longer-term objective is to expand coverage and maintain freshness while reducing unnecessary fetching.
For Cloudflare customers, the intended benefit is lower avoidable load on their websites. For OpenAI, the aim is to devote more search resources to information that helps users.
Website owners retain access controls
The update emphasised that identifying a page does not create permission to access it. A discovery signal and a website owner’s decision to allow crawling remain separate.
Cloudflare’s AI Crawl Control allows domain owners to understand and manage how AI systems access their websites. The companies positioned those controls as an important part of participation, allowing publishers and businesses to make their own access decisions.
They also distinguished the pilot’s search-discovery purpose from model training, stating that the research uses web metadata rather than sharing page content for training.
That distinction is relevant to publishers assessing how their material becomes available through AI search and how they wish automated systems to interact with their sites.
Measuring business outcomes in the next phase
The next stage will focus on improving page selection, testing faster and more reliable change signals, and measuring what happens after content is discovered.
Cloudflare and OpenAI said they would examine how improved discovery translates into business outcomes. The update did not establish whether increased indexing had yet produced sustained gains in visits, customer enquiries or sales.
The announcement also referred to GPT-6 and Intelligent UI as part of OpenAI’s work on more visual and interactive experiences for shopping, learning and finding answers. The discovery pilot is intended to complement those developments by broadening access to useful, current web information.
For publishers and website operators, the research points to an evolving measure of online visibility: whether AI systems can locate relevant material, represent it accurately and connect users with its original source.
