A crawl-to-refer ratio counts how many pages an AI platform requests for each referral request recorded. The ratio can identify a large imbalance, but a platform-wide figure does not establish whether a particular crawler is worth allowing on your site.
The numerator and denominator describe different activities. Page requests measure access, while referral requests record traffic with a recognized referrer. That traffic still needs to be deduplicated and connected to business outcomes.
The methodology is public
Cloudflare's launch explanation defines the ratio using HTML requests: crawler requests identified by platform-associated user agents divided by requests with a platform-associated referrer. It aggregates different crawler purposes under a platform. The post also acknowledges missing referrers from native apps, which can inflate the result by an unknown amount. [1]
For June 19–26, 2025, Cloudflare reported 70,900:1 for Anthropic. That is a dated observation with a stated collection method, not a permanent characteristic of Claude. In the same release, a 19.4% weekly fall in Google's ratio was associated with reduced Googlebot crawling.
Read the definition before using a figure from a slide or headline. Ask for the dates, included bots, population of sites, and referral rules. If those details differ, a change between two published ratios may reflect different measurements.
Watch the two counts separately
Here is a hypothetical site with a stable number of measured referrals:
| Period | Eligible crawler requests | Measured referrals | Ratio |
|---|---|---|---|
| Week one | 10,000 | 100 | 100:1 |
| Week two | 5,000 | 100 | 50:1 |
The ratio falls by half. The site has not received one extra referral. That could still be valuable if crawling was expensive, but it is an infrastructure improvement, not evidence of better acquisition.
Now suppose the site has 10,000 crawler requests in both weeks and referrals fall from 100 to ten. The ratio worsens tenfold. Before changing access, check whether traffic fell, attribution changed, or a previously identifiable app stopped sending a recognizable referrer.
For zero measured referrals, show the request count and “no observed referrals.” Do not replace the zero denominator with one to produce a tidy number. A dashboard should expose the absence of data rather than manufacture precision.
Separate crawlers by purpose
OpenAI documents distinct roles for OAI-SearchBot, GPTBot, and ChatGPT-User. OAI-SearchBot supports search; GPTBot collects material that may be used for model training; ChatGPT-User handles certain user-initiated actions. Its documentation allows search and training preferences to be set independently, and notes that user-triggered actions may not follow robots.txt in the same way. [2]
This distinction changes the operational question. If your aim is referral acquisition, start by measuring search-related access and the pages it reaches. If your concern is compute expense, inspect the expensive requests, regardless of how impressive the provider's referral total looks.
A provider-level total remains useful as a summary. Keep the underlying bot groups available so the summary does not become the only possible decision.
Build a site-level decision sheet
Use a fixed observation window and collect the following for each bot group. A four-week starting window is an operational choice, not a universal statistical minimum; seasonal sites and long sales cycles need a different view.
- Access: requests by path, status code, content type, and verification method.
- Cost: origin requests, bytes transferred, cache behavior, and any measurable incremental expense.
- Outcomes: identifiable referrals, engaged visits, qualified leads, and revenue where available.
- Coverage: which valuable pages were reached and which returned errors or challenges.
- Visibility: whether those pages appear as citations in a defined answer sample.
The verification field matters. A claimed user-agent string alone is weak evidence of identity. OpenAI publishes IP ranges in its crawler documentation; use provider-supported verification where available, and retain a separate category for requests you cannot authenticate. [2]
Compare edge and origin logs carefully. A cached request can reach the edge without reaching your application. If the concern is server load, counting every edge request as an origin hit will overstate the problem. Label which layer each count represents.
Change one thing, then inspect the result
Choose an action that addresses the observed issue. Repeated errors on an important article call for a delivery fix. Expensive requests to low-value paths call for a narrower access or caching change. A broad block is a much larger intervention and makes the result harder to interpret if several bot purposes are bundled together.
Save the old configuration and record the change time, affected paths, bot group, and expected outcome. Check both delivery and measurement afterward. A fall in bot traffic is expected after a block; it does not establish whether human discovery improved or deteriorated.
Keep the comparison window, analytics definitions, and citation sample stable. Note publication activity and major site changes. If an observation cannot distinguish fewer referrals from lost referrer information, report that uncertainty instead of calling the test a win or loss.
Record the decision at the level you can act on: the bot group, the pages it can reach, the observed cost, and the measured outcomes. Revisit it when those conditions change.