AI crawlers are consuming your pages, but are they returning measurable value? Microsoft Clarity's AI scrape-to-referral ratio gives analytics and SEO teams a direct way to compare crawler activity with visits from AI-powered experiences.
The metric is especially timely as Cloudflare's September 15, 2026 policy update approaches. For new domains, Cloudflare will default to blocking bots classified as Training or Agent on pages that display ads, while Search bots will remain allowed. Rather than allowing or denying every bot on instinct, you can evaluate server demand, referral quality, citations, and business outcomes together.
To evaluate an AI crawler:
- Open the Scrape-to-Referral Card in Clarity.
- Compare each crawler's scrapes with its referrals.
- Inspect referral sessions and cited content.
- Choose whether to allow, monitor, or block the crawler.
Prerequisites: Connect the Data Clarity Needs
You need a live Clarity project, its tracking code installed, and enough activity to produce a meaningful sample. Sites with modest traffic should aim for a few weeks of data before drawing conclusions. Administrative access to robots.txt or Cloudflare is also required for implementation.
The client-side Clarity snippet cannot observe crawler requests because bots often skip analytics JavaScript entirely. The feature therefore requires server-side log data from a supported CDN or server integration. Confirm that connection before treating a blank card as zero bot activity. Microsoft Clarity's bot analytics provides useful setup context.
Step 1: Find and Decode the AI Scrape-to-Referral Ratio
Sign in, select the relevant project, and open AI Visibility from the left navigation. Enter Bot Analytics and locate the scrape-to-referral card. Microsoft added the card on August 13, 2026, according to its Scrape-to-Referral insights announcement.
Read its components as follows:
- Scrapes: pages requested by an identified AI crawler in the connected server logs.
- Referrals: tracked user sessions arriving through a click from an AI-powered experience.
- Ratio: scrapes divided by referrals. A bot making 12,000 page requests and producing 20 visits has a 600:1 ratio.
- Zero referrals: the ratio is effectively infinite for the selected period.
The Clarity AI Visibility Dashboard must reconcile server requests with client-side visits, so compare identical dates and domains. Search Engine Land's 2026 worked example showed roughly 6,000 scrapes per referral, useful for illustrating scale, not as a universal benchmark.
Step 2: Analyze AI Crawler ROI, Not Just the Ratio
A lower ratio can indicate more referral return per scrape, but the number alone does not establish AI crawler ROI. The fuller picture requires conversions, engaged time, revenue, assisted conversions, bandwidth, origin load, content sensitivity, and citation exposure. AI bot traffic value is the value of resulting visits and visibility minus technical and commercial costs.
| Crawler signal in Clarity | What it may indicate | Recommended action |
|---|---|---|
| Low scrape-to-referral ratio + valuable sessions | Crawler access is producing measurable discovery value | Allow and monitor |
| High ratio + meaningful citations | Click return is limited, but visibility may still have value | Monitor |
| High ratio + weak referrals + weak citation value | Heavy consumption with limited measurable return | Review access |
| Zero referrals but meaningful citations | Visibility exists without direct traffic | Evaluate citation value before blocking |
| Extreme scraping + no referrals/citations + material infrastructure cost | Little measurable return from crawler access | Consider restricting or blocking |
| Base each action on your own Clarity, citation, session-quality, and infrastructure data. |
Microsoft has published no official definition of a good ratio, and regional baselines shift. Digiday reported in 2026 that TollBit measured European publishers worsening from 150:1 in Q1 to 227:1 in Q2, reinforcing the need for publisher-specific evaluation.
Step 3: Validate Referrals with Recordings and Citations
Open recordings and filter by the referring AI source or sessions surfaced through the card. A session recordings AI referral review should answer three questions: Did the visitor land on the expected page, engage with the answer, and complete a meaningful next action?
Validate the quantitative ratio with:
- Behavior: scroll depth, dead clicks, navigation paths, and conversion events.
- Content: pages receiving AI crawler referral traffic versus pages being heavily scraped.
- Visibility: Clarity's AI Citations report shows which URLs and passages appear in answers.
- Attribution: verify mapped domain coverage across subdomains, regional domains, and canonical hosts.
Tip: A crawler can create strategic value without an immediate click if it repeatedly cites authoritative content. Record that visibility separately rather than assigning it zero value.
Step 4: Make and Implement the AI Crawler Access Decision
Apply a three-state policy:
- Allow: low ratio, useful citations, or high-quality referral sessions.
- Monitor: high ratio with some value, uncertain identity, or insufficient data.
- Block with caution: extreme or infinite ratio, no valuable visibility, material server cost, or unacceptable content exposure.
To block AI crawlers in robots.txt, use the crawler's documented token: User-agent: ExampleAIBot followed by Disallow: / on the next line. Replace the placeholder and test the file. Robots.txt is advisory, so a noncompliant bot can ignore it. Follow the full process for how to block AI crawlers using robots.txt.
Cloudflare AI crawler blocking provides stronger edge enforcement and simpler category-level controls. Robots.txt offers transparent, bot-specific instructions; Cloudflare can stop requests before they reach the origin entirely. Review the exact bots covered by a category before enabling a broad rule.
Common Mistakes and Troubleshooting
- No card appears: confirm server-log integration, permissions, supported infrastructure, rollout status, and date range.
- Scrapes show zero: check whether CDN caching or an incomplete log connection hides requests.
- Referrals look low: inspect referrer loss, consent settings, cross-domain tracking, and unmapped hosts.
- One week drives policy: use several representative periods and account for launches or viral content.
- Every AI bot gets one rule: distinguish training crawlers, search crawlers, user-triggered agents, and referral-producing bots.
Summary and Next Steps
The AI scrape-to-referral ratio turns disconnected bot logs and analytics referrals into a practical operating metric. Connect server data, inspect the card by crawler, validate visits through recordings and citations, then document an allow, monitor, or block decision with a review date.
Treat access control as an ongoing SEO, AEO, and GEO process. Improve the return side through LLM referral traffic optimization. Vizup's Organic Autopilot for modern discovery helps brands monitor, create, optimise, publish, and learn across Search, Social, Communities, AI Answer Engines, and Local Discovery using AI agents, human experts, and live SEO, pSEO, AEO, and GEO tools. Paid ads are available as an amplification add-on.
Frequently Asked Questions
What is a good AI scrape-to-referral ratio in Microsoft Clarity?
Microsoft provides no official benchmark. Lower is generally more efficient, but the ratio only makes sense alongside conversion quality, citation value, server cost, and your site's specific goals.
Why don't I see the AI Visibility Dashboard in my Clarity account?
Check project permissions, feature availability, the selected date range, and whether supported server-side logs are connected. The standard tracking snippet alone cannot detect crawler requests.
Will blocking AI crawlers in robots.txt hurt my regular SEO?
Not if you target only the intended AI bot token. Blocking regular search crawlers such as Googlebot can damage crawling and indexing, so validate every user-agent before publishing the file.
How is the scrape-to-referral ratio different from bot traffic in server logs?
Logs show requests and resource use. The ratio connects those requests with tracked human referrals, making it possible to assess return on crawler access rather than crawl volume alone.
How often should I review my AI crawler access decisions?
Monthly during periods of rapid change, and at least quarterly afterward. Reassess after crawler-policy updates, infrastructure migrations, unusual traffic shifts, or new AI referral sources appear.
