AI search has a lot of people asking the same questions: How do you track it? Which prompts work for your site or brand? What’s the impact of query fan-out? The questions seem endless, and the answers often contradict each other. But when you filter out all the noise, you can find reliable answers. You just need to know which approaches actually deliver trustworthy data.
I’ve become really interested in understanding how AI search works and looking at the actual data we have that can help us answer these questions. In this series, my goal is to make AI search measurement as transparent as possible.
In my previous article, “AI bots explained: What powers platforms like ChatGPT?” I concluded that,
“Reliable AI search data remains one of the biggest challenges facing SEO professionals today. Log analysis offers a solution, providing the most valuable field data currently available for understanding AI platform behavior.”
This brings us to the next question: How can we effectively measure success in AI search?
In this article, we will explore how to turn AI user bot visits into reliable AI search metrics and why this approach is most reliable to date.
The AI search data challenge
Although AI referral traffic is already decreasing and barely brings any clicks, that doesn’t mean it’s useless to invest in and optimize for AI search.
The user journey is changing. ChatGPT, Perplexity, Claude, and other AI platforms are answering questions directly, often citing sources instead of sending users to traditional search results. But the journey doesn’t end there. A potential client or customer might discover your brand in ChatGPT and will often follow up with a search for your products in Google. You need to be visible in both.
This means visibility in AI search is your objective, not referral traffic.
The challenge, however, is that LLM platforms such as ChatGPT don’t currently provide search metrics (unless you have a partnership) like Google does. It’s a black box.
In SEO, before your website generates any clicks, you can track impressions and rankings to see if you’re making progress. Even more importantly, you can target keywords based on search volume.
None of that is available for AI search, you only have clicks. But there’s a not so secret weapon I’ve mentioned before: AI user bot traffic in your server logs!
I strongly believe log analysis is the only reliable way to currently measure visibility performance in LLMs.
[Ebook] Crawling & Log Files: Use cases & experience based tips
Understanding the difference between lab data and field data
If AI search is a data black box, the first step is understanding what you can actually measure.
In fact, SEOs have access to two types of data points: lab data and field data.
Lab data comes from prompt tracking tools that test specific queries to see if and how your brand appears in AI responses. This approach is useful for benchmarking against competitors and monitoring brand mentions across platforms. However, because it relies on simulated prompts rather than real user queries, it can’t tell you about actual traffic volume or ROI.
Field data, on the other hand, comes from analyzing your server logs to track real AI bot behavior. This shows you which pages AI platforms are actually crawling, citing, and sending traffic to. It’s the difference between testing what might happen and measuring what is happening.
Below is a table that outlines the differences:
| Lab Data | Field Data | |
|---|---|---|
| Definition | Synthetic data from simulations of AI/LLM usage | Real data from actual AI/LLM usage |
| Metrics |
|
|
| How it works |
|
|
| Useful for |
|
|
| Pros |
|
|
| Cons |
|
|
| Best for | Competitive positioning and brand monitoring | Performance tracking and technical optimization |
When prompt tracking makes sense
Prompt tracking tools are useful when you need to track owned media and PR objectives. If you need to benchmark your brand’s visibility compared to competitors, or monitor how your brand is being mentioned and cited across AI platforms, prompt tracking provides valuable insights.
It’s particularly useful for tracking sentiment and understanding which sources AI platforms associate with your brand.
When log analysis makes sense
As I explained in my previous article, AI user bot visits can be used as an AI search visibility metric: a proxy for citations (or impressions).
1 user bot visit = 1 potential citation
Every bot hit from ChatGPT-User on your website represents an attempt to read your content for a citation. It doesn’t guarantee the source was actually used in the answer to a user prompt, but it confirms the page was listed in the sources. ChatGPT search selected this page among other search results as a trusted source likely to contain the information needed to answer the prompt.
Limitations to account for
Not all bot hits guarantee that the page will generate a citation. Several factors can prevent this from happening:
- Redirects – Bot hits on redirect pages don’t result in citations
- Status code errors – Server errors (500) or client errors (404) block access
- Missing content – Pages with thin or no content can’t be cited
- JS-heavy pages -Content behind JavaScript may not be readable to bots
Additionally, LLM chatbots use a caching system, which means they don’t systematically run a crawl for each user query. When a page is in cache, the LLM doesn’t need to send an AI user bot to read the content it would like to include as a citation. Therefore, part of the visibility cannot be measured using logs.
However, current data and tests show that AI user bot activity has increased since ChatGPT-5’s release. The caching system appears to have more of an impact in the context of a user conversation rather than as a global caching system. Log data is still a reliable way to measure AI visibility despite this blind spot.
One other caveat to keep in mind is that AI user bots can crawl multiple pages of your website for one “prompt session.” This is a marginal case, but it could artificially bloat the volume of citations for brand-related prompts.
Oncrawl’s approach
Oncrawl has just released the new AI Search Lens, designed to measure and analyze how LLMs interact with your website, based on real crawl and log data. To avoid some of the above-mentioned limitations, by default we filter on status code 200 and 304 to only keep AI user bot visits that have a chance to generate a citation.
To complement that approach, we have segmentation and custom OQL (Oncrawl Query Language) filtering available in the app for our users to add their own filtering rules on top of that:
- Word count filtering
- Page type to exclude because of low quality or heavy reliance on JS
Server logs represent the most reliable approach for measuring AI search visibility available today. While they don’t capture 100% of citations, they provide consistent, actionable data that allows you to track onsite optimization success with visibility volume progression.
The full picture
Lab data and field data aren’t competing approaches, they’re complementary. Prompt tracking helps you understand your competitive landscape and brand positioning, while log analysis, through Oncrawl, shows you how to improve technical performance and drive measurable results. Together, they give you a complete view of your AI search optimization journey.
However, keep in mind that AI bot analysis isn’t only for AI visibility success measurement, it also helps find actionable insights.
In the next article, I will dig into the steps you can take to combine logs and crawl data with Google Search Console data.
