Monitor search engine activity on your site for Google, Bing, Yandex and Baidu

May 5, 2021 - 3  min reading time - by Rebecca Berbel
Home > Product updates > Oncrawl now supports Bing, Yandex and Baidu bots!

We’re extending our log monitoring metrics to include SEO traffic and bot hits from Bing, Yandex, and Baidu, in addition to Google.

We’ve seen plenty of ways that log monitoring for Google can provide you an advantage in SEO, whether it’s making sure your migration has gone smoothly, checking non-sampled and complete data on visits coming from Google results pages, or examining the distribution of your site’s crawl budget in order to optimize how it’s spent.

But if log monitoring is so important, why stop with Google?

With Oncrawl, you don’t have to. If your users are not all Google users, a significant part of your SEO strategy can require understanding not only how Google sees your site, but how other search engines do as well. Expanding SEO log analysis to the search engines that matter in your market can make all the difference:

  • Bing: North American markets
  • Yandex: Eastern European markets, Russia, or Turkey
  • Baidu: Asia and Pacific markets, particularly China

Examine bot traffic on your site — from all of the bots that matter

SEO log monitoring focuses on different elements–and different search engine bots–depending on the site, the vertical, and the market in question.

With our new support for additional search engines, you not only have a view of activity on your site related to Google, Baidu, Yandex and Bing. We also provide detail for individual bots for each of these search engines, particularly the difference in mobile and desktop user agents for a single search engine.

Furthermore, if your site relies on efficient image or video indexing, search engine news aggregation, or even paid search, we’ve got you covered.

Log monitoring metrics for additional search engines

All of our log monitoring metrics are available for all of the supported search engines. They can be viewed in the report dashboards, or queried, filtered, sorted and downloaded in Oncrawl’s Data Explorer.

Live monitoring

If you need up-to-the-minute monitoring, such as during a migration or during algorithm updates that seem to be affecting your site, our live monitoring is available for all four search engines.

Crawl behavior

Get an overview of search engine activity for any or all of the supported search engines, during whatever time period you’re interested in.

This covers information about the total bot hits and pages explored by search engines, as well as newly crawled pages, and crawl frequency. It also includes the breakdown between mobile and desktop versions of search engine bots.

This information is also available for page groups on your site. Switch easily between segmentations to see how different ways of grouping your pages make patterns in crawl behavior stand out.

SEO impact

Visitors from each search engine leave their mark in your server logs. With Oncrawl, for Google, Bing, Baidu and Yandex, you can track a variety of metrics, including:

  • The number of pages that receive organic visits
  • The number hits that represent organic visits
  • How long it takes between the time when a search engine first visits your pages, and when you earn your first organic traffic from that search engine on those pages.
  • How organic visits are distributed across the page groups in your segmentations.

Exploration sanity

Understand how healthy your site looks to each search engine by examining the status codes returned to bots, the size of pages seen by bots, response times to bot requests, and the number and size of non-HTML resources requested by bots.

SEA and vertical bots

Search engines also use bots in addition to the bots that scan URLs to potentially be indexed. They also use bots to validate pages for ads, to index images and videos, or to populate new search.

These bots have different crawling patterns, and might not be pertinent to all sites. However, in some verticals, they are key to visibility. Whether you’re looking for Bing News, Yandex ads, or Baidu images, Oncrawl has the answers.

How to access data from Bing, Baidu or Yandex in addition to Google

To add monitoring for the bots you are interested in, you’ll need to use Oncrawl’s SEO Log Analyzer. While monitoring for Google’s bots is included by default, you’ll need to reach out to us using the chat function in the interface to add additional search engines for a one-time fee.

Once monitoring for additional search engines is activated:

  • Remember to include log lines related to these search engines in your log files if you provide filtered files.
  • Find the additional bot information in the Log Monitoring reports, using the “Bot” drop-down menu at the top of each log monitoring screen.

If you’re looking for more information, check out the help for this feature.


Rebecca Berbel See all their articles
Rebecca is the Product Marketing Manager at Oncrawl. Fascinated by NLP and machine models of language in particular, and by systems and how they work in general, Rebecca is never at a loss for technical SEO subjects to get excited about. She believes in evangelizing tech and using data to understand website performance on search engines. She regularly writes articles for the Oncrawl blog.
Related subjects: