- Use cases
- Customer Success
- LOG IN
- Start free trial
We have now integrated 7 additional bots into our log monitoring tool. Until now, we only monitored bots related to SEO such as googlebot desktop and mobile. Why should you follow these bots? Discover how monitoring SEA and vertical search optimization bots can be helpful for your SEO strategy.
Your global crawl budget includes the cost of crawls from all of Google’s bots, not necessarily just the SEO bots. Monitor the changes in your crawl budget and their eventual impact on your SERP positions by keeping an eye on all of the bots that can crawl your site.
The Search Console presents crawl frequency and crawl budget as aggregate values for all Google bots. This means that hits from all bots are added together in Search Console, and that the number of SEO bot hits that you see in OnCrawl will always be inferior to what you see in the Search Console. The number will be inferior but more accurate to precisely understand how crawl budget is spent regarding SEO visits.
With the addition of the rest of the official Google bots, the total number of bot hits will be much closer to what is indicated in Google Search Console, with the additional benefit of providing you with the crawl breakdown by bot.
There may still be differences, in part due to the increasing presence of Google bots that aren’t officially listed yet, and aren’t yet counted in OnCrawl, such as the AMP bot.
Whether AdsBots and SEO bots share information gathered about your URLs is a frequently asked question in SEO. If AdsBots share information with SEO bots, we would expect Google to need to crawl your pages less frequently when they have already been crawled recently by AdsBots.
Monitoring the behavior of both types of bots allows you to determine if this sort of relationship exists between SEO bots and AdsBots.
Tracking both types of bots can provide answers to questions such as:
Ecommerce: track googlebot-images, if image searches provide you with a significant number of visits. Understand how often your pages are crawled for images, and whether crawl rate has an impact on clicks.
Online publishers: track googlebot-image, googlebot-video, mediapartners-google to understand how frequently your news is picked up by google and more accurately predict impressions and clicks.
If you are responsible for both SEO and SEA policies, having access to SEA bot behavior is a major advantage.
Even if you are not in charge of SEA policy, monitoring SEA bots will help you coordinate SEO and SEA policies better. Knowing how both types of bots are behaving can prevent conflicting campaigns on selected URLs.
These bots crawl pages for AdSense.
These bots check the quality and pertinence of pages used in paid ads campaigns.
These bots crawl your site looking for indexable media.
Note: you can track the behavior of Google’s SEO bots, Googlebot (mobile) and Googlebot (desktop), in the other reports under Logs Monitoring.
For more practical uses, see how you can improve your SEO by monitoring these bots.
Contact OnCrawl to add SEA and vertical search optimization bots to your plan, or start your free trial!
SEA and vertical search optimization bots are an add-on to the Log Monitoring option available with + plans for OnCrawl Advanced. If you already use the Log Monitoring option, this add-on is free to activate but will have an impact on the number of lines analyzed by OnCrawl and therefore the pricing.