We have now integrated 7 additional bots into our log monitoring tool. Until now, we only monitored bots related to SEO such as googlebot desktop and mobile. Why should you follow these bots? Discover how monitoring SEA and vertical search optimization bots can be helpful for your SEO strategy.
Why monitor additional bots
A global view of Crawl Budget
Your global crawl budget includes the cost of crawls from all of Google’s bots, not necessarily just the SEO bots. Monitor the changes in your crawl budget and their eventual impact on your SERP positions by keeping an eye on all of the bots that can crawl your site.
Google Search Console Data
The Search Console presents crawl frequency and crawl budget as aggregate values for all Google bots. This means that hits from all bots are added together in Search Console, and that the number of SEO bot hits that you see in Oncrawl will always be inferior to what you see in the Search Console. The number will be inferior but more accurate to precisely understand how crawl budget is spent regarding SEO visits.
With the addition of the rest of the official Google bots, the total number of bot hits will be much closer to what is indicated in Google Search Console, with the additional benefit of providing you with the crawl breakdown by bot.
There may still be differences, in part due to the increasing presence of Google bots that aren’t officially listed yet, and aren’t yet counted in Oncrawl, such as the AMP bot.
Understand if there is a relationship between Adwords and SEO crawls
Whether AdsBots and SEO bots share information gathered about your URLs is a frequently asked question in SEO. If AdsBots share information with SEO bots, we would expect Google to need to crawl your pages less frequently when they have already been crawled recently by AdsBots.
Monitoring the behavior of both types of bots allows you to determine if this sort of relationship exists between SEO bots and AdsBots.
Tracking both types of bots can provide answers to questions such as:
- Is there a positive correlation between SEO bots and AdsBots behavior? (Does their crawl frequency increase and decrease together?)
- Do increases in AdsBot hits lead to a decrease in SEO bot hits?
Use media googlebots to understand trends
Ecommerce: track googlebot-images, if image searches provide you with a significant number of visits. Understand how often your pages are crawled for images, and whether crawl rate has an impact on clicks.
Online publishers: track googlebot-image, googlebot-video, mediapartners-google to understand how frequently your news is picked up by google and more accurately predict impressions and clicks.
Monitor the interaction between your SEO and SEA policies
If you are responsible for both SEO and SEA policies, having access to SEA bot behavior is a major advantage.
Even if you are not in charge of SEA policy, monitoring SEA bots will help you coordinate SEO and SEA policies better. Knowing how both types of bots are behaving can prevent conflicting campaigns on selected URLs.
New Monitored Bots in Oncrawl
Adsense
These bots crawl pages for AdSense.
- Mediapartners-Google (mobile)
- Mediapartners-Google (desktop)
Adsbot
These bots check the quality and pertinence of pages used in paid ads campaigns.
- AdsBot-Google-Mobile
- AdsBot-Google (desktop)
- AdsBot-Google-Mobile-Apps
Media
These bots crawl your site looking for indexable media.
- Googlebot-Image
- Googlebot-Video
Note: you can track the behavior of Google’s SEO bots, Googlebot (mobile) and Googlebot (desktop), in the other reports under Logs Monitoring.
Best practices for SEA & vertical search optimization bots in Oncrawl
- You must have the log monitoring option enabled in order to add the ability track hits by SEA and vertical search optimization bots. (This information is extracted from analyses your log files.)
- It is possible to follow only a selection of the additional bots by filtering your log files before sending them to Oncrawl. Remove log file lines related to bots you do not want to follow.
- If you filter your log files before sending them to Oncrawl, you will need to adjust your filters. Leave lines by the bots you are interested in. You can base your filter on information from the official list of Google bot user-agents.
- You may need to switch to the “Resources” tab on many of the charts if only a selection of bots visit your site, or if you only monitor some bots. This is the case if your pages attract media bots, but not AdSense or AdsBot bots.
- Use the Log Monitoring: Events dataset in the Data Explorer to create reports with this information.
- If a large portion of your traffic comes from Google news, or from image or video searches, monitor media googlebots to understand and predict trends in user behavior.
For more practical uses, see how you can improve your SEO by monitoring these bots.
Get started with SEA and vertical search optimization bots
Contact Oncrawl to add SEA and vertical search optimization bots to your plan, or start your free trial!
SEA and vertical search optimization bots are an add-on to the Log Monitoring option available with + plans for Oncrawl Advanced. If you already use the Log Monitoring option, this add-on is free to activate but will have an impact on the number of lines analyzed by Oncrawl and therefore the pricing.
