Live log monitoring for SEO website events

February 24, 2021 - 5  min reading time - by Rebecca Berbel
Accueil > Product updates > Live log monitoring for SEO website events

We’ve just released a new SEO log monitoring report that will change the way you think about how you use log analysis in SEO.

There are plenty of cases when up-to-the-minute data is mission-critical in SEO, particularly about how Google and organic traffic react to changes on your website.

If you’ve ever done a website migration, a roll-out of major SEO changes, or a website redesign, you know how important it is to know when something goes wrong. Live monitoring can help confirm that everything’s going according to plan—and it can let you know that something’s not right, as soon as the first signs of danger appear.

If you’ve been hit by the effects of an algorithm update, you’ve seen how changes in Google’s crawl pattern can help you catch changes just before they happen, if you can see them fast enough.

And if you’re a news publishing, tracking hour-by-hour activity on new content can bring key insights.

This is exactly what we’ve made sure you can do. With live log monitoring for SEO, see exactly what search engine bots visit and where organic visitors land on your site, as it happens.

Follow Googlebot behavior as it happens

Live log monitoring is the only up-to-the-minute source of information for what Googlebot is doing on your website: although crawl statistics are increasingly available in the Google Search Console, they’re not always up-to-date, and never up-to-the-minute. For major website events that impact your SEO, this is the level of detail you want.

You can track events hour-by-hour over a rolling multi-day window, or zoom in on the last few hours to see if crawling patterns on your site have changed.

When viewing Googlebot behavior for your site as a whole is misleading, you can concentrate on individual page groups to view bot behavior on different sections of your website, or even switch between different types of segmentations in order to view your site from a different angle. This can be a quick way to detect when whole site sections are orphaned, or when a particular page template presents an issue.

Whether you’re just counting bot hits, or looking at statistics like the HTTP status codes for requested URLs, the information is there.

Zoom in on specific periods

Our live analysis period is deliberately long enough to allow you to look at different periods of an extended event, like a staged roll-out, or even a multi-day algorithm update. This allows you to focus on the periods when change happens, even if they were yesterday.

You can view the whole live monitoring period, or zoom in on specific sections.

Filter by user agent

Google uses many bots, which may respond differently to changes, both on your site and in the world of Search. Filter your live log monitoring by user agent to focus on different types of changes and their impact.

Altered crawling behaviors in some bots may indicate changes initiated by Google, such as changes in news policies in your region, or a new method of indexing, as was the case with the move from desktop to mobile-first (or mobile-only) indexing. Examining live behavior in news bots, or comparing sudden differences in mobile vs desktop bots can give you a precise look at when changes come into effect and how they affect your site.

Modified behavior in other bots can show how a change is used by different Google services. Inserting monetization on all of your site pages will likely cause changes in how your pages are crawled and ranked for organic search, but also in how Google AdsBots crawl your pages. This helps you attribute peaks in crawl behavior to the right bot and the right cause.

See changes in organic traffic when they occur

Live log monitoring also gives you instant information on SEO as an acquisition channel. Getting attribution data for site visits from official sources is slow and sometimes unreliable: Google Analytics aggregate data per day and often doesn’t count all visitors; Google Search Console can frequently take up to three days to show data. Live log monitoring shows data as soon as possible and depends only on how often you provide log data.

See how on-site changes and internal linking affect organic traffic to landing pages across your whole site, or focus on changes in a specific section. Like with crawl data, site-wide data for organic traffic can be misleading, particularly if organic traffic is not acquired evenly across all pages of your site.

Remove dependency on IT departments

If you are interested in log monitoring, or if you are still manually providing log files, live log monitoring is a great excuse to get the ball rolling. Because live monitoring depends on a fluid stream of data–whether per-event, on an hourly basis, or at a different interval that’s right for your infrastructure–, automating your SEO log monitoring is a necessity.

Once set up, you don’t have to depend on the IT department to upload your log files, or to share them with you so that you can do it. And if something in the automated process breaks, not only do we let you know, but a glance at your live log monitoring dashboard will show the absence of data instantly, not days later.

How to get access to live log monitoring with Oncrawl

Live log monitoring is included with all of our SEO log analyzer plans. If your plan includes log monitoring, you already have access to it.

To start using live monitoring right away, or get a feel for how it works, head over to the new dashboard in your log monitoring analysis report. From a project home page with log monitoring enabled, click on “Show log monitoring”, then “Live logs” in the left-hand sidebar.

Remember, you’ll need to have provided log files within the live monitoring window for any information to show up here.

Interested? Check out how you can use live monitoring in your SEO process.

Rebecca is the Product Marketing Manager at Oncrawl. Fascinated by NLP and machine models of language in particular, and by systems and how they work in general, Rebecca is never at a loss for technical SEO subjects to get excited about. She believes in evangelizing tech and using data to understand website performance on search engines. She regularly writes articles for the Oncrawl blog.
Related subjects: