One of the difficulties with website migrations is how much is in play. You want to maintain or increase your traffic and rankings, but those are signals that you can only measure significantly after the migration has been completed.
If this is a source of migration stress for you, live log monitoring will be a game changer. With Oncrawl, monitoring server log events as soon as they occur can help you determine within minutes whether your migration is going as planned, or whether you need to fix a few things quickly–or even if you need to roll it back and get to work again.
Here’s a quick tour of a few basic ways in which live log monitoring can make your life easier if you’re working on a website migration.
Rolling out your migration and setting up monitoring
Prepare your migration
Before live log monitoring can play it’s part, you’ll need to have completed the actual migration work. This might include the following steps:
- Following a migration checklist
- Creating a staging site to plan your changes
- Comparing the old, live site to the staging site using a crawl over crawl
- Correcting errors found in the crawl over crawl
- Mapping old URLs to new ones and implementing your redirect plan
If you’re using Oncrawl to ensure your migration goes smoothly, we’ve also provided a set of custom report templates to help you out at each of the steps along the way.
Set up a segmentation for new vs old URLs
It will be extremely useful to be able to separate old and new URLs. The easiest way to do this is to create a segmentation to split your site’s URLs into two categories: URLs on the old version of your site that don’t exist on the new one, and new URLs that didn’t exist on the old version.
If you want, you can also subdivide these groups to go into further detail.
Make sure that log monitoring events are available in Oncrawl
You’ll need to provide server log data at least once a day, and preferably much more frequently during the migration period, if you want a look at the live aspect.
The more frequent the updates are, the closer to “real-time” the data you’ll be looking at will be. Once per hour during the day of the migration will give you plenty of information.
To check how frequently Oncrawl receives information about your log files, click on “Log manager tools”:
Monitoring the success of a migration: immediate feedback
Once the migration has gone live, there are immediate changes you can expect to see. These will be visible in your server logs, through Oncrawl’s live log report. Instead of waiting for data to be professed, aggregated and made available–something which, in Google Search Console, can take up to a few days–you have the critical information within minutes.
The live log report is included in all of Oncrawl’s log monitoring plans. Navigate to the report in the left hand sidebar. You’ll want to set the date range and scale, and switch to the migration segmentation you just created.
Status codes of old URLs
When the migration is rolled out, your old URLs will stop responding with a 200 status code. At the exact time of the switch to the new site, you should see a cut-off point. This is impossible to see in daily aggregate values, but extremely evident in minute-by-minute monitoring.
Using live log monitoring, you can see within a few minutes whether or not this is the case. If there are still pages responding with 200 or if there’s a spike in 404s, you know immediately that there’s a problem. It’s possible that you forgot to redirect a part of the site.
This can then be fixed before a large number of 404 responses affects the indexing of your pages.
Status codes of new URLs
Conversely, your new URLs will start responding with 200 status codes as soon as they are available. If your redirect plan is working correctly, this should occur rapidly, as old URLs are redirected to their new equivalents.
404 and 500-level errors are a sign that something has gone wrong. They should be dealt with immediately. Again, the ability to find and fix this type of error so quickly is one of the major advantages of live log monitoring during website migrations.
Traffic on old URLs
Depending on the size and authority of your site, you should start, within the next few hours or next few days, to see a decline in Googlebot hits as Google understands that the old URLs have been removed in favor of the new ones.
At this point, organic traffic should also decline, since the old URLs should gradually disappear from Google’s index.
Traffic on new URLs
You’ll be able to see the opposite occurring on new URLs. You should be looking for a sharp increase in Googlebot activity during the migration period, as Google re-crawls your site to see what has changed.
If your website ranks for images, videos, or other elements that are crawled by bots other than the standard Googlebot, it can be particularly useful to keep an eye on when traffic from Googlebot Images, for example, returns to the same rate as before the migration.
You will also begin to see the first organic traffic landing on new URLs, the ultimate sign of a successful migration: these pages are already indexed and ranking in the SERPs.
Smoother migrations with live log monitoring
Using live log monitoring to track how migrations go live makes it possible to confirm that everything is going as planned. And when that’s not the case, live monitoring allows you to spot and correct problems without delay, limiting their impact on SEO.
Live log monitoring can also be used during any other major website event, or even as a day-to-day monitoring tool to spot trends in crawling, or correlations between errors, traffic, and times of day.
