Using SEO log file analysis is a very relevant and important solution to improve your technical and on-site SEO. Currently, you cannot get your site crawled, parsed, indexed and ranked by search engines without good, relevant SEO. Log files are the only data that are 100% accurate and truly help you understand how search engines crawl your website. Log analysis helps you rank higher and get more traffic, conversions and sales. But before digging into the benefits, let’s understand what log analysis is.
What is a web server log?
A web server log is a journal of outputs made by a web server containing ‘hits’ or records of all requests that the server has received.
The data received is stored anonymously, and includes details such as the time and date in which the request was made, the request IP address, the URL/content requested, and the user-agent from the browser.
These files exist typically for technical site auditing & troubleshooting, but can be extremely valuable for SEO auditing as well. Builtvisible
In fact, when someone fills types a URL, like https://www.oncrawl.com, into a browser, here is what happens: the browser translates the URL into 3 parts:
- Protocol
- Server name
- File name
The server name (oncrawl.com) is converted into an IP address via the domain name server. The connection made between the browser and the dedicated web server allows to you reach the requested file. An HTTP Get request is then sent to the web server for the right page which then sends back the content of page you see displayed on the screen. Each of these requests is thus regarded as a ‘hit’ by the web server.
The appearance of a log file depends on the type of server and configurations used (Apache, IIS, etc.) but there are elements of a record that you will always be able to find:
- Server IP
- User-agent
- Timestamp (date & time)
- HTTP status code
- Method (GET / POST)
- Requested URL (aka: URL stem + URL query)
- Referrer (the external site, such as Google, from which a user arrives)
Additional attributes can be added, such as:
- Host name
- Bytes downloaded
- Time taken
- Request/Client IP
Log file analysis and SEO, what’s the point?
Log file analysis allows you to understand exactly how search engines crawl your website as every request made on the hosting web server is saved. You just need to filter by user-agent and the client IP to access crawl details for search engine bots. You can thus analyze crawlers’ behavior on your website by examining when, how frequently, and on what pages crawlers are present. This lets you answer the following questions:
- Is your crawl budget spent efficiently?
- What accessibility errors were met during crawl?
- Where are the areas of crawl deficiency?
These 3 questions are just a quick overview of the potential of log analysis. Log analysis can also help you determine whether your site architecture is optimized, if you have site performance issues, and so on.
Technical SEO insights you can find in log data
There are different metrics you can look at in your log files to improve your SEO.
Bot crawl volume
Bot crawl volume refers to the number of requests made by GoogleBot, BingBot, Baidu, Yahoo and Yandex, etc for a given period of time. Bot crawl volume can show you if you have been crawled by a specific search engine. For instance, if you want to get found in China but Baidu is not crawling you, that is an issue.
Crawl budget waste
A crawl budget refers to the number of pages a search engine will crawl on your site in a given period of time, usually a day. This budget is linked to the authority of the domain and proportional to the flow of link equity through the website.
This crawl budget is often wasted to irrelevant pages. Let’s say you have a budget of 1 000 pages per day. You want these 1 000 pages that are crawled to appear in the SERPs. But bots may be crawling old pages, duplicate pages, redirected pages, or other pages that aren’t important to your SEO strategy. If you have fresh content you want to be indexed but no budget left, then Google won’t index this new content. That’s why you want to watch where you spend your crawl budget with log analysis.
Temporary 302 redirects
Those kind of redirects are not optimized for your SEO as they use a lot of crawl budget: search engines return often to see if the temporary redirect is still in place from the old URL to the new one. Prefer permanent 301 redirects. Log data analysis can help you spot these redirections.
Response code errors
Log data analysis can also help you spot status code errors like 4XX and 5XX that can badly impact your SEO. Understanding the different HTTP status codes can help you rank higher.
Crawl priority
You can set up your crawl priority in your XML sitemap or by checking your internal linking structure. This prevents Google from ignoring some pages or sections of your site.
In fact, analyzing your logs can highlight URLs or directories that are not often crawled by bots. For instance, if you want a specific blog post to rank for a targeted search query but it is located in a directory that Google only visits once every 6 months, then you will miss the opportunity to gain organic search traffic from this specific post for up to 6 months before Google comes to crawl it again.
Duplicate URL crawling
URL parameters like filters or tracking can result in crawl budget waste, since in this case search engines are crawling different URLs with the same content. Search Engine Land published an excellent article about how to fix this problem.
Last crawl date
Log file analysis can point out when Google last crawled a specific page you want to be indexed quickly.
Time between the first crawl and the first organic traffic
If you monitor your logs regularly, you can also track how long it takes between the time you publish content, the time it is crawled, and the time you get your first organic hits. This will allow you to set up your content calendar for seasonal campaigns or for events with a specific date.
Crawl budget
As we said before, a crawl budget is quite linked to the authority of the domain and proportional to the flow of link equity through the website. Because Google does not want to waste crawling time on a low quality website, you will want to monitor Googlebot’s real-time activity on your website to see if bots are spending enough time on your content.
Changes in bot activity on your website are early warning signs of algorithm changes or site changes that affect your SEO. With log analysis, you can spot changes before they become issues.
We offer a SEO log file analyzer at OnCrawl. You can download it as open source and carry out your own log analysis for free (you only pay your hosting costs). Or you can use our Advanced Data Platform, upload your logs to a secure, private ftp space, and get additional insights through cross-analysis with additional data.
Thanks for this.
What would be great is to know HOW to use the data in a meaningful way. What do you do with the information once arrived at? Can I change my sitemap or something to get search bots to crawl the pages I want first etc?
How can this be of value to me? Isn’t much of this data also available through analytics?
Thanks
Once you have the information, you can utilize it to see where any weak points in your indexing may be, this is technical SEO stuff. That means, it’s not going to be Problem A + Implementation B = Fix C , It’s more like Problem A + Problem B side effects – Knowledge C * Solution D + Problem E ( Solution F / Problem A ) = Data that needs to be processed and results that need to be waited on and then gleamed.
All joking aside, seeing how bots are crawling can help you see if having a sitemap setup a certain way helps, or if there are deeper issues of internal linking needing to be addressed. It’s a way to see deeper and how you use the data is really endless, start with relevancy and move onward wherever it may help.