OnCrawl Labs: Prediction in SEO

November 16, 2020 - 4  min reading time - by Rebecca Berbel
Accueil > Technical SEO > Prediction in SEO

SEO prediction involves using data from previous periods to estimate future periods, adjusting for recurring seasonal events.

The ability to forecast elements in SEO is incredibly valuable. This is one of the things that allows SEOs to provide realistic estimations both for our own use, and for our clients and managers who need this information in order to prioritize and invest in SEO actions.

Why should you predict your organic traffic?

While you can predict many things in SEO, organic traffic is one of the most important.

Since the objective of SEO is to drive quality traffic from search engines to a website, estimating how much traffic you can have in the future is extremely useful:

  • It allows you to set realistic expectations.
  • It makes expected advantages much more concrete and measurable, making it easier for decision-makers to understand the potential ROI of SEO projects.
  • It allows you to accurately differentiate between normal growth or seasonal peaks, and the effect of your SEO projects.
  • It gives you a better understanding of the impact of SEO, allowing you to better balance budgets for different inbound channels.

How can you predict organic traffic?

For years now, SEOs and agencies have discussed how to get a rough idea of how much change an SEO project can make. There are lots of strategies out there. For example, Kevin Indig’s recommendations from October 2020 lay out an extremely effective prediction process. But it’s a process that can be very labor-intensive per keyword.

To see why this is so hard, let’s go back a few years. In 2018, Simon Heseltine laid out the elements that need to be taken into account when making accurate SEO predictions:

  • An established baseline
  • Seasonality
  • Annual trending
  • Upcoming projects
  • Search engine algorithm updates
  • Competitive changes
  • Constant change

And to be accurate, you need to take this into account for each keyword.

Why should you predict your SEO traffic with machine learning?

A year later in 2019, Alice Roussel, working with OnCrawl, wrote up two methods to predict organic traffic. One uses a prediction library for R and linear smoothing techniques for data analysis to account for seasonal events; the other manually links data sources in Excel, but doesn’t account for seasonal trends.

What made Alice choose a prediction library — that is, machine learning?

The answer, in short, is the complexity of search behavior and of the lifecycle of any website. Particularly if you have “noisy” data, with influences from search engine algorithm updates, major on-site projects, and multiple seasonal trends, then machine learning can be necessary to help spot patterns and determine the baseline.

Machine learning also greatly simplifies the process of projecting past patterns, like seasonality, onto future forecasts, and simulating the effects of other major events that can affect your traffic.

What do you need to put prediction in place for your SEO?

To accurately predict organic traffic, you’ll need traffic data from a reliable source. For Google, that’s Google Search Console, via their API.

You also need access to at least a year of back data. This back data is also provided by Google Search Console.

If you’re doing this on your own and not comfortable with machine learning and APIs, you might want a data scientist or a software engineer to run the operations following the steps and the code provided in the OnCrawl Labs project.

Why use OnCrawl Labs for your SEO predictions?

Not all prediction algorithms work the same way. The project available in OnCrawl Labs uses Facebook Prophet because we tested several and found that, for keyword and traffic data, it worked well.

Furthermore, we know that you’re probably convinced that predicting traffic is a great idea — and that you likely already have something in place to do that. This is why OnCrawl Labs concentrates on detecting traffic from specific types of queries.

We chose to concentrate on multi-word queries, since long-tail queries are known to bring in very qualified traffic and typically contain more words than shorter, broader keyword terms. This allows you to focus on predicting a segment of traffic that likely contributes more to the ROI of SEO than other segments.

By showing you how to filter the queries from Google Search Console for this OnCrawl Labs, we hope to get you thinking about other segments you might want to concentrate on, and adapting the code we provide to filter for them. This could be branded — or non-branded — keywords, for example. We hate to say “it depends”, but it does depend on your website and your particular niche.

Finally, this OnCrawl Labs project also walks you through the steps of setting up historical events, adjusting their influence and their likelihood of recurring. If you’re new to machine learning, or to prediction, OnCrawl Labs will get you off to the right start when it comes to taking into account exceptional events and seasonality.

What other uses of OnCrawl Labs can improve your SEO?

Rebecca is the Product Marketing Manager at Oncrawl. Fascinated by NLP and machine models of language in particular, and by systems and how they work in general, Rebecca is never at a loss for technical SEO subjects to get excited about. She believes in evangelizing tech and using data to understand website performance on search engines. She regularly writes articles for the Oncrawl blog.
Related subjects: