Product Update: Crawl over Crawl now available

Our crawl over crawl last feature allows you to compare two different crawls and stresses out crawl evolutions.
It follows our last release about ‘Trends’ that gave you the opportunity to spot global trends between different crawls. Here, you can access complete views of your SEO improvements and highlight differences between crawls on a given theme. We have thus included new types of graphs to read your data.

How to read a sunburst?

You read a sunburst like a traditional pie. These graphics are very useful to follow the evolution of a website, crawl after crawl or to check differences between two versions of a website (between a live version and during a restructuration for instance).
This multi-level pie chart lets you compare two crawls depending on a given theme:

  • The first level and inner circle : shows the pages belonging to the first crawl (the older crawl).
  • The second level and outside circle : shows the pages of the second crawl (the newest one) according to each segment of the inner circle.

So you can easily find, for example, indexable pages in the first crawl that are not anymore in the second one and vice versa.

crawl over crawl indexability overview

In this chart, the inner circle shows the repartition of pages from the first crawl point of view (the older one). You can see that there are indexable pages, no indexable pages and pages that were not in the first crawl but appear in the second (the grey section).
Then, for each section of the inner circle, you can see the pages’ repartition of a given section in the second crawl. The inner grey section means that those pages did not exist in the first crawl but appear in the second one (the outside green and red section belonging to the inner grey one).
Grey sections mean pages are whether new or not existing in the structure depending on which cercle they belong.
By clicking on the legend, you can decide which data you want to display or focus on. Crawl 2 offers a more global view.

Let’s have a look at the inner circle.

The distribution of pages in the first crawl according to their indexability

distribution of pages in the first crawl according to their indexabilityThe first crawl contains 10 854 indexable pages and 177 no indexable pages. 1 661 pages have only been found in the second crawl.

Now have a look at the outside circle. For each segment of the first circle, we find the distribution of these pages in the second crawl.

indexability comparison SEO
Within the 10 854 indexable pages in the first crawl, only 10 104 are still indexable in the second one, 560 are now non indexable in the second one and 190 are not in the second crawl anymore.

Let’s focus on a small section : non indexable pages in the first crawl

indexable pages crawl over crawlindexable pages in the first crawlWithin the 177 non indexable pages from the first crawl, 94 are now indexable in the second one and 13 remain indexable.

non indexable pages from the first crawl oncrawl

Within the 177 non indexable pages in the first crawl, 70 are not located in the second crawl.  94 + 13 + 70 = 177. We find as expected the repartition of the 177 non indexable pages found in the first crawl.

Focus on new pages : pages only found in the second crawl

It allows you to see the status of new pages in terms of indexation.

status of new pages in terms of indexationAll new pages: 1 661 pages.
Within the 1 661 newly created pages, 709 are not indexable.
Within the 1661 newly created pages, 952 are indexable.

Summary : all pages from second crawl

Summary : all pages from second crawl10 104 pages were indexable in the first crawl. 11 150 are now indexable in the second one. 177 pages were non indexable in the first crawl but 1 282 are now non indexable in the second one.
1661 pages have been created. 260 pages have been deleted from the structure.

Crawl over crawl available data

This new feature is divided by business expertises and between the following tabs:

  • Structure
  • Internal linking
  • Content
  • Status Codes
  • Performance

For instance, in the ‘Content’ section, you acccess a strong focus on your duplications’ evolution between crawls:

content-quality-comparison crawl over crawl

Also, you can analyze how your page depth has evolved crawl over crawl. In the graph below, you can see depth movements:

depth movement crawl over crawl

For example, if we look at depth 5, we can see pages that are gone to shallower or deeper depth or pages that come from shallower or deeper depth between crawl 1 and 2. Here, 264 pages that were in crawl 1 and in depth 5 have gone to a shallower depth (depth 4, 3 or 2).

This is just a snapshot of what is available. Our Data Explorer also lets you dig into more than 300 metrics to run crawl comparisons.

How to get our Crawl over Crawl feature?

You need to have at least a Business plan to enjoy our Crawl over Crawl data. You can do unlimited comparaison inside a same project and between a crawl and its previous one with identical start URLs.

crawl over crawl configuration

By ticking the dedicated checkbox in your crawl parameters, you can compare the crawl you are about the launch with the last one.

How to launch a crawl over crawl comparison

You can also check that your crawl configuration is the same by clicking on ‘crawl config comparison’:

crawl configuration comparison

You can also compare two existing crawls, or two crawls with different configurations inside a same project. The number of those types of custom comparisons are however limited.
Also, when running a crawl over crawl, you should do complete crawls if you want to get accurate results. You can’t compare crawls and get reliable results if you don’t compare all URLs.
For the moment, this feature is in beta so you will need us to activate it manually to start using it. So just ask!
Do you have any feedbacks regarding that release ? We don’t stop here as we are already currently working on additional crawl configurations to run your comparisons.

About Emma

Emma is Communication & Marketing Manager at @Cogniteev and write about SEO and search engine updates.

Entries by Emma