Additional crawl behaviors available

June 8, 2016 - 2  min reading time - by Emma Labrador
Home > Product updates > Additional crawl behaviors available

Summer is coming and we keep releasing new product updates! After our SEO trends, we have now added additional crawl behaviors to handle your projects. Moreover, your crawl settings design has been revamped in order to make it easier to setup, and you can deal with new crawl actions.

Let’s see what’s new:

Pause your crawl and restart it when you want

You can decide to pause a crawl whenever you want. You will then be able to relaunch it where you stop it. To do so, click on ‘ACTIONS..’ while your website is analyzed:

pause your crawl

Then, click on ‘PAUSE CRAWL’.

Pause your crawl during the day

Your crawl is now pausing.

pause your crawl and restart it when you want

If you go back to your projects, you will clearly see it:

pause a crawl

Then you can whether resume your crawl or finish it. If you resume it, your crawl will keep crawling from where it stops. If you finish it, you will access a partial report.

new crawl behavior

Why would you like to pause a crawl ? If you want to only run your crawl during the night, you can pause it during the day for example. It can be convenient if our analysis is running for a few days.
You can also launch a crawl and see that you don’t have enough credits left. You can pause it and wait for your credits’ refund to get a complete report.

Stop your crawl and access a partial report

You can now also stop a crawl before it finishes. You will access a partial report of your performance and the credits you did not used will be reported on a next crawl.

You just need to click on ‘ACTIONS…’ and then ‘FINISH’.

Pause your crawl during the day

Then, our crawler will be analyzing the URLs discovered and create a partial report.

stop your crawls

Why should you want to stop your crawl before it ends? We now display current depth and new URLs discovered while we crawl your website. If your crawl is running for a few days and you realize that the depth is high with a few URLs discovered, you can decide to stop your crawl. You will access a partial report if you think it is not relevant to keep crawling.

Find new crawl settings

The way we set up your crawl configuration has evolved. When you fill out your start URL, we automatically configure your crawl settings.

setup your crawl settings

You have either the ability to click on ‘Edit Settings’ and change as many parameters as you want and then save them or directly launch your crawl without any editing. Configuration goes from robots.txt to HTTP headers, subdomains and many more.

customize your crawl settings

edit your crawl settings

edit crawl parameters

Then, click on ‘CONFIRM CRAWL CONFIG’ and then ‘LAUNCH A NEW CRAWL’. Customize your crawl as you wish!

Emma Labrador See all their articles
Emma was the Head of Communication & Marketing at Oncrawl for over seven years. She contributed articles about SEO and search engine updates.