SEOs we are working with were asking for those features for a while and here there are! We are pretty proud to introduce you these little changes that will have big impact on your SEO crawls.

You can now add further parameters in your crawl settings. Get fine reports tailored on your needs.

Schedule your crawls

You can now decide when you want to launch your crawl. Actually, you can set up your parameters and pick a different date to start it. When you are about to launch a new crawl, just go on “Scheduling”, choose your start date and then check your current timezone.

better SEO audits with further crawl parameters

Add a crawl recurrence

You wanted to schedule your crawl recurrence? It is now possible. If you want to run SEO audit on a regular basis, you don’t need to think about it anymore. You can set up your crawl frequency and choose whether you want them to be launched every week or every month. Super easy!

seo audits with oncrawl

Virtual robots.txt

You can use a virtual robots.txt to overwrite your current file. You however need to verify your site using a Google Analytics account to use the virtual robots.txt. Go to “Advanced Settings” and then click on “Edit virtual robots.txt” to modify your file.

modify your robots.txt with the SEO crawler OnCrawl

oncrawl onsite SEO audits

Then, you will be able to modify your robots.txt and even to add your subdomain.

Subdomain crawling

Another important parameter is you can now tick the subdomain crawling.

crawl your subdomain with OnCrawl

User agent choice

You can also now choose your user agent between desktop and mobile.

oncrawl seo audit

Modification of your maximum crawlable URLs

You can now choose how you want to spend your URL credit. If you do not want to use all your credits for one site or if you want to run a quick crawl, you can setup your max URLs for a project.

oncrawl maximum urls crawled by crawl

All those new parameters will offer you more precise SEO audits and will help you increase your rankings. Of course, you can still enjoy our 30 days-trial to try them.