OnCrawl’s Technology

OnCrawl uses reliable, secure and auto-scalable technology to meet your SEO needs.

Auto-Scalable and Cloud-Based Storage

OnCrawl has an unlimited storage capacity thanks to the most powerful cloud in the world: the Google cloud. Our solution also has access to many data centers across the world to ensure robust and reliable storage.
Data hosting in the Google cloud allows us to adjust our storage according to the volume of data and the number of customers using OnCrawl. In other words, OnCrawl adapts to your needs for storage limits and processing power.
Our solution is auto-scalable: our clusters grow depending on their activity and the allocated resources vary according to the data handled in real time.

Unlimited Crawling Capacity

The technology used by OnCrawl allows us to crawl at an unlimited speed. The only limit is the efficiency of our clients’ infrastructure. OnCrawl is able to adapt its capacity to the desired speed by our customers and adopt a “safe” mode. Currently, we collect up to 25 millions of URLs per day and per website, or approximately 1 billion web pages and 150 billions of links per month.

Secure Data System & GDPR Compliance

We guarantee comprehensive data security for our customers. OnCrawl is completely GDPR compliant and meets all data confidentiality norms. For our European clients, we can ensure data storage on the European continent. We have set up a policy for restricting internal access to data. Every log file that we collect is stored in our isolated and secure FTP space. Our service is protected by the HTTPS protocol and by isolated software components. It also profits from extra protection from a software stack that we maintain up to date.

Check your Crawls in Real Time

With OnCrawl, you can check the progress of your crawls and the analysis of your logs in real time. In order to facilitate your work, we have a notification system which keeps you informed of the progress of your SEO analysis. You can check on the status of your data at any moment.

Go Further with the Access to our API

OnCrawl provides a SaaS app which offers a complete analysis of your data. We know that some of our users would like to build their own dashboards with raw data from OnCrawl and others sources. That is why we offer complete access to our API. Our solution gives total freedom to users who are interested in integrating OnCrawl’s data into their own ecosystem. Thanks to the API, you can go even further with the OnCrawl data!

Want to go beyond our application’s core capabilities and work on innovative projects?

Made for technical SEO and data science experts, OnCrawl Labs provides a portfolio of algorithms to address strategic SEO issues and work on R&D projects. OnCrawl Labs relies on Google Colab, Python and R languages to offer features not yet available on the SEO market.

Discover →

Get started today with OnCrawl and see what everyone’s talking about

Transparent cost – Downgrade or upgrade anytime