OnCrawl’s Technology

OnCrawl uses a reliable, secure and auto-scalable technology to meet your SEO needs.

Auto-Scalable and Cloud-Based Storage

OnCrawl has an unlimited storage capacity thanks to the most powerful cloud in the world: the Google cloud. Our solution also has access to many data centers across the world to ensure robust and reliable storage.
Data hosting in the Google cloud allows us to adjust our storage according to the volume of data and the number of customers using OnCrawl. In other words, OnCrawl adapts to your needs for storage limits and processing power.
Our solution is auto-scalable: our clusters grow depending on their activity and the allocated resources vary according to the data handled in real time.

cloud

Unlimited Crawling Capacity

The technology used by OnCrawl allows us to crawl at an unlimited speed. The only limit is the efficiency of our clients’ infrastructure. OnCrawl is able to adapt its capacity to the desired speed by our customers and adopt a “safe” mode. Currently, we collect up to 25 millions of URLs per day and per website, or approximately 1 billion web pages and 150 billions of links per month.

illimited crawl

Secure Data System & RGPD Compliance

We guarantee comprehensive data security for our customers. OnCrawl is completely GDPR compliant and meets all data confidentiality norms. For our European clients, we can ensure data storage on the European continent. We have set up a policy for restricting internal access to data. Every log file that we collect is stored in our isolated and secure FTP space. Our service is protected by the HTTPS protocol and by isolated software components. It also profits from extra protection from a software stack that we maintain up to date.

Secured Data

Real-Time Crawl Checking

With OnCrawl, you can check the progress of your crawls and the analysis of your logs in real time. In order to facilitate your work, we have a notification system which keeps you informed of the progress of your SEO analysis. You can check on the status of your data at any moment.

crawls in real time

Go Further with the Access to our API

OnCrawl provides a SaaS app which offers a complete analysis of your data. We know that some of our users would like to build their own dashboards with raw data from OnCrawl and others sources. That is why we offer complete access to our API. Our solution gives total freedom to users who are interested in integrating OnCrawl’s data into their own ecosystem. Thanks to the API, you can go even further with the OnCrawl data!

Access to our API