Introducing OnCrawl Labs: an R&D platform for SEO, Data Science and Machine Learning

April 22, 2020 - 4  min reading time - by Rebecca Berbel
Home > French > OnCrawl Labs: an R&D platform for SEO

OnCrawl has released a new R&D platform exploring the intersection of technical SEO, data science and machine learning: OnCrawl Labs.

What is OnCrawl Labs?

OnCrawl Labs is a platform that offers a portfolio of algorithms as solutions to address strategic SEO issues.

Created with those who have advanced SEO skills in mind, OnCrawl Labs is meant for people who are curious about what data science can bring to technical SEO. The platform is designed to offer the greatest impact to those who have either a basic knowledge of data science or who can collaborate with others in data science roles.

Using Google Colab and written in the languages Python and R, the projects in OnCrawl Labs provide detailed context, explanations, and documentation for each notebook, which can be run as-is as examples, or adapted to any user’s website.

This allows OnCrawl to offer a behind-the-scenes look at some of the research and development carried out by our teams. More importantly, though, it allows us to offer features not yet available on the SEO market.

What is included in OnCrawl Labs?

At the time of its release, OnCrawl Labs includes three complete data science and machine learning projects and their documentation, with two more in production for release during the summer of 2020.

Real-time indexing

Index your new and top-priority URLs as soon as they are discovered by a crawl by submitting them using the Bing API, or facilitate their discovery by including them in your sitemaps for Google.

Getting new pages indexed is a challenge for SEOs in industries with frequently evolving sites, particularly e-commerce, classifieds, and online publishers, where the rapid search visibility of new pages directly affects the business.

The major search engines provide means of manually submitting pages for indexing. However, in the use cases described above, this can require SEO teams to maintain extensive daily lists of pages created–sometimes automatically–by production and content teams. Obtaining a complete list can be difficult. And depending on the number of new pages, manual submission is often not a feasible option.

Real-time, automated submission addresses these issues.

This project was released with the initial release of OnCrawl Labs.

SEO text generation

Test Transformers with your own data and generate new qualitative texts in any language.

Good SEO requires good content, but content creation is arguably one of the most expensive elements in creating and maintaining a website.

Using the award-winning methods presented at TechSEO Boost earlier this year, harness some of the technology behind BERT to automate the mass-creation of short texts, such as meta descriptions, anchors and titles, with the level of natural language quality required for effective CRO and SEO.

This project was released with the initial release of OnCrawl Labs.

Anomaly report

Use unsupervised machine learning methods to detect under- and over-performance on any SEO metric tracked with OnCrawl.

Anomaly detection allows you to know whether a change seen in an audit is within the “normal” range for the website, or whether the change represents an unusual event that needs to be addressed.

At the same time, using machine learning to find anomalies revealed by crawls has the advantage of allowing you to take seasonal events into account, along with gradual changes to the website over time

Examining anomalies can also reveal whether certain metrics are key to a website’s SEO, and which are only incidental.

This project was released with the initial release of OnCrawl Labs.

SEO long-tail prediction

Predict future long tail trends using Facebook Prophet algorithm.

Long-tail keywords are a key to SEO, as they often bring in more traffic together than any top, highly competitive keyword. However, as they depend on such a small number of searches, they can be difficult to predict and plan for.

With long-tail prediction:

  • Be certain of your investment
  • Balance expenses between the SEO budget and the investment in paid search

This project was released summer 2020.

Internal linking generator

Access insights for internal linking improvements and generate lists of candidates for addition / deletion of internal links.

This project will be released this summer.

How does OnCrawl Labs help OnCrawl?

OnCrawl Labs is first and foremost a laboratory of ideas.

OnCrawl will perfect and expand on popular notebooks in order to bring new features to the OnCrawl platform. Notebook subjects and OnCrawl Labs user feedback will also help define and prioritize the OnCrawl product roadmap.

How to get access to OnCrawl Labs

Access to the OnCrawl Labs platform is available for free to all OnCrawl users. If you are already an OnCrawl user, you can find OnCrawl Labs here.

The ability to examine, copy, and adapt the contents of all projects on the platform for your own use is included with the OnCrawl API option.

Rebecca is the Product Marketing Manager at Oncrawl. Fascinated by NLP and machine models of language in particular, and by systems and how they work in general, Rebecca is never at a loss for technical SEO subjects to get excited about. She believes in evangelizing tech and using data to understand website performance on search engines. She regularly writes articles for the Oncrawl blog.
Related subjects: