The relation between Javascript and SEO is a highly debated subject in the SEO sphere. Whether your site is using that language or whether you are planning to audit a client’s website rendering Javascript, there are a few things you should take into account before crawling it.
What is Javascript?
JavaScript is a programming language used to make web pages interactive and dynamic. It can be used for comment or AJAX recommendation widgets for instance. Javascript is operating on visitor’s computer and doesn’t need constant downloads from your website.
On the other hand, as Javascript is an interpreted language, it does not need any special program to create usable code. Basic plain text editor are enough to write JS.
It is also the only scripting language supported by all web browsers that handle client-side scripting.
Is Google crawling Javascript?
Back in the day, search engine bots were not able to crawl and index dynamic content using JavaScript but were only able to see what was hold in the static HTML source code.
However, Google has changed their old AJAX crawling scheme guidelines of escaped-fragment #! URLs and HTML snapshots last October, 2015. They are now often able to render and understand web pages.
Javascript usage has increased over the years due to the adoption of Google’s own JavaScript MVW framework AngularJS, progressive web apps and other frameworks like React.
When it comes to crawling JavaScript websites, it is essential to be able to read the DOM loaded after JS has constructed the web page.
Old SEO crawlers that analyzes website links and content are only able to crawl static HTML returned by server. Google can now do both with JS crawlers and Oncrawl has released the same kind of crawler. In fact, Oncrawl enables you to render javascript of any website and any size. But we will discuss how to crawl Javascript using Oncrawl in a next article.
Key pain points about crawling JS
It is critical for sites using JavaScript to access a comprehensive view of their site structure and data for more complete SEO analysis. That’s why before digging further into crawling JavaScript without limits, there a few technical points you should be aware of:
- Javascript crawling takes longer and is more intensive for the server, especially for large websites, because all resources need to be fetched.
- Different JavaScript frameworks exist and so the SEO implications behind them. Start by analyzing how the website you’re auditing is built.
- Many sites are still relying on the old AJAX crawling scheme and a unique set-up. In this case, you are not purely relying on rendering JavaScript for crawling, indexing and scoring.
- Links or click zones must have ng-href attribute (not ng-click) if using the Angular.js framework.
- href attribute will be understood by crawlers only if it is on an Anchor link.
Beside that, Google still has some limitations regarding JS crawling:
- Google needs unique and clean URLs with links located in a HTML.
- The rendered page snapshot is taken at 5 seconds, so content needs to be loaded by that time, or it just won’t be indexed for each web page.
- All the resources of a page (JS, CSS) need to be available to be crawled, rendered and indexed.
Google also says that:
“If you’re starting from scratch, a good approach is to build your site’s structure and navigation using only HTML. Then, once you have the site’s pages, links, and content in place, you can spice up the appearance and interface with AJAX. “
Why crawling Javascript?
Linking and content are part of the most important ranking factors to analyze. As search engines can find and index your JS-rendered content; it’s critical to know what they are seeing and how it’s impacting your SEO.
- Find links: see the full scope of a page’s outgoing links, including those dependent on JavaScript ;
- Find text content: get accurate content quality analysis with the inclusion of content like product reviews and user-generated comments often rendered in JS
If search engines can find and index it, it’s important to include those elements in your SEO analysis. JavaScript is used for valuable content and that is why your SEO crawls must be comprehensive for an exhaustive view of your site.
How we can crawl JavaScript using Oncrawl
Our JavaScript-inclusive crawl includes:
- Evaluation of text content rendered with JavaScript ;
- Pre-render of Angular/backbone Js to transform ng-href attributes on JS links to keep exploring without the a@href markup, as Google will do. You will ensure that all your website can be crawled and enter in our modeling process.
