In this article, we will see how to quickly and easily test your JS rendering of all your pages using Oncrawl.
SEO and JS
JS and Google: good practices
For Google this is a problem because it requires a lot of time and especially resources. It forces it to go through your page twice, once to retrieve the code, then a second time after rendering the HTML of the JS.
As a direct consequence of the CSR for your SEO, the complete content of your pages will not be visible immediately by Google and therefore it can delay the indexation of them. Moreover, the crawl budget that is granted to your site is also affected because your pages need to be crawled twice.
The SSR (server side rendering)
On the other hand, the cost of resources to achieve this rendering on the server-side can be important. This is where the third option comes in, pre-rendering.
In this hybrid configuration, the execution of the JS is done on the client-side for all visitors (CSR) except for the search engine bots. A pre-rendered HTML content is served to Google bots in order to keep the SEO advantages of SSR but also the economical advantages of CSR.
This practice which at first sight could be considered as cloaking (offering different versions to bots and visitors of a web page) is in fact an idea of Google which is highly recommended. We can easily guess why.
There are many ways to diagnose SEO errors in JS implementation. By using Oncrawl you will be able to test all of your pages automatically without having to do any manual comparisons.
- A crawl with JS rendering enabled
- A crawl with JS rendering disabled
Note that in the case of pre-rendering the second crawl should be done with a Google user-agent in order to crawl the pre-rendered version of the site.
This test can be done in three steps:
- Create the crawl profiles
- Crawl the site with each profile and generate a crawl over crawl
- Analyze the results
Create the crawl profiles
The profile with JS
From your project page, click on “+ Set up new crawl”.
This will bring you to the crawl settings page. Your default crawl settings are displayed. You can either change them or create a new crawl configuration.
A crawl profile is a set of settings that has been saved under a name for future use.
To create a new crawl profile, click on the blue “+ Create Crawl Profile” button in the upper right corner.
Name it “Crawl with JS” and copy your usual crawl profile (the Default for example).
To activate the JS on this new profile, you have to display the additional parameters which are hidden by default. To access them, click on the “Show extra settings” button at the top of the page.
Then go to the Extra settings and click on “Enable” in the Crawl JS option.
The profile without JS
For this second crawl profile, follow the same steps and uncheck the JS enable box.
Note: it is important to have two profiles with an identical scope for the comparison to be meaningful.
If your site is in Server Side Rendering, go to the next step.
If your site is in pre-rendering based for Google bots, you should send us a request to modify the User Agent for the crawl. Once the profile is created, send us a message via Intercom directly in the application so that we can replace the Oncrawl User Agent with a Google bot User Agent.
Launch your crawls and generate a Crawl over Crawl
Once the two profiles have been created, you just have to crawl your site with these two profiles in turn. To make it easier, you can use the crawl programming function.
Schedule a crawl
- On the project page, click on the “Scheduled crawls” tab at the top of the crawl tracking box.
- Click on “+ Schedule crawl” to schedule a new crawl.
- You will then need to choose:
- The crawl profile you want to use for the future crawl
- The frequency to repeat the crawl, choose “Just once”.
- The date, time (in 24 hour format) and time zone (by city) of when you want the crawl to start.
- Click on “schedule crawl”.
Once both analyses of your crawls are available you need to generate a crawl over crawl.
Generate a Crawl over Crawl
- From the home page of the project, launch a crawl over crawl:
- Under “Tasks”, click on the “Running Crawl over Crawls” tab.
- Click on “+ Start crawl over crawl”.
- Select the two crawls you want to compare.
When you click “+ Run Crawl Over Crawl”, Oncrawl analyzes the differences between the two existing crawls and adds the Crawl Over Crawl report to the analysis results of the two crawls.
You can follow the progress of this crawl over crawl in the “Start crawl over crawl” tab on the project homepage. As the crawl is already finished, the over crawl will skip the “Crawling” state and start directly with “Analysis”.
Analyze the results
Go to the crawl over crawl report on the following three views:
- Internal linking
You can also download our custom dashboard.
Which metrics to look at?
Page crawled, Average word count per page and Average text to code ratio
The first indicator Page crawled shows you immediately if the two profiles have crawled the same number of pages.
If the difference is not significant, you can check two on page indicators:
- Average word count per page
- Average text to code ratio
If there are less words per page on average, it means that part of the page content is not available without JS rendering.
Similarly, if the text to ratio is lower, it means that some of the page content is not available without JS rendering.
The text to code ratio measures how much of a page’s content is visible (text) and how much is encoded content (code). The higher the percentage reported, the more text the page contains compared to the amount of code.
Depth, Inrank and Inlinks
You can then look at the metrics related to your internal mesh that are more sensitive. That a small part of the page content is not available without JS rendering is not necessarily problematic for your SEO, but if it impacts your internal mesh, the consequences on the crawlability of your site and the crawl budget are more important.
Compare the average depth, the average Inrank, the average number of Inlinks and internal outlinks.
An increasing average depth, a decreasing average inrank and a decreasing average number of inlinks and outlinks are indicators of the existence of mesh blocks managed in JS not pre-rendered on the server side. As a result, some of the links are not immediately available to the google bot.
The data explorer will allow you to play with the filters to highlight these elements.
Go further with the data explorer and the URL details
In the data explorer
When you look at the Crawl over Crawl data in the data explorer, you will see two columns of URLs: one for the Crawl 1 URLs and one for the Crawl 2 URLs.
You can then add each of the metrics mentioned above (crawled pages, wordcount, text to code ration, depth, inrank, inlinks) twice each to display the value of Crawl 1 and Crawl 2 side by side.
By using the filters you will be able to identify the URLs with the largest differences.
If you’ve identified differences between the SSR and/or Prerendered version and the client-side rendered version, then you’ll need to go into more detail to understand which JS elements are not optimized for SEO.
By clicking on a page in the data-explorer, you switch to the URL details and you can then view the source code as seen by Oncraw by clicking on the “view source” tab.
You can then retrieve the HTML code by clicking on Copy HTML source.
At the top left, you can switch from one crawl to another to retrieve the other version of the code.
By using an html code comparison tool, you can compare the two versions of a page, with JS and without JS executed on the client side. For the rest, it’s up to you!