We’re excited to announce the availability of public beta version of OnCrawl’s Splunk integration.

If you’re a Splunk user, tune in to the rest of this product update to find out how OnCrawl makes it easy for you to merge data from your log files with your crawl data for extra-intelligent analysis. You take care of the Splunk setup; we’ll take care of the rest.

What’s new with OnCrawl and Splunk?

OnCrawl now offers a direct connection with Splunk to integrate log data managed by Splunk.

Splunk is a data aggregation tool that processes, among other things, web server log files. If you already use Splunk, you can take advantage of our new integration in order to automate and streamline the process of filtering and uploading your log files to OnCrawl.

Why analyze your log files with OnCrawl?

Your log files represent all activity on your website as recorded by the website’s server itself. It is the most complete and most reliable source of information regarding the number and frequency of bot hits, the number and frequency of SEO organic hits coming from SERPs, breakdown by device type (desktop vs. mobile) or by URL type (page vs. resource), precise page sizes and actual HTTP status codes.

Spotting peaks or changes in crawl behavior, knowing how long it takes for the first visitors to arrive on a new page, or monitoring how bot and user activity affect a page’s rankings are just some of the many advantages provided by SEO log analysis.

Advantages for Splunk users

If you’re a Splunk user, the OnCrawl Splunk integration automates the uploading of log files to the private, secure FTP space associated with your account. This facilitates the process not only by eliminating manual actions that take up your (or your web developers’) time. It also ensures that log files are uploaded daily, preventing gaps in the data.

Using the Splunk integration, you benefit from additional log file analysis compared to the information Splunk can offer. OnCrawl provides in-depth cross-analysis between log data from Splunk and data from other sources.

This cross-analysis gives you access to advanced OnCrawl metrics, including:

  • Crawl behavior breakdown by individual bots
  • Time between the first crawl and the first organic visit
  • Comparison of desktop vs. mobile bot hits
  • Comparison of desktop vs. mobile organic visits
  • Number and size of resources (js, images, css…) crawled by Google
  • Correlations between ranking and crawl frequency
  • Correlations between number of impressions and crawl frequency
  • Correlations between CTR and crawl frequency

What data does OnCrawl use from Splunk?

OnCrawl fetches your log files, filters for lines related to SEO (organic) visits and Googlebot hits, uploads the filtered results to the OnCrawl ftp, and queues the files for analysis. The parsing and analysis of your logs is carried out by the OnCrawl Log Analyzer.

Your log files provide raw data concerning:

  • The number of Googlebot hits for the different bots belonging to Google.
  • The number of organic hits (visitors coming from a Google SERP).
  • The timestamps of Googlebot and organic hits.
  • URLs receiving Googlebot and organic hits.
  • HTTP status reported for URLs receiving Googlebot and organic hits.
  • Page size of URLs receiving Googlebot and organic hits.
  • Reported load time for pages receiving Googlebot and organic hits.
  • Crawl frequency.

This data is combined with other data on the OnCrawl platform, including crawl data, to provide precise and actionable insights.

What versions of Splunk are supported?

OnCrawl supports Splunk Cloud and Splunk Enterprise versions, which both allow third party connections.

How to connect your Splunk and OnCrawl accounts

Interested? Reach out to your sales contact or click on the blue Intercom button at the bottom right of any screen in the OnCrawl application and we’ll be happy to help you set it up.

If you don’t have an OnCrawl account yet, now’s a perfect time to start a free trial.