Home > Connectors > Server log files + Oncrawl

Server log data and Technical SEO

A complete, in-depth look at search engine bot and organic user hits recorded by your website’s server

Building the right connection between your log data and Oncrawl

Manual or automated upload to FTP/FTPS

Oncrawl supports most log files formatted as a .txt or .json file, uploaded to a secure, private FTP space. This includes support for most common log file formats:

  • Apache
  • Nginx
  • IIS
    • W3C Extended Log File Format
    • NSCA Common Log File Format
    • CloudFlare (Enterprise)
  • And more!

FTP/SFTP Mirroring

Oncrawl supports FTP/SFTP mirroring for your log files.

Connectors to cloud platforms and data streams

Oncrawl has its own connectors to cloud platforms and data streams:

  • Google Cloud Storage
  • Azure Blob
  • Akamai NetStorage

Data Streams:

  • Amazon: AWS S3
  • Google Cloud Storage

Monitoring for latest data received

  • Monitor the quality of the data in your latest log files, the proportion of fake Googlebot hits, and the status of processing for your files.
      • Receive notifications if new log data is not received as expected.

Native connectors in Oncrawl for server log data stored on a cloud platform

Google Cloud Storage

Cloud-based storage services for businesses, all running on Google Cloud’s infrastructure, where businesses can store any type of data, any amount of data, and retrieve it as often as they’d like. Google Cloud Storage replaces a data center and helps businesses modernize their data lake.

Azure Blob

Azure Blob Storage is a Microsoft cloud product that helps you create data lakes for your analytics needs, and provides storage to build powerful cloud-native and mobile apps. Optimize costs with tiered storage for your long-term data, and flexibly scale up for high-performance computing and machine learning workloads.

Akamai NetStorage

Incorporated in 1998, Akamai is a content delivery network (CDN), cybersecurity, and cloud service company. Akamai has deployed the most pervasive, highly distributed content delivery network (CDN) with approximately 340,000 servers in 130+ countries and over 1,200 networks around the world. Akamai’s NetStorage optimizes and scales content delivery with persistent, geographically replicated cloud storage, including for server log data.

Data Stream (AWS S3 and GCS)

Data streams capture new data in a cloud storage system and pass only that data to an outside service, like Oncrawl. Oncrawl supports data streams for AWS S3 and for GCS.

Gain a technical SEO perspective on SEO-related traffic in server logs