In late February and early March, a number of websites using Cloudflare as a CDN may have noticed an increase in host status errors within Google Search Console. Several host data errors are related to the Cloudflare/cdn-cgi endpoint, some with the /rum URI appended.
This endpoint is managed by Cloudflare and it cannot be modified or customized in any way. While the endpoint is not used by every Cloudflare product, several products utilize it, including JavaScript detection, image transformation, and email address obfuscation.
The/cdn-cgi/rum endpoint is related to the Web Analytics product. When you try to access the URL from a web browser, it will return a 405 status code; however, for a number of websites, this is returning 5XX errors in Search Console.
Moderate fail rates relating to /cdn-cgi. For some websites, they have seen upwards of a 15% fail rate per day.
Having looked at numerous websites experiencing this issue, no discernible patterns have emerged: the problem is platform, region, and Cloudflare plan agnostic.
What is the host status report?
The host status report in Google Search Console shows when Google has encountered problems accessing URL pathways. These problems can be caused by your robots.txt file, issues with DNS resolution, or server connectivity problems.
Robots.txt exclusion is one of the more common host status issues on websites. It occurs only when the robots.txt file specifically blocks Google from accessing your website. This is a relatively easy fix and can be validated by checking your robots.txt allow and disallow rules.
Host status can also be affected if there is an issue with resolving your DNS. This typically means there is an issue with how your domain name has been set up, which prevents Google from connecting to your website.
To resolve this, it’s usually necessary to contact your domain registrar or update your DNS settings to ensure all of your records are pointing to the correct servers.
A server connectivity issue relates to how your server is configured or whether your server has uptime when Google attempts to crawl the site.
To resolve these issues, you should speak with your infrastructure teams, web host, or server administrator to understand how to fix and maintain uptime. This could be done through a server load balancer or by regulating which bots can access your site to prevent server overload.
What is the /cdn-cgi endpoint?
The cdn-cgi endpoint is a unique path Cloudflare uses to serve various performance, security, and optimization features. It allows Cloudflare to execute certain built-in functionalities without needing server-side configurations on your origin server.
Common URL paths you may encounter, or have seen in your Google Search Console errors, include:
- /cdn-cgi/challenge – Used for Cloudflare CAPTCHA and bot protection.
- /cdn-cgi/trace – Provides debug information about your connection, including your IP address, Cloudflare data center, and other details.
- /cdn-cgi/image – Used by Cloudflare Image Resizing.
- /cdn-cgi/pe/bag2?r[]=https://example.com/script.js – Related to Rocket Loader (JS optimization).
- /cdn-cgi/trace – The debugging tool shows whether HTTP/3 or Cloudflare is active.
- /cdn-cgi/rum – Used for Real User Monitoring (RUM), which collects visitor performance metrics to analyze website speed and user experience.
Does this issue impact my organic performance?
There is no valuable content for Google on this pathway, so this issue will not affect your organic performance. If your host issues involve other URLs on your website that should return 200 status codes and be reachable, crawlable, and parsable by Google, then these issues may affect your ongoing performance.
However, if the problem you’re seeing relates specifically to the Cloudflare Endpoint, then the answer is no – your organic performance should remain unaffected by this issue.
[Case Study] Managing Google’s bot crawling
Why aren’t these errors appearing in my server logs?
Another key insight is that these 5XX errors won’t appear in your origin server log files.
I’ve tested a number of sites to validate this, and it’s because the errors are triggering in the Cloudflare middleware rather than on your origin server.
Therefore, the request from Googlebot isn’t reaching your origin server to trigger a response, so you won’t have any log events in your traditional server logs showing these 5XX errors occurring for the /cdn-cgi pathway.
Can I safely block /cdn-cgi in my robots.txt file?
In short, yes.
Cloudflare recommends blocking this endpoint in your robots.txt file as there is nothing of value to crawl that will add any value to any agent other than Cloudflare.
Websites that have blocked this URL path in the robots.txt have seen the Google Search Console host status issue resolve within approximately 48 hours with no adverse effects.
