Skip to content
Metricsjar

Documentation · Updated July 2026

Allow Metricsjar Site Audits Through Cloudflare

Metricsjar visits your public pages as an automated crawler. Cloudflare can challenge that request even when the same page opens normally in your browser. This is common when bot protection sees a datacenter network, an unfamiliar crawler, or a request that cannot complete a browser challenge.

When this happens, Metricsjar stops the audit. It does not treat Cloudflare's challenge page as your website or report the challenge page's noindex tag as an SEO problem.

Confirm what Cloudflare blocked

Do this before changing a security rule:

  1. Open your domain in the Cloudflare dashboard.
  2. Open Analytics → Events. In the previous dashboard, open Security → Events.
  3. Set a short time window around the audit.
  4. Filter for the Ray ID shown by Metricsjar. If no Ray ID was returned, filter for the user-agent MetricsjarSiteAudit/2.0 (+https://metricsjar.com).
  5. Expand the matching event and read its Service and Rule fields.

Cloudflare recommends Security Events as the starting point for false positives. A Ray ID connects the response Metricsjar received to the security event that produced it. Security Events can be sampled, so use the shortest available time window if the event does not appear.

If the service is Super Bot Fight Mode

Super Bot Fight Mode supports scoped exceptions:

  1. Open Security → WAF → Custom rules.
  2. Select Create rule.
  3. Name the rule Allow Metricsjar site audits.
  4. Match this expression:
(http.user_agent eq "MetricsjarSiteAudit/2.0 (+https://metricsjar.com)")
  1. Set the action to Skip.
  2. Select Super Bot Fight Mode only.
  3. Save the rule, place it before rules that challenge automated traffic, and run the audit again.

Skipping only Super Bot Fight Mode leaves your managed WAF rules and rate limits in place. User-agent strings can be copied, so do not use this expression to skip every security product.

If the service is Bot Fight Mode

Cloudflare's free Bot Fight Mode does not support WAF Skip rules. A matching IP Access Allow rule can prevent Bot Fight Mode from triggering, but it also bypasses several other security checks and requires a stable, trusted source IP. Metricsjar does not currently publish one, so do not add a broad IP or ASN exception for this audit.

Cloudflare documents two practical choices:

  1. Open Security → Settings and filter by Bot traffic.
  2. Turn Bot fight mode off.
  3. Run the Metricsjar audit again.
  4. Turn Bot Fight Mode back on after the audit.

For a permanent crawler exception, move to Super Bot Fight Mode or Bot Management and use a scoped Skip rule. A WAF custom rule cannot bypass plain Bot Fight Mode.

If a WAF rule caused the block

Use the rule ID from Security Events. Exclude the Metricsjar request from that specific custom rule or create an exception for the specific managed rule that fired. Do not skip the complete managed ruleset unless Cloudflare support has confirmed that the broader exception is necessary.

If the event shows an origin response instead of a Cloudflare security action, check your hosting firewall, security plugin, and origin access logs for the same timestamp.

Why robots.txt does not fix this

robots.txt tells a cooperative crawler which URLs it may fetch. It does not override Cloudflare's WAF or bot protection. Cloudflare can challenge a request before it reaches your origin or reads your site's crawler policy.

Metricsjar respects robots.txt after it can reach the site.

Why established SEO crawlers are blocked less often

Ahrefs and Semrush publish stable crawler identities, dedicated IP ranges, and distinct user-agents. Cloudflare lists their SEO crawlers in its Verified Bots directory. Their support docs still ask site owners to allowlist their crawler IPs when custom firewalls return a 403.

Metricsjar currently identifies itself with a stable user-agent but does not publish a dedicated IP range. Do not allowlist an entire hosting-provider ASN for a site audit.

Primary references

Keep reading

Documentation

Get Started With Metricsjar

Documentation

Read Your Dashboard