Day: August 12, 2026

Before enabling automatic blocking on our website, we ran a monitoring-only mode that logged activity without actually blocking anything. Of the 30-plus IPs flagged as “should be blocked”, 4 turned out to be the real Googlebot. Turning blocking on would have been like punching ourselves in the SEO, and no alarm would have been raised. This article explains why rate-based blocking inevitably hits crawlers, a nearly zero-grey-area identifying trait (the 404 ratio: crawlers ≈0%, scanners 87% or more), the trap of a scanner whose domain contains “google”, the correct three steps for forward and reverse DNS verification, and a ten-minute monthly self-check for small and medium businesses.