A few days ago we added an automated blocking layer to our own website — to keep malicious scanning IPs out. Before going live we kept one step: observe only, do not actually block, and logged the IPs that “would have been blocked” for a few days.
That step saved us: among the thirty-odd IPs flagged as “should block” during the observation period, 4 were genuine Googlebot.
Had we taken the shortcut and enabled it straight away, we would have shut Google’s crawler out — and nothing would have raised an alert. The site opens as usual, the admin panel looks perfectly normal; only rankings and index coverage would slowly decline over the following weeks, and by the time you notice, the cause is long gone.
This deserves an article of its own, because it is a mistake many companies are making without any idea: they install a security plugin, put the site behind a CDN, configure firewall rules — the protection is in place, but it may well have blocked search engine and AI crawlers along with the attackers.
Let us start with the conclusion: protection and SEO are inherently in conflict, because good guys and bad guys look too much alike
The attackers you want to block and the search engine crawlers you want to welcome share highly overlapping behaviour: both access automatically, neither looks at ads nor clicks buttons, both fetch large numbers of pages in a short time, and both come from IPs you do not recognize.
So any “block by request frequency” rule will inevitably hit crawlers as collateral damage. This is not a matter of badly tuned settings; the criterion itself is wrong. One of our four Googlebots fetched 647 pages in a row over a single connection — on that number alone, it is indistinguishable from a scanner.
A discriminating signal with almost no grey area: look at the “not found” ratio
Frequency is unusable, so what should we use? We found a criterion with such strong separation that you barely have to hesitate: of the pages this visitor requests, how many simply do not exist?
The reason is straightforward: a legitimate crawler follows the leads you provide — whatever URLs are listed in the sitemap and whatever links appear on the pages, that is what it fetches, so its hit rate is very high. A scanner guesses — probing one by one for leaked configuration files, database backups, admin panels, and the vast majority of those, of course, “do not exist”.
In practice the gap between the two is so wide that there is no middle ground: the “not found” ratio for crawlers is essentially 0, while scanners generally exceed 87% and many hit 100%. Using the “not found ratio” as a precondition for blocking is far more reliable than looking at frequency alone.
A counter-intuitive pitfall: a domain containing “google” does not mean it is Google
The observation period also turned up something interesting: an IP whose reverse lookup returned a domain containing “google” (a subdomain under Google Cloud) was at that moment probing our /.git/config — a textbook attack.
The reason: such domains belong to cloud server tenants; anyone can rent one, and it has nothing to do with Google’s own crawlers. So the convenient shortcut “allow anything with google in the domain” will invite real scanners straight in. By the same token, relying on the identity a visitor claims in its user agent is even less reliable — anyone can fill that in at will, and impersonating Baidu’s crawler is the most basic trick there is.
The correct verification method: forward and reverse lookups must match
The industry-standard approach is called forward-confirmed reverse DNS, it appears in the official documentation of the search engines, and it has three steps:
- ① Reverse lookup: resolve the visitor’s IP to a domain name.
- ② Check: confirm whether that domain is the search engine’s official domain (for example, the crawler domains published by Baidu and Google). Match the suffix exactly on dot boundaries; do not use “contains” — otherwise a domain like
xxx-google.comwould slip through. - ③ Forward lookup: resolve that domain forward back to an IP; only if it matches the visitor’s IP is it confirmed genuine.
Only when all three steps pass should you allow the visitor. Step one alone is not enough, and relying on the user agent alone is as good as doing nothing.
Another detail we ran into: if DNS itself fails, the reverse lookup will fail — and you must never treat that as “suspicious”, or search engines will be blocked en masse. Distinguish “there really is no reverse record” from “we could not resolve it on our side”; in the latter case you should skip the decision rather than block.
How small and medium businesses can check themselves: three things that require no technical skills
All of the above is technical detail that most companies do not need to implement themselves. But you do need to know how to check the following three things, because the problem usually lies exactly where you assumed it could not.
- Check “crawl errors” in the webmaster platform. This is the most direct evidence. Baidu and Bing webmaster consoles both report crawl failure rates and error details; if you suddenly see a large number of crawl failures, connection timeouts or access denials, your own protection is almost certainly blocking them. In a healthy state this should be close to 0.
- List everything that can block a visitor. Many companies stack several layers of protection: one at the CDN, one in the server firewall, another in a website security plugin, and possibly another configured by the web agency years ago that nobody knows about. Make an inventory and establish what each layer is blocking.
- Make sure every block has an expiry. This is a hard rule of our own: never issue a permanent block. Rules will eventually misjudge something; a block with an expiry affects at most a few hours, whereas a permanent block keeps doing damage — and nobody remembers whom it blocked years later.
Before enabling any blocking rule, first “observe only, do not block”
This is the single point we most want to make, and the only one that saved us: run a new blocking rule in observation mode first — log whom it would have blocked, without actually blocking. Let it run for a few days, go through the list, confirm it contains no guest you actually wanted to invite in, and only then enable it for real.
This step costs a few extra days, but what it saves you is “three months of inexplicable ranking decline with no identifiable cause”. That is exactly how we recovered those four Googlebots. The trade-off between security and SEO should not rest on guesswork, but on real data examined in advance.
A ten-minute monthly self-check
| What to check | Where to check | What normal looks like |
|---|---|---|
| Crawl error rate | Baidu / Bing Webmaster Tools → Crawl Diagnostics, Crawl Errors | Failure rate close to 0, with no “access denied” errors |
| List of interception layers | CDN admin panel + server firewall + website security plugin | Each layer can clearly explain what it blocks, with no rules of unknown origin |
| Blocklist | Blacklist / blocked IP list in the protection admin panel | Every entry has an expiry time; no search engine IPs are listed |
| robots file | Open yourdomain.com/robots.txt |
No legitimate pages blocked by mistake, and no Disallow: / |
| Indexing trend | Total from site:yourdomain.com / webmaster tools graph |
Rises slowly as you publish, rather than turning downward from some month onward |
As in the previous article on “silent failure,” the point is to record, not just to check: log it once a month so trends become visible. The damage from mistakenly blocking a crawler is gradual; a single day’s snapshot will never reveal the problem.
A few common questions
Q: Should I just skip protection entirely and let everything in?
A: No. Scanners really are looking for your vulnerabilities. Ignoring them slows your servers and, in serious cases, leads directly to data leaks or compromise. The right question is not whether to protect, but how to protect with the right criteria — judge by behavior (such as the not-found ratio mentioned above), not by a blanket rate limit.
Q: After a mistakenly blocked crawler is unblocked, can it recover?
A: Yes, but it takes time. Once a search engine sees repeated access failures from your site, it automatically lowers its crawl rate; returning to the previous cadence usually takes several weeks. So catching the problem early matters far more than fixing it later, which is why you should review the crawl error rate monthly.
Q: Should I consider AI crawlers too?
A: Yes, and it is increasingly important. For AI to cite your content, it first has to be able to fetch it. Keep it out and you simply do not exist in AI answers — far worse than ranking low in search, because AI gives you no “second page.”
Q: How do I know how many layers of protection I have?
A: Asking “what sits in front of our website?” usually yields unclear answers. A more reliable approach is to trace the DNS resolution path and server configuration together and see which components traffic actually passes through. Sort it out once, draw a diagram, and keep it — it will be useful for troubleshooting any access issue later.
Worried your own protection is shutting out search engines and AI?
Siwenlide provides cybersecurity protection and website SEO/GEO operations for small and medium-sized businesses in Beijing and surrounding areas — we handle both, so we know exactly where they conflict: we map how many interception layers you have, review the rules layer by layer, clear mistaken crawler blocks, add a whitelist backed by forward and reverse DNS validation, and then monitor crawl errors and indexing trends monthly. Those 4 Googlebots that nearly got blocked in this article? We caught them on our own site by “observe first, then enable.” Visit siwenlide.com (or take a look at our company profile first) or call 400-686-2011 to talk.
