Have AI Crawlers Visited Your Site? Check Your Server Logs, Plus Two Pitfalls We Found (2026)
Most claims about “how AI sees you” are indirect guesswork. Only server logs are direct evidence: who came, how often, which pages they fetched, and whether you shut them out at the door. This article gives a three-step check (count each crawler’s fetch volume, see whether status codes show rejections, see whether the pages fetched are your key pages), plus two pitfalls we found when we ran it on our own site: (1) all six sitemap addresses listed in the robots file were dead; after redirects they returned web pages, not XML, and the one that actually worked was missing entirely, so every crawler entering the site received an invalid directory. (2) One search engine crawler was hammering 404s against an address structure that disappeared long ago, burning crawl quota for nothing. A ten-minute self-check checklist is included.