AI crawler tracking from server logs
See which AI crawlers like GPTBot, ClaudeBot and PerplexityBot, and which search bots, read your pages, from server or CDN logs with no IP addresses stored.
Crawlers don't run JavaScript, so the tracker never sees them. The AI crawlers report is built from your server or CDN logs instead.
What you see#
Daily hits per crawler and per path, grouped into:
| Category | Examples |
|---|---|
| AI (training, AI search and assistants) | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider, CCBot, Amazonbot, Meta-ExternalAgent |
| Search | Googlebot, Bingbot, DuckDuckBot, YandexBot, Baiduspider, Applebot |
| SEO tools | AhrefsBot and similar |
| Other | Any other bot |
Use it to decide what to allow in robots.txt, to see which pages AI
assistants fetch when answering users, and to spot crawlers ignoring your
rules.
AI assistants sending people to your site is different: those visits appear in the AI assistants channel.
Connect your logs#
Logs are sent with the site's server ingest key (Site settings → Tracking → Server ingest key) to the endpoint shown on the site's AI crawlers page. Connectors:
- Cloudflare: a Worker (or Logpush job) that forwards request logs.
- Vercel and Netlify: a log drain.
- Nginx or Caddy: a small log shipper that posts lines in the "combined" log format.
Privacy#
Log lines contain visitors' IP addresses, so we parse them immediately and keep only the day, the path, the status code and the crawler's name. Lines from normal browsers are skipped. The IP, the referrer and any user field are discarded as soon as a line is split and are never stored.