Know which AI crawlers are reading your site

The idea

A self-hosted tool that reads your existing web server access logs and answers one question properly: which AI crawlers are hitting this site, what are they taking, and what is coming back. It classifies every request against the published IP ranges and reverse-DNS records of the known crawlers, so a spoofed user agent is reported as a spoof rather than as GPTBot. On the other side of the ledger it counts inbound visits whose referrer is an assistant — chatgpt.com, claude.ai, perplexity.ai, copilot.microsoft.com — and joins the two into a per-page table: crawled 340 times this month, sent 12 visitors back.

Output is a dashboard plus a weekly digest, and a generated robots.txt and llms.txt you can actually serve.

Why build this

Anyone who publishes documentation, a blog, a recipe archive, or a product catalogue is now being crawled by a dozen agents that are not search engines and do not behave like them. Site owners have no idea of the volume, cannot separate the training crawlers from the retrieval fetchers that are answering a live user question about their product right now, and have no basis for the robots.txt decision everyone keeps telling them to make. The usual analytics stack is no help: Plausible and GA4 see browsers, not crawlers, and the crawl happens entirely below the JavaScript tag.

The data exists and nobody is looking at it. Access logs already contain every one of these requests; the missing pieces are a maintained classifier and a way to read the result. The distinction that matters most — a training crawler taking your archive versus a retrieval fetcher pulling one page because a user asked — is visible in the logs as a difference in user agent, request pattern, and burst shape, and it is the difference between "block this" and "make sure this works".

Stack sketch

• Go single binary, no runtime to install. Reads nginx/Caddy JSON access logs by tailing the file, or ingests Cloudflare Logpush batches from object storage.
• DuckDB as the store, one file on disk. Crawl volumes are tens of millions of rows a year for a decent-sized site; this is an aggregation workload, not OLTP, and a columnar file beats a Postgres you have to operate.
• Bot verification in two stages: match against the published range files (openai.com/gptbot.json, Anthropic's and Perplexity's equivalents, Google's googlebot.json) refreshed daily, then forward-confirmed reverse DNS for the crawlers that publish hostnames instead. Unverified claims get a spoofed label, never the bot's name.
• A YAML crawler registry shipped as data and updated independently of the binary: user agent pattern, verification method, declared purpose (training, retrieval, search index), vendor opt-out mechanism.
• Embedded dashboard built with htmx and server-rendered templates. No build step, no SPA.
• Digest email over SMTP, and a webhook for the alert path.

Scope for v1

In:

• One site, one log source, local DuckDB file.
• Verified classification across the dozen or so crawlers that publish ranges, plus a bucket for unverified self-declared bots.
• Per-page, per-crawler, per-day aggregates with a volume trend and a bandwidth figure.
• Referral counting from assistant hostnames, joined to the crawl table by URL path.
• Weekly digest: new crawlers seen, biggest movers, top pages taken, spoof count.
• Generate robots.txt and llms.txt from the registry and your per-crawler allow/deny choices.

Out: multi-site and multi-tenant, real-time streaming, any kind of blocking or WAF enforcement, and tracking whether assistants cite you in their answers. That last one is tempting and should wait — it needs querying the assistants themselves, which is unreliable, rate limited, and a different product.

Where it could go

Enforcement is the obvious second step, and the tool is already holding the only hard part: verified identity. A small middleware or Caddy module that consults the same registry can serve full content to verified retrieval fetchers, a 403 to spoofed ones, and a paywall stub to training crawlers you have not agreed terms with. That is a real policy engine rather than a robots.txt suggestion, and it only works because verification happened first.

After that, the interesting direction is valuation. Join crawl and referral data against your own conversions and you can put a number on what AI traffic is worth per vendor, which is the input every publisher currently lacks when a licensing conversation starts. An agency edition — many sites, one dashboard, benchmarks across a portfolio — is the straightforward commercial version, and a Cloudflare Worker edition that skips log plumbing entirely is the easiest on-ramp for people without server access.

Watch out for

The crawler landscape churns monthly: new agents appear, vendors rename theirs, published range files move or quietly change format. Keep the registry as versioned data with a signed update feed, and never let a classifier change require a release. Access logs are also personal data in most jurisdictions, so truncate or hash client IPs after verification and keep raw lines for days, not years.