Bot traffic filtering is the process of separating automated traffic from human traffic so that your metrics describe people. It matters more every year: Cloudflare reported in 2026 that more than half of all internet traffic is now non-human, with AI training crawlers alone accounting for 52% of crawler requests, up from 22% in spring 2025.
Unfiltered bot traffic does not just inflate session counts. Because bots almost never complete your goal, they act as a constant dilution of the numerator in every conversion rate you report — and in every experiment you run.
The Four Detection Methods
Ordered from least to most reliable:
- User agent strings. Read the
User-Agentheader and match against known bot tokens. Trivial to implement and trivial to spoof — a user agent can be set to any value, so this offers no guarantee. - Reverse DNS verification. Confirm the requesting IP resolves to the operator's domain, then confirm the forward lookup matches. Defeats naive spoofing, but fails for agents running on generic hosting.
- Published IP ranges. Major operators publish their address blocks; OpenAI publishes JSON ranges for its SearchBot. Effective for declared indexers, useless for browser-based agents on third-party infrastructure.
- HTTP Message Signatures (RFC 9421). The agent cryptographically signs its requests with keys published in a
.well-knowndirectory. This is the only method that survives a determined spoofer, and it is what OpenAI's ChatGPT agent uses — necessarily so, since it presents a standard Chrome user agent from hosting-provider IPs.
Filtering for Analytics vs. Filtering for Experiments
These are different jobs and are often wrongly conflated.
| Analytics filtering | Experiment filtering | |
|---|---|---|
| Goal | Accurate traffic reporting | Valid causal comparison |
| Timing | Can be applied after the fact | Best applied before variant assignment |
| Cost of a miss | Inflated counts | Diluted effect size, longer tests |
| Risk of over-filtering | Undercounting | Post-hoc exclusion correlated with variant, which itself creates bias |
The timing distinction is the important one. Excluding traffic after assignment, in a way that happens to correlate with the variant a visitor saw, is a known cause of sample ratio mismatch — you can create the very bias you were trying to remove.
Common Mistakes
- Trusting the user agent alone. It is the easiest check and the weakest one.
- Blocking agentic browsers outright. They represent a real person with real intent; blocking them removes a customer, not a bot.
- Never re-checking. Bot composition shifts fast. Tracking agent share of sessions as a guardrail metric surfaces the drift before it quietly corrupts a quarter of results.