how we detect AI agent traffic on our website

Around February we noticed a massive spike in web traffic from China. Weird. We have China geo-blocked in Cloudflare and have some basic bot detection in place. After a quick Google search we found many others having this same problem.

screenshot from our google analytics - rise in Chinese traffic correlates with rise in \"Chrome\" browser traffic (Modified Chrome browsers are used by AI agents)

We happen to be a web security platform building an AI agent detection tool that identifies AI agents on your site. Ironically we were now in need of the exact product we were building.

I summarized some of the signals we are now using to detect AI agents, in case you are curious or want to try this at home:

▸ Identity signals: The obvious one is user-agent strings. This is a low hanging fruit and accounts for 90%+ of AI traffic on your site (crawlers, LLM model trainers, agents to answer questions). Some automation platforms like Browserbase add a public identity to their agents which makes them easier to classify.

▸ Network signals: The major AI platforms usually declare themselves or have a known IP/ASN. This picks up major platform AI traffic when user-agent fails. Also check geographic consistency between IP, location, timezone. TLS fingerprints as well.

Nothing shocking so far. What separates traditional bot detection from AI agent detection are the signals below.

▸ Browser signals: AI agents use some form of "headless browser". Automation in these browsers leave clues. Accessibility elements m stripped or inconsistencies in WebGL, Canvas, and Audio contexts.

Our researchers test out the popular headless browsers / stealth extensions to find specific giveaways. Many of these "stealth browsers" will suppress the widely known signals. Lately we've been looking at the top ~5 used stealth browsers (like Puppeteer with stealth extensions) to find nuances that give them away.

▸ Behavior signals: You really do have to go down to this layer to reliably distinguish AI agents from humans. These are things like click patterns, typing speed, page navigation habits, and consistencies (or lack thereof) across different sessions.

For example - one of our researchers looked at Chrome extensions (think ChatGPT Atlas or Claud in Chrome) which are very hard to detect with regular bot detection. One signal we found was click behavior. One of the platforms always clicked buttons dead centre. Another always slightly off center. A third was almost always center but occasionally picked a random location. That was done to add noise, but we still caught the pattern.

Continuous tests are always active on our end, we capture behavior data on page sessions, and run AI models against the data to catch patterns.

This is a condensed version of an article we wrote: https://cside.com/blog/guide-to-detect-ai-agent-traffic-on-your-website

Overall, detection a cat and mouse game. We find a strong signal. It gets patched. We find another. That's why it's best to use a specialized vendor that will take care of this for you.

(we detect more than just the 'crawlers'. cside is focused on detecting fraudulent AI agents like credit card testing bots or bots creating multiple accounts)