How to see what AI crawlers actually see on your page (fetch your own site the way they do)

Short answer: request your own page from outside your own network, with no cookies, no JS execution, and from the country you care about. That is most of it. The rest is noticing how different that response is from what you see in your browser.

I got into this because a client swore their pricing page was fine and ChatGPT kept describing prices from two years ago. Turned out their edge was serving a cached variant to anything without a session cookie. Their own team never saw it because they were all logged in. Great.

So the routine I use now. Fetch the URL cold, no cookies, plain HTTP client, no headless browser. Then fetch it again through an exit in the country you actually care about. Then diff the two against what your browser shows. If your content only exists after JS runs, assume a chunk of crawlers never see it.

The geo part is where it gets annoying, because a lot of sites vary content by region and you cannot test that from one office. I run the same URLs through a couple of pools, Bright Data and Byteful, mostly because residential country targeting is the cheap way to check whether your page is even the same document in Germany.

Anyone found a target where the cold fetch and the browser render differ more than you expected?