AI Bot Compliance and Metadata File Support for AEO and GEO

I would like to share a table that illustrates how various AI LLMs handle different metadata files. These files are important because they help communicate the policies that AI systems should follow when crawling sites. For example, is AI crawling allowed? Are content citations required? Should AI learning be allowed?

Companies like Cloudflare have a product in beta as of this writing that will BLOCK AI from crawling websites. This is an extreme measure to take, but logical for some publishers who are losing subscriber revenues, advertiser money, and traffic to AI. However, blocking all AI is risky because that could potentially reduce exposure and visibility to an audience that increasingly uses AI for answers and information.

Google just announced that they will not use the llms.txt file when crawling websites. llms.txt is a proposed standard for AI to use to see crawling and citation policies for different websites. Since Google refuses to use this proposed standard, the only file that is widely recognized is the robots.txt file. robots.txt can be adapted for some AI LLM crawling rules, similar to what it communicates with search engines.

I created this table that shows different metadata files and whether they are recognized and used by various AI LLMs. It's still early days. I'm hopeful these AIs will work with content producers to recognize what can be crawled nd under which conditions.

Part of your AI strategy should be determining what content you want crawled, and whether you want citations, and do you prefer your content to be referenced. These metadata files can help in that endeavor, especially if and when AI LLMs recognize them.