The IETF's AI Preferences working group is building a way for a site to attach usage preferences to its content, in a draft written by Gary Illyes of Google and Martin Thomson of Mozilla that updates the robots.txt standard (IETF). It defines two places to put a preference: a Content-Usage line inside robots.txt, and a Content-Usage HTTP response header (IETF). The working group set an August 2026 milestone to send the specification toward approval, though it is still a draft and the companion vocabulary does not yet have full consensus.

Access and use are treated separately

The old robots.txt could only say which paths a crawler may fetch. It could not say what you may do with what you fetched. The new rule splits those into two stages. Allow and Disallow still control access. A new Content-Usage rule controls use, for example whether the content may be used to train an AI model (IETF). Usage preferences apply only to paths that are allowed to be crawled. A disallowed path has no usage preference, because there is nothing there to use.

What it looks like

A short robots.txt example from the draft:

User-Agent: *
Allow: /
Disallow: /never/
Content-Usage: train-ai=n
Content-Usage: /ai-ok/ train-ai=y

This lets every crawler fetch the whole site except the /never/ path. Everything it can fetch carries train-ai=n, meaning do not train on it, except the /ai-ok/ section, which is marked train-ai=y (IETF). The same preference can also be sent on a response as a header, written as Content-Usage: train-ai=n.

It is a preference, not enforcement

Here is the part that matters if you scrape. This does not block anything at the server. Like robots.txt itself, it depends on the crawler choosing to obey (RFC 9309). A recent study found that scrapers already follow robots.txt only selectively (arXiv, "Scrapers Selectively Respect robots.txt Directives"). So on its own, Content-Usage changes what a site can clearly state, not what it can physically stop. Any real force would come from a crawler's own policy or from future law, not from the file.

Where Cloudflare fits

Cloudflare released a similar idea ahead of the standard, called Content Signals, using a Content-Signal line in robots.txt with categories such as ai-train and search (Cloudflare). It added these signals to the managed robots.txt of sites it fronts. The category names differ slightly from the IETF draft, but the goal is the same, and Cloudflare describes its version as aligned with the IETF work.

What it means for proxy users and scrapers

Two points. First, this is separate from access. A residential proxy gets you the page. It does nothing about a Content-Usage signal, because the signal is about what you may do with the page, not whether you can reach it. Second, it changes the usage question. If a site marks content train-ai=n and you train on it anyway, you are ignoring an explicit, machine-readable statement of the owner's preference. That does not make fetching the page illegal, but it removes the argument that you did not know the owner objected, which can matter in a contract or policy dispute. For collection that only reads or indexes and does not train, the signal may not restrict you at all, depending on the categories a site sets.

Do you check for AI-usage signals like this before a job, or only the robots.txt Allow and Disallow lines? And do you expect crawlers to honor Content-Usage on their own, or only if a law requires it?