AI preferences
AI preferences are a proposed standard way for websites to say how AI may use their content, such as for training or for answers.
AI preferences are a way for a website to say how AI companies may use its content. For example, a publisher might be happy for its pages to be used in search answers, but not to train AI models. The idea is to write that down once, in a form every AI company's software can read.
The work is happening at the IETF, the body behind many of the web's technical standards, in a group called AI Preferences.
Why robots.txt isn't enough
Today, the main tool you have is robots.txt. It says which bots may visit which pages. It doesn't say what a bot may do with a page once it's read it.
That's why AI companies split their bots by job: one for training, one for search, one that fetches pages for a user. You block or allow each bot by name. If a company adds a new bot or changes what an old one does, your rules may no longer mean what you meant.
What the standard aims to do
The group's charter sets out three pieces:
- A shared list of words for the uses you can allow or refuse, such as training a model
- Ways to attach those choices to your content, including through robots.txt and HTTP headers
- A rule for what happens when two of your choices disagree
It also leaves some things out. The standard won't force anyone to follow your choices. Like robots.txt, it relies on AI companies agreeing to respect it.
For example, Harrow & Pine could say once, in its robots.txt: use our pages in search answers, but don't train on them. Any AI company following the standard would read that the same way, whatever its bots are called.
In AnswerOrbit
The Audit tab in Website reads your robots.txt every day and shows which AI bots it allows, each with its purpose: training, search, or fetching for a user.