OpenAI, Google, Microsoft, and Anthropic to support new standard for opting out of AI training
Four major AI companies have announced support for a new "policy token" standard in robots.txt, letting publishers use a single signal to prohibit AI training on their content across all participating companies at once.
OpenAI, Google, Microsoft, and Anthropic have announced that they will support a new shared standard known as the "policy token," which will allow publishers to use a single signal to refuse to have their web content used for training these companies' AI models. The standard was proposed by Kevin Kelley, technical lead for standards at Anthropic, within the IETF. The token will be placed in the robots.txt file, which has so far been used to signal which crawlers may crawl a page for purposes of search, archiving, or AI training.
Until now, websites had to set their AI training preferences separately for each company. The new standard simplifies this process by letting publishers set a preference once, which then applies to all participating AI companies. The announcement comes at a time when publishers are grappling with a growing volume of AI crawlers scraping their content; other initiatives in this area include the Really Simple Licensing standard, which allows publishers to charge for crawler access, and Cloudflare's default blocking of AI crawlers.
According to OpenAI, Google, Microsoft, and Anthropic, their AI crawlers will respect this token and will not crawl a page if they encounter it. Microsoft is said to offer more granular control, where publishers will be able to choose to opt out of AI training separately from AI grounding, the process by which a model draws on a verified source of information to reduce hallucinations. According to Microsoft, pages will still appear in Bing search results regardless of the preferences chosen.
Anthropic has already adopted the standard. Google and Microsoft plan to implement it next year, while OpenAI says it will complete implementation in the coming months.
Why it matters
The standard gives website operators and content publishers a simpler tool for controlling the use of their content for AI training, without having to set preferences separately for each AI company. It relies on voluntary compliance with the signal by the AI companies, rather than technical enforcement or charging, as with other initiatives.
Two audiences, two different impacts
What this means
For individuals
Operators of websites, blogs, or personal pages will be able to use a single entry in robots.txt to refuse the use of their content for training models from these four companies, instead of setting preferences separately for each one.
For a business
Companies that run websites and publish content will get a unified mechanism for opting out of AI training with respect to the major AI providers, which reduces the administrative burden, but the rollout timing differs among the individual companies: Anthropic has already adopted it now, OpenAI in the coming months, and Google and Microsoft next year.
Risks and complianceCheck the original
Event sources
only one source so far · 1 publisher, 1 independent. We count feeds from the same owner only once.