One line of text unites the AI industry's answer to "don't train on my content"

For three years, telling an AI company not to train on your website meant knowing the name of its crawler: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, a meta tag for Microsoft. Each vendor had its own switch, syntax and documentation. On Friday, the four biggest AI companies agreed to read a single line instead.

OpenAI, Google, Microsoft and Anthropic announced they will honour a public signal in robots.txt, the small text file that has governed how machines treat websites since 1994. A publisher can write one rule that reaches every participating AI provider at once, saying whether its content may be used to train generative models. OpenAI says it will implement the signal in the coming months, Google plans support in 2027, and Microsoft's widest-ranging controls take effect from February.

The proposal adds no new directive to robots.txt. A site writes an ordinary rule group under a policy token called __usage_ai-training, using the usual Allow and Disallow lines. A wildcard group never matches the token, and a file without the token shows no preference - so no existing site's meaning changes overnight.

The idea came from the Internet Engineering Task Force's AI Preferences working group, with the specific draft submitted by Anthropic engineer Kevin Kelley. It is not yet an adopted standard, and expires in May 2027 unless the working group advances it. Anthropic says full implementation across its services will take time. This is a voluntary pact among competitors, not regulation; there is no penalty beyond reputation.

The fine print matters. Every company reads the same line but acts differently. Google treats a public opt-out like a Google-Extended block, removing content from future Gemini training and from grounding in Gemini apps, but leaving Google Search untouched. Microsoft splits the decision in two - one preference for training, another for grounding, meaning whether your pages can take part in answers Copilot and Bing's AI Mode generate. Microsoft is also retiring the NoArchive and NoCache tags many sites rely on; administrators have until March 2027 to record their choices again, and sites that miss the deadline default to allowing grounding.

Opting out of training is not opting out of answers. A company can keep its content out of a model's weights while letting AI products quote it, or vice versa, and each vendor draws that line somewhere different. Microsoft warns directly that restricting grounding reduces a brand's visibility in AI-generated answers, though not in ordinary search results.

There is an honest limit to what changed: the signal is voluntary for everyone, it says nothing about payment, licensing or credit, and it does nothing for content already absorbed into today's models. A publisher wanting compensation still needs a contract; one wanting removal still needs a lawyer.

Still, training data shifts from something the industry took to something it must ask for - and the asking is standardised the way the internet standardises things: a token, a vocabulary, a working group, a deadline. Leaders should decide deliberately whether their content may train models, assist answers, both or neither; write that decision once in one place; and not let a default answer for them.