OpenAI وقوقل ومايكروسوفت وأنثروبيك تدعم إشارة وحدة في robots.txt تخلي المواقع ترفض تدريب الذكاء الاصطناعي على محتواهاOpenAI, Google, Microsoft and Anthropic back one robots.txt signal that lets websites opt out of AI training

OpenAI, Google, Microsoft and Anthropic say they will adopt a new standard that lets website owners opt out of having their content used to train AI, according to The Verge. The idea is that a site owner sets the preference once and it applies across all of these companies, instead of writing a separate rule for each company as they do today.

The shared "policy token" was proposed by Kevin Kelley, a technical lead for standards at Anthropic, within the Internet Engineering Task Force (IETF). It lives in a site's robots.txt file, which has long told crawlers (programs that browse sites and collect their content) whether they may access a site, whether for search indexing, archiving or AI training. The four companies say their AI crawlers will stay away from a site if they find the token.

But the timing differs: Anthropic has already adopted it, OpenAI says "in the coming months", and Google and Microsoft plan to support it next year. Microsoft says it will offer more granular controls to opt out of AI training or AI grounding, which means pointing a model to a verified source of information to reduce mistakes, and sites will still appear in Bing search results whatever they choose.

As for Meta, communications head Andy Stone signalled support for the idea but did not say whether the company will honour it. The announcement comes as publishers deal with a continuing flood of AI crawlers scraping their sites and feeding the content into AI models.