Anthropic and Microsoft join the shared robots.txt opt-out from AI training
Anthropic and Microsoft have joined Google and OpenAI in backing one shared robots.txt signal that lets a website tell several AI companies at once not to train on its content, Nieman Lab reports. The signal was proposed by Anthropic engineer Kevin Kelley as part of work at the Internet Engineering Task Force (IETF), and Anthropic is backing it. Timing differs by company: OpenAI says it will support it in the coming months, while Google and Microsoft say next year. We covered Google and OpenAI's plans on 31 October; new here are the two extra companies and their timelines, and Meta's communications chief Andy Stone called the signal "an important step".
Why it matters: If you publish content and want it kept out of AI training, this should become one line in robots.txt instead of a list of crawler names. Keep your existing crawler-specific blocks until each company you care about says its support is live.
Nieman Lab ↗AI-assisted, reviewed by Sorted AI
▲ 0
0 comments