OpenAI, Google, Microsoft and Anthropic back single AI scraping opt-out

The latest:

Four of the largest AI developers — OpenAI, Google, Microsoft and Anthropic — say they will adopt a shared signal that lets publishers opt out of AI scraping across multiple crawlers at once, The Verge reported. The control would live inside a site’s robots.txt file and replace the current patchwork of company-by-company settings. Meta’s communications chief signaled support without committing.

Details:

• The mechanism: The proposal centers on a shared policy token placed in a website’s robots.txt file, according to The Verge. Robots.txt has long told web crawlers which parts of a site they may access, whether for search indexing, archiving or AI training. The new token extends that same file to carry AI training preferences readable by every participating company.
• The author: The signal was proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of work at the Internet Engineering Task Force, the body that develops core internet standards. Routing it through the IETF positions the control as an open standard rather than any single company’s policy.
• The commitments: OpenAI, Google, Microsoft and Anthropic each say their AI crawlers will stop accessing a site when they encounter the shared token. The four firms have not published a timeline for when the restriction takes effect across their crawler fleets, and no joint statement accompanied the proposal.
• Meta’s position: Meta communications head Andy Stone signaled support for the setting but did not say whether the company will actually honor it. Stone said a common signal conveying such preferences to multiple providers at once is an important step, and that Meta looks forward to working with other AI companies on implementation across the industry.
• The problem: The announcement lands as publishers contend with a continuing influx of AI crawlers scraping their sites and feeding the content into AI models, The Verge reported. Under the existing arrangement, a publisher wanting to be excluded from AI training has to issue a separate instruction to each AI company individually.
• Rival approaches: Other efforts have taken harder lines. The Really Simple Licensing standard gives publishers a way to charge crawlers that visit their sites rather than merely refuse them, while Cloudflare has moved to block AI crawlers by default for sites on its network. The new token instead relies on voluntary compliance.
• The limits: Robots.txt is an advisory file, not a technical barrier — it works only when a crawler operator chooses to respect it. The announcement covers the four named companies and does not bind other AI developers operating crawlers, and no enforcement mechanism was described for firms that ignore the token.
• The reporting trail: The adoption of the shared standard was reported earlier by Search Engine Roundtable, before The Verge detailed the companies’ commitments and noted that Microsoft’s position was being set out further.

Background:

Robots.txt dates to the early web as a voluntary convention for site owners to guide search crawlers. Its reuse as an AI training control reflects the absence of any purpose-built mechanism for publishers to govern how their content feeds machine-learning models.

Between the lines:

The four firms backing the token are the same ones whose crawlers publishers most want to limit, which makes the standard an act of self-regulation as much as a concession. Meta’s support without a commitment to honor the signal shows the gap between endorsing a standard and obeying it. Because robots.txt carries no enforcement, the token’s value rests entirely on whether a crawler operator chooses compliance.

What’s next

Watch whether the IETF advances the policy token proposal toward a formal standard, whether Meta commits to honoring the signal, and whether the four companies publish activation dates for their crawlers.