OpenAI announced on October 30, 2026 that it will honor a shared robots.txt signal for AI training preferences, letting website operators communicate one training preference to multiple participating AI providers at once. In its announcement, the company described the move as voluntary and said it aims to implement the signal in the coming months, with another update on implementation details to follow.

Until now, website operators had to identify individual crawlers and maintain separate rules for each provider. OpenAI said the shared signal reduces that work: “This voluntary effort makes it easier for publishers and website operators to communicate the same preference to multiple participating providers at once.”

How the Signal Works

The signal is defined in an individual Internet-Draft titled Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00), authored by Kevin Kelley of Anthropic and dated October 30, 2026. The IETF datatracker notes the draft is an individual submission with no formal standing in the IETF standards process, and it expires on May 3, 2027.

The draft introduces one cross-vendor policy token: __usage_ai-training. A site operator writes an ordinary robots.txt group keyed by that token in the User-Agent line. Its worked example pairs “User-Agent: __usage_ai-training” with “Disallow: /” and “Allow: /press-kit/”, expressing a preference against AI training for everything except the press-kit section. The draft stresses that it introduces no new directive names and no new grammar: the path language, longest-match precedence and group structure of the robots.txt standard published as RFC 9309 are unchanged, so existing parsers keep working as before and simply ignore policy-token groups as they would any unrecognized user agent.

Two details matter. First, wildcard groups (“User-Agent: *”) must not be read as matching the policy token, and if no group contains the token, no preference has been expressed. Second, the token is not a directive for crawling behavior: usage preferences and crawler traffic-management signals are independent, and private agreements may override preferences expressed through the mechanism.

Fact card, the __usage_ai-training signal at a glance

Who Is on Board

OpenAI framed the step as part of the IETF AI Preferences working group’s ongoing work, saying it joins other working-group participants in the announcement and will keep adapting its practices as standards discussions evolve. “We encourage other AI providers to honor the signal,” the company said. It added that the new signal will not replace its existing robots.txt controls: “This new signal will not replace the controls we already provide through robots.txt,” which will continue to apply and be respected.

Google announced the same day that it will voluntarily adopt the proposal. Search advocate John Mueller wrote that website owners will be able to use a shared “policy token” within robots.txt to set broad preferences across participating providers instead of managing separate instructions for every AI developer. Google plans to support the control in 2027, interpreting a training opt-out as equivalent to disallowing its Google-Extended crawler unless Google-Extended is explicitly allowed, while existing company-specific controls keep working alongside the joint signal.

The working group’s participants span OpenAI, Google, Microsoft, Meta, Cloudflare, the Common Crawl Foundation, News Corp, the Financial Times, the BBC, the Electronic Frontier Foundation and Creative Commons, though the IETF notes participants contribute as individuals.

Fact card, who is on board with the shared signal

Why It Matters

For publishers, the appeal is practical: one robots.txt entry instead of a growing list of per-provider rules. For the industry, the joint announcement by OpenAI and Google gives the draft its first real momentum, even though honoring it remains voluntary and no technical mechanism can force a non-cooperative scraper to comply. The signal also arrives as regulators push for simpler controls; the EU’s code of practice for general-purpose AI under the AI Act already recognizes robots.txt as a primary method for publishers to express preferences.

OpenAI said it will share implementation details when it rolls the signal out in the coming months. For sites deciding whether to adopt it, the immediate takeaway is narrower: the draft changes nothing about existing parsing, so adding a policy-token group is safe to experiment with today.

Sources