OpenAI joins IETF group to launch shared AI crawler signal
OpenAI and other participants in an IETF working group announced a new shared signal to simplify how website operators communicate AI training preferences.
Website operators previously had to identify individual crawlers and maintain separate rules for each AI provider. To address this complexity, OpenAI joined other participants in the IETF working group to announce a new shared signal today. This common signal aims to reduce the work required for publishers to communicate their preferences to multiple providers simultaneously.
OpenAI plans to implement the new common signal in the coming months. The organization states that this development is a practical step toward an open web where website owners have simpler ways to communicate content usage choices. OpenAI encourages other AI providers to honor the new common signal as well.
The new common signal will not replace existing controls provided through robots.txt. It serves as an additional mechanism for expressing preferences rather than a complete overhaul of current standards.
Sources
Every sentence of this story was checked against facts quoted from these articles. How we make the news
Spotted an error? Tell us