IETF Eyes New Web Standards for AI Content Control

The open web has long faced challenges with AI models scraping and using content without creators’ consent, leading to a “data free-for-all.” Previous efforts like llms.txt lacked widespread adoption or enforcement, with Google explicitly not supporting them.

To address this, the Internet Engineering Task Force (IETF) launched the AI Preferences Working Group. This group, comprising leaders from Google, Microsoft, and Meta, aims to standardize machine-readable rules for how AI systems can access and utilize web content, potentially integrating these into robots.txt. The IETF, responsible for core internet protocols, is now defining standards for the AI era, focusing on allowing content owners to express preferences for AI model development, deployment, and use.

Early documents from the group propose updates to the Robots Exclusion Protocol, defining new rules and vocabulary. AI systems would be categorized with labels such as `search`, `train-ai` (general AI training), `train-genai` (generative AI training), and `bots`. Site owners can specify `y` (allow) or `n` (disallow) for each category. These rules can be set at folder levels and customized for different bots using a new `Content-Usage` field within robots.txt. An example shows `Content-Usage: train-ai=n` to broadly disallow AI training, while `Content-Usage: /ai-ok/ train-ai=y` permits it for a specific subfolder.

This initiative promises significant benefits by providing site owners with clear, standardized, and potentially enforceable control over their content’s AI usage. The involvement of major tech companies, particularly Google’s Gary Illyes, fosters optimism that these new robots.txt standards will gain wide adoption and be respected by AI crawlers, transforming content governance in the AI landscape.

(Source: https://searchengineland.com/robots-exclusions-new-rules-definitions-ietf-464990)