]> Google

garyillyes@google.com

Mozilla

mt@lowentropy.net

WIT AI Preferences skynet training wheel Content creators and other stakeholders might wish to signal their preferences about how their content might be consumed by automated systems. This document defines how preferences can be signaled as part of the acquisition of content in HTTP. This document updates RFC 9309 to allow for the inclusion of usage preferences. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the AI Preferences Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction The automated consumption of content by crawlers and other machines has increased significantly in recent years. This is partly due to the training of machine-learning models. Content creators and other stakeholders, such as distributors, might wish to express a preference regarding the types of usage they consider acceptable. Entities that might use that content need those preferences to be expressed in a way that is easily consumed by an automated system. This document describes two mechanisms for associating preferences with content:

• A Content-Usage header field for HTTP ; see .
• A Content-Usage directive for the Robots Exclusion Protocol (colloquially known as "robots.txt") ; see .

For automated systems that use HTTP to gather content, these allow for the automated gathering of preferences in the same way that content is obtained.

Preference Expressions The format of preference expressions is defined in the preference vocabulary . The preference vocabulary defines:

• what preferences can be expressed,
• how multiple expressions of preference are combined, and
• how those preferences are turned into strings or byte sequences for use in a protocol.

This document only defines how the strings or byte sequences are conveyed so that the preferences can be associated with content.

Examples A server that provides content using HTTP could signal preferences about how that content is used with the Content-Usage header field as follows: Alternatively, or additionally, a server might include the same directive in its "robots.txt" file:

Embedded Preferences This document does not define a means of embedding preferences in content. Embedding preferences is expected to be an effective means of associating preferences with content, because it ensures that metadata is always associated with content. The main challenge with embedding is that a different method is needed for each content type. That is, a different means of conveying preferences needs to be defined for each audio, documents, images, video, or other content format. Furthermore, some content types, such as plain text (text/plain), offer no universal means of carrying metadata. Though preferences might still be embedded in content with these formats, those preferences would not be reliably accessible to an automated system. The mechanisms in this document are therefore universal, in the sense that they apply to any content type. They are not universal in that they rely on the content being obtained using HTTP (and maybe FTP). Future work might define how preferences might be indicated for alternative content distribution or acquisition methods, such as email.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

HTTP Content-Usage Header Field The Content-Usage field is a structured field dictionary, as defined in . This field follows the vocabulary and processing rules in . This field indicates usage preferences regarding the content of the HTTP message. That is, the representation data, as defined in , not the resource. Servers MUST retain any preferences associated with a request if the content of that request is used to answer later requests. For example, the content of a PUT request that is used to answer subsequent GET requests. Note that servers that have not been updated to understand this field will not comply with this requirement. The Content-Usage field does not have any special effect on caching.

Robots Exclusion Protocol Content-Usage Directive A Content-Usage directive is added to the Group definition in the Robots Exclusion Protocol format . That is, the ABNF is extended as follows: ]]> This directive updates the definition of a group to be more expansive. Where a group was previously a set of user-agents (either "*" or a set of one or more identifiers), a Group is updated to include zero or one Content-Usage preferences.

Processing Multiple Groups The effect of this change is that a crawler might need to consider multiple groups. A crawler needs to consider this both to decide whether content can be requested and to determine what preferences apply to content. Rather than looking for a group based on a specific User-Agent identifier, such as "ExampleBot", then falling back to the wildcard group ("*"), a crawler might have multiple groups, each with a different set of preferences. Where there are multiple groups, a crawler first looks for groups with a matching User-Agent identifer. If any groups match the crawler identity (as defined in ), all matching groups are considered. If there are no matching groups, all groups that include a User-Agent of "*" are considered. In determining which group applies for a given resource, the crawler evaluates each group in turn. Any group for which the resource is disallowed (as defined in ) is excluded. If all groups are excluded in this way, the resource is not crawled. If any group allows the crawling of the resource, content can be retrieved. If multiple groups allow crawling, the usage preference from the group with the longest Allow rule match applies to that content. For example, given the following "robots.txt" document: A crawler that identifies as "ExampleBot" would be able to obtain all content and apply preferences of "ai=y" (processed as defined in ). All other crawlers would use the same two groups. The first group allows the retrieval of most resources, excluding resources starting with "/never/", and applies a usage preference of "ai=n" across those resources. The second group creates a specific rule for resources under "/ai-ok", where the usage preference is "ai=y". This might result in the following outcome after crawling:

Path	Allowed	Saved Preference
/test	yes	ai=n
/never/test	no	n/a
/ai-ok/test	yes	ai=y

Security Considerations TODO Security

IANA Considerations TODO request registration of field

Normative References This document describes a set of data types and associated algorithms that are intended to make it easier and safer to define and handle HTTP header and trailer fields, known as "Structured Fields", "Structured Headers", or "Structured Trailers". It is intended for use by specifications of new HTTP fields. This document obsoletes RFC 8941. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230. This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching. Open Future Mozilla This document proposes a standardized vocabulary of use cases that can be targeted when expressing machine-readable opt-outs related to Text and Data Mining (TDM) and AI training. The vocabulary is agnostic to specific opt-out mechanisms and enables declaring parties to communicate restrictions or permissions regarding the use of their digital assets in a structured and interoperable manner. It defines three key use cases—TDM, AI Training, and Generative AI Training—which can be referenced by opt-out systems to ensure consistent interpretation across different implementations. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings.

Acknowledgments TODO acknowledge.