]> Cisco

mzanaty@cisco.com

Cisco

snandaku@cisco.com

Microsoft

pthatcher@microsoft.com

Web and Internet Transport Media Over QUIC LOC MOQ MOQT QUIC media This specification describes a Low Overhead Media Container (LOC) format for encoded and encrypted audio and video media data to be used primarily for interactive Media over QUIC Transport (MOQT). It may be used in the WARP streaming specification, which defines a catalog format for publishers to annouce and describe their LOC tracks and for subscribers to consume them. Examples are also provided for building media applications using LOC and MOQT. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the Media Over QUIC Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction This specification describes a low-overhead media container (LOC) format for encoded and encrypted audio and video media data. "Low-overhead" refers to minimal extra encapsulation as well as minimal application overhead when interfacing with WebCodecs . The container format description is specified for all audio and video codecs defined in the WebCodecs Codec Registry . The audio and video payload bitstream is identical to the "internal data" inside an EncodedAudioChunk and EncodedVideoChunk, respectively, specified in the registry. (Note: Do we need to support timed text tracks such as Web Video Text Tracks (WebVTT) ?) In addition to the media payloads, critical metadata is also specified for audio and video payloads. (Note: Align with MOQT terminology of either "metadata" or "header".) A primary motivation is to align with media formats used in WebCodecs to minimize extra encapsulation and application overhead when interfacing with WebCodecs. Other container formats like CMAF or RTP would require more extensive application overhead in format conversions, as well as larger encapsultion overhead which may burden some use cases like low bitrate audio scenarios. This specification can also be used by applications outside the context of WebCodecs or a web browser. While the media payloads are defined by referring to the "internal data" of an EncodedAudioChunk or EncodedVideoChunk in the WebCodecs Codec Registry, this "internal data" is the elementary bitstream format of codecs without any encapsulation. Referring to the WebCodecs Codec Registry avoids duplicating it in an identical IANA registry.

• defines the core media payload formats.
• defines the metadata associated with audio and video payloads.
• defines the usage of end-to-end encrypted LOC payloads.
• provides examples with details for building audio and video applications using LOC over MOQ.

Requirements Notation and Conventions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Terminology Track, Group, Subgroup, Object, and their corresponding identifiers (ID or alias) are defined in and used here to refer to those aspects of the MOQT Object Model.

Payload Format The WebCodecs Codec Registry defines the contents of an EncodedAudioChunk and EncodedVideoChunk for the audio and video codec formats in the registry. The "internal data" in these chunks is used directly in this specification as the "LOC Payload" bitstream. This "internal data" is the elementary bitstream format of each codec without any encapsulation.

Video Payload Format For video formats with multiple bitstream formats in the WebCodecs Registry, such as H.264/AVC or H.265/HEVC, the LOC Payload can use either the "canonical" format ("avc" or "hevc") often used in storage containers like MP4 / ISO BMFF, or the "annexB" format used in some non-MP4 applications. These formats differ in how they carry initialization and configuration information called parameter sets as well as how parts of a video frame are delimited with length prefixes or start codes.

Parameter Sets In Payload Parameter sets can be sent in the bitstream payload before key frames, similar to "annexB" formats. Newer "canonical" formats such as "avc3" and "hev1" codec strings also support parameter sets in the bitstream payload or outside it in "extradata" metadata headers.

Parameter Sets in Headers Parameter sets can be sent in headers before key frames, as described in the Config LOC Header Extension , similar to the original "canonical" formats such as "avc1" and "hvc1" codec strings. The Config contents are the "extradata" bytes defined by the corresponding codec specification, which map to the WebCodecs VideoDecoderConfig description property in the EncodedVideoChunkMetadata.

Length Prefixes in Payload A 4-byte length prefix can be sent before each NAL Unit, similar to "canonical" ("avc" or "hevc") formats. A length value of 1 should be interpreted as a start code rather than a length. The length is in network byte order, i.e. big endian, and SHOULD be 4 bytes long to disambiguate from start code prefixes. A length prefix less than 4 bytes long, which is uncommon, MAY be specified in the Config .

Start Code Prefixes in Payload A 4-byte start code can be sent before each NAL Unit, similar to "annexB" formats. The start code value is 1 in network byte order, i.e. big endian, and SHOULD be 4 bytes long to disambiguate from length prefixes. A 3-byte start code, which is uncommon, MAY be used if the track never uses length prefixes or any Config .

MOQ Object Mapping An application object when transported as a object is composed of a MOQ Object Header, with optional Extensions, and a Payload. Media objects encoded using the container format defined in this specification populate the MOQ Object Payload with the LOC Payload, and the MOQ Object Header Extensions with the LOC Header Extensions, as shown below. The LOC Payload is the "internal data" of an EncodedAudioChunk or EncodedVideoChunk. The LOC Header Extensions carry optional metadata related to the Payload. +----------+--------------+-----------+ | MOQ | MOQ Header | MOQ | | Header | Extensions | Payload | +----------+--------------+-----------+ | | | | +--------------+-----------+ | LOC Header | LOC | | Extensions | Payload | +--------------+-----------+ LOC Header Extensions = some MOQ Object Header Extensions LOC Payload = all MOQ Object Payload LOC Payload = "internal data" of EncodedAudio/VideoChunk ]]>

LOC Header Extensions The LOC Header Extensions carry optional metadata for the corresponding LOC Payload. The LOC Header Extensions are contained within the MOQ Object Header Extensions. This metadata provides necessary information for end subscribers, relays and other intermediaries to perform their operations without accessing the media payload. For example, media switches can use this metadata to perform their media switching decisions without accessing the payload which may be encrypted end-to-end (from original publisher to end subscribers). The following sections define specific metadata as LOC Header Extensions and register them in the IANA registry for MOQ Object Header Extensions. Other specifications can define other metadata as LOC Header Extensions and register them in the same registry. Each extension must specify the following information in the IANA registry.

• Name: Short name for the metadata (not sent on the wire)
• Description: Detailed description (not sent on the wire)
• ID: Identifier assigned by the registry (varint)
• Length: Length of metadata Value in bytes (varint if ID is odd, omitted if ID is even)
• Value: Value of metadata (varint if ID is even, Length bytes if ID is odd)

Common Header Data

Capture Timestamp

• Name: Capture Timestamp
• Description: Wall-clock time in microseconds since the Unix epoch when the encoded media frame was captured, encoded as a varint.
• ID: 2 (IANA, please assign from the MOQ Header Extensions Registry)
• Length: Varies (1-8 bytes)
• Value: Varies

Video Header Data

Video Config

• Name: Video Config
• Description: Video codec configuration "extradata", as defined by the corresponding codec specification, which maps to the WebCodecs VideoDecoderConfig description property in the EncodedVideoChunkMetadata.
• ID: 13 (IANA, please assign from the MOQ Header Extensions Registry)
• Length: Varies
• Value: Varies

Video Frame Marking

• Name: Video Frame Marking
• Description: Flags for video frames which are independent, discardable, or base layer sync points, as well as temporal and spatial layer identification, as defined in , encoded in the least significant bits of a varint.
• ID: 4 (IANA, please assign from the MOQ Header Extensions Registry)
• Length: Varies (1-4 bytes)
• Value: Varies

Audio Header Data

Audio Level

• Name: Audio Level
• Description: The magnitude of the audio level of the corresponding audio frame as well as a voice activity indicator as defined in section 3 of , encoded in the least significant 8 bits of a varint.
• ID: 6 (IANA, please assign from the MOQ Header Extensions Registry)
• Length: Varies (1-2 bytes)
• Value: Varies

Payload Encryption When end to end encryption is supported, the encoded payload is encrypted with symmetric keys derived from key establishment mechanisms, such as , and the payload itself is protected using mechanisms defined in .

Examples This section provides examples with details for building audio and video applications using MOQ and LOC; more specifically, it provides information on:

• Using a WARP catalog to describe track information,
• Packaging media into LOC streaming format, and
• Mapping application media objects to the MOQT object model and transport.

The figure below shows the conceptual model for mapping media application data to the MOQT object model and underlying QUIC transport.

Application with one audio track An example is shown below for an Opus mono channel audio track at 48Khz. When ready for publishing, each encoded audio chunk, say 10ms, represents a MOQT Object. In this setup, there is one MOQT Object per MOQT Group, where the GroupID in the object header is increment by one for each encoded audio chunk and the ObjectID is defaulted to value 0. These objects can be sent as QUIC streams or datagrams. When mapped to QUIC datagrams, each object must fit entirely within a QUIC datagram, and when mapped to QUIC Streams, each such unitary group is sent over an individual unidirectional QUIC stream since there is just one SubGroup per each MOQT Group.

Application with one single quality video track An example is shown below for an H.264 video track with 1280x720p resolution and 30 fps frame rate at 1 Mbps bitrate. When ready for publishing, each encoded video chunk is considered as input to MOQT Object payload. If encrypted, the output of encryption will serve as the object's payload. The GroupID is incremented by 1 at IDR Frame boundaries. The ObjectID is increment by 1 for each encoded video frame, starting at 0 and resetting to 0 at the start of a new group. The first encoded video frame, MOQT Object with ObjectID 0, shall be the Independent (IDR) frame and the rest of the encoded video frames corresponds to dependent (delta) frames, organized in the decode order. When mapping to QUIC for sending, one unidirectional QUIC stream is setup to deliver all the encoded video chunks within a MOQT group. When decoding at the 'End Consumer', the objects from each of the QUIC streams are fed in the GroupID then ObjectID order to the decoder for the track.

Application with single video track with temporal layers An example is shown below for an H.264 video track with 1280x720p resolution and 2 temporal layers at 30 fps and 60 fps frame rate. When ready for publishing, each encoded video chunk is considered as input to MOQT Object payload. If encrypted, the output of encryption will serve as the object's payload. The GroupID is incremented by 1 at Independent (IDR) frame boundaries. Each MOQT group shall contain 2 SubGroups corresponding to the 2 temporal layers as shown below: Within the MOQT group, ObjectID is increment by 1 for each encoded video frame, starting at 0 and resetting to 0 at the start of a new group. The first encoded video frame, MOQT Object with ObjectID 0, shall be the Indepedent (IDR) frame and the rest of the encoded video frames corresponds to dependent (delta) frames, organized in the decode order. When mapping to QUIC for sending, one unidirectional QUIC stream is used per SubGroup, thus resulting in 2 QUIC streams per MOQT group. When decoding at the 'End Consumer' for a given MOQT group, the objects must be fed in the GroupID then ObjectID order. This implies that the consumer media application needs to order objects across the SubGroup QUIC streams.

Application with mutiple dependent video tracks An example is shown below for an H.264 video track with 2 spatial qualities at 360p and 720p each at 30 fps When ready for publishing, the mapping to the MOQT object model and to underlying QUIC, follows the same procedures as described in for each video track. When decoding at the 'End Consumer' for a given MOQT group, the objects must be fed in the GroupID then ObjectID order in the ascending quality track order. For the example in the section, this would imply following pattern when decoding group 5.

Application with mutiple dependent video tracks with dyadic framerate levels. An example is shown below for an H.264 video track with 2 spatial qualities at 360p and 720p, however, the framerate between tracks vary dyadically. When ready for publishing, the mapping to the MOQT object model and to underlying QUIC, follows the same procedures as described in for each video track. When decoding at the 'End Consumer' for a given MOQT group, the objects from across the tracks must be fed in the timestamp order to the decoder, if no frame reordering is present in the encoding. If the encoding uses frame reordering, or if timestamp cannot be obtained, the object to choose next shall follow the below formula.

Application with multiple simulcast qualities video tracks An example is shown below for an H.264 video track with 2 simulcast spatial qualities at 360p and 720p each at 30 fps. When ready for publishing, the mapping to the MOQT object model and to underlying QUIC, follows the same procedures as described in for each video track. When decoding at the 'End Consumer', the objects from the QUIC stream are fed in the GroupID then ObjectID order to the decoders setup for the corresponding video tracks.

Security and Privacy Considerations The metadata in LOC Header Extensions is visible to relays, since the MOQ Object Header Extensions are often not encrypted end-to-end (from original publisher to end subscribers) in common schemes. In some cases, this may be an intentional design intent for proper relay operation. In other cases, this may be unintentional or undesirable leaking of the metadata to relays. Each metadata that is defined should consider the security and privacy aspects of granting relays visibility to the metadata. End-to-end encyption schemes should support end-to-end encryption of sensitive metadata. The metadata defined and registered in this specification (Capture Timestamp, Video Frame Marking, and Audio Level) may be sensitive metadata that should be encrypted end-to-end. They are used by media switches, which are not merely relays, and likely have access to some media keys. This may require end-to-end encryption schemes with multiple different security key contexts for payload versus metadata.

IANA Considerations The IANA registry for MOQ Object Header Extensions is populated with the entries specified in section , referencing this specification. This document creates a new entry in the "MoQ Streaming Format" Registry (see Sect 8). The type value is 0x002, the name is "LOC Streaming Format" and the RFC is XXX.

References Normative References Cisco Google Google Meta This document defines the core behavior for Media over QUIC Transport (MOQT), a media transport protocol designed to operate over QUIC and WebTransport, which have similar functionality. MOQT allows a producer of media to publish data and have it consumed via subscription by a multiplicity of endpoints. It supports intermediate content distribution networks and is designed for high scale and low latency distribution. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This document describes a Video Frame Marking RTP header extension used to convey information about video frames that is critical for error recovery and packet forwarding in RTP middleboxes or network nodes. It is most useful when media is encrypted and essential when the middlebox or node has no access to the media decryption keys. It is also useful for codec-agnostic processing of encrypted or unencrypted media, while it also supports extensions for codec-specific information. This document defines a mechanism by which packets of Real-time Transport Protocol (RTP) audio streams can indicate, in an RTP header extension, the audio level of the audio sample carried in the RTP packet. In large conferences, this can reduce the load on an audio mixer or other middlebox that wants to forward only a few of the loudest audio streams, without requiring it to decode and measure every stream that is received. [STANDARDS-TRACK] Informative References Akamai Twitch Google Cisco Meta This document specifies the WARP Streaming Format, designed to operate on Media Over QUIC Transport. Cisco Cisco Cisco This document describes an end-to-end authenticated encryption scheme for application objects intended to be delivered over Media over QUIC Transport (MOQT). We reuse the SFrame scheme for authenticated encryption of media objects, while suppressing data that would be redundant between SFrame and MOQT, for an efficient on-the-wire representation. Cisco Cisco Cisco The Media over QUIC system allows relays to assist in the delivery of real-time media. While these relays are trusted to facilitate media delivery, they are not trusted to access the media content. The document describes an end-to-end security system that prevents relays from accessing media content. MLS is used to establish keys that are available only to legitimate participants in a session, which are then used to protect media data using SFrame.

Acknowledgements Thanks to Cullen Jennings for suggestions and review.