]> Vivoh, Inc.

erik@vivoh.com

Media Over QUIC This document updates the MOQT Streaming Format (MSF) by defining a new optional feature for the streaming format. It specifies the syntax and semantics for adding Neural Video Codec (NVC) packaged media to MSF. NVC codecs use learned neural network transforms for video compression, and their bitstreams require a distinct packaging model from traditional block-based codecs. NMSF maps neural keyframes (Intra) and delta frames (Inter) onto MoQ Groups and Objects, and introduces a multi-track model that separates hyperprior side information from latent bitstreams for priority-aware delivery. This enables real-time neural video streaming over any standard MoQ relay. About This Document This note is to be removed before publishing as an RFC. Source for this draft and an issue tracker can be found at .

Introduction Neural Video Codecs (NVCs) represent a new class of video compression that uses learned neural network transforms instead of block-based motion estimation and discrete cosine transforms. NVCs such as DCVC-RT , SSF, FVC, and RLVC produce compressed bitstreams that differ fundamentally from traditional codecs:

• No container format. NVC bitstreams consist of entropy-coded latent tensors, not fMP4 boxes, LOC containers, or NAL units.
• Two-layer compression. NVCs using hyperprior-based entropy coding (such as those built on ) produce two distinct bitstreams per frame: a small hyperprior containing statistical side information, and a larger latent tensor containing the compressed frame data. The latent cannot be decoded without the hyperprior.
• Stateful decoding. The decoder maintains a learned context buffer (analogous to a decoded picture buffer) that must be initialized from a full neural keyframe before delta frames can be decoded.
• Variable-rate representations. Compressed frame sizes vary significantly based on scene complexity, as the codec allocates bits adaptively in a learned feature space.

The existing MSF packaging types -- LOC and the timeline types -- do not accommodate these bitstreams. The CMSF extension adds CMAF packaging for traditional block-based codecs. Neither is suitable for NVC data, which has no container structure and requires a different model for keyframe semantics and decoder state management. This document defines NMSF, an MSF extension that adds NVC packaging. NMSF follows the same extension pattern established by CMSF: it registers a new "packaging" value, defines the Object payload format, and specifies Group-level requirements for decoder random access. Additionally, NMSF introduces a multi-track model that maps the hyperprior and latent bitstreams to separate MoQ tracks, enabling priority-based relay delivery under congestion. A single MoQ Broadcast MAY contain tracks using any combination of packaging types. For example, an NVC video track may coexist with a LOC or CMAF audio track in the same catalog.

MSF Extension All of the specifications, requirements, and terminology defined in apply to implementations of this extension unless explicitly noted otherwise in this document.

NVC Packaging

Track Model NVC packaging uses two MoQ tracks per video stream to carry the two layers of the neural codec's compressed output:

Hyperprior track:

Carries the entropy-coded hyperprior (side information) for each frame. The hyperprior describes the statistical distribution of the latent tensor and is required by the entropy decoder before the latent can be decompressed. This track is small (typically 5-15% of total bitstream) and SHOULD be assigned higher delivery priority than the latent track.

Latent track:

Carries the entropy-coded latent tensor (the primary compressed frame data) for each frame. The latent track is larger and depends on the corresponding hyperprior Object having been received and decoded first.

Both tracks MUST use "nvc" packaging and belong to the same MoQ Broadcast. The tracks are linked via a "depends" field in the catalog (). Objects in the two tracks are correlated by Group sequence number and Object index: Object N in Group G of the hyperprior track corresponds to Object N in Group G of the latent track. This two-track model reflects the inherent two-layer structure of hyperprior-based neural codecs. It enables MoQ relays to prioritize the small hyperprior track under congestion, ensuring that the decoder can begin latent decompression as soon as latent data arrives. The relay requires no awareness of the NVC payload format -- standard MoQ priority mechanisms are sufficient. Implementations MAY use a single combined track instead of two separate tracks. In this case, the payload sub-format defined in carries both bitstreams within each Object. The single-track mode is simpler but loses the ability to prioritize hyperprior delivery independently.

Object Wire Format Each MoQ Object payload for a track with "nvc" packaging consists of a fixed 26-byte header followed by a variable-length compressed bitstream:

frame_type:

1 byte. The type of neural video frame contained in this Object. See .

qp:

1 byte, unsigned 8-bit integer. The quality parameter index used for encoding this frame. Identifies the rate-distortion operating point from the codec's trained quality levels. The decoder MUST use the same QP index to select matching quantization step sizes. Values 0-63 are defined; values 64-255 are reserved.

frame_number:

4 bytes, unsigned 32-bit integer, big-endian. The absolute sequence number of this frame within the stream, starting from zero.

pts_ms:

8 bytes, unsigned 64-bit integer, big-endian. The capture-side wallclock timestamp in milliseconds since the Unix epoch (1970-01-01T00:00:00Z). This timestamp is set by the publisher at the time of frame capture and is carried through the relay unmodified. Subscribers MAY use the difference between pts_ms and their local wallclock to estimate end-to-end latency. A value of zero indicates that no timestamp is available.

width:

4 bytes, unsigned 32-bit integer, big-endian. The frame width in pixels.

height:

4 bytes, unsigned 32-bit integer, big-endian. The frame height in pixels.

payload_len:

4 bytes, unsigned 32-bit integer, big-endian. The byte length of the payload field that follows.

payload:

Variable length. The NVC compressed bitstream for this frame. In two-track mode, the hyperprior track carries the hyperprior bitstream and the latent track carries the latent bitstream. In single-track mode, this field carries the combined payload defined in .

Frame Types The frame_type field identifies the role of the frame in the neural codec's prediction structure:

Intra frames are analogous to SAP Type 1 access points in CMAF. They enable random access by fully initializing the decoder's context buffer without dependence on any prior frame. Inter frames are analogous to non-SAP frames (P-frames). They depend on the decoder's context buffer, which contains the reconstructed output of the immediately preceding frame. Unlike traditional codecs with multi-frame reference picture buffers, current NVCs maintain a single context buffer containing learned features from the previous reconstructed frame. The reserved range (0x02-0xFF) accommodates future NVCs that may introduce bidirectional prediction, hierarchical quality layers, or multi-reference architectures.

Object Packaging The payload of each Object is subject to the following requirements:

• MUST contain exactly one NVC frame (or frame component, in two-track mode) in the wire format defined in .
• MUST NOT span multiple frames. Each frame is carried in a separate Object per track.
• Objects within a Group MUST be sequentially ordered by frame_number. Out-of-order processing causes encoder-decoder context buffer divergence.
• In two-track mode, the hyperprior Object and the corresponding latent Object for the same frame MUST have identical frame_type, qp, frame_number, pts_ms, width, and height header values.

Group Packaging Each MOQT Group:

• MUST begin with an Object containing an Intra frame (frame_type = 0x00).
• MUST contain one contiguous Group of Pictures (GOP): one Intra frame followed by zero or more Inter frames.
• The Group boundary aligns with the publisher's neural GOP boundary. Typical GOP sizes are 30-120 frames (1-4 seconds at 30 fps).
• In two-track mode, both the hyperprior and latent tracks MUST use the same Group sequence numbers and contain the same number of Objects per Group.

This structure ensures that a subscriber joining mid-stream or recovering from loss can begin decoding from the next Group boundary. The Intra frame at the start of each Group fully initializes the decoder context buffer, enabling immediate playback without waiting for a future keyframe.

Payload Format In two-track mode, each track's payload contains a single bitstream component:

• Hyperprior track payload: the entropy-coded hyperprior tensor z, preceded by tensor shape metadata.
• Latent track payload: the entropy-coded latent tensor y, preceded by tensor shape metadata.

Each payload component uses this sub-format: The channels, height, and width fields describe the spatial dimensions of the tensor prior to entropy coding. These are required by the decoder to allocate output buffers and configure the entropy decoder. In single-track mode, the payload carries both components concatenated: the hyperprior component followed immediately by the latent component, using the same sub-format for each.

Catalog: NVC Packaging Type This specification extends the allowed packaging values defined in to include one new entry:

Every Track entry in an MSF catalog carrying NVC-packaged media data MUST declare a "packaging" type value of "nvc".

Catalog: NVC Track Fields This specification adds the following track-level catalog fields for tracks with "nvc" packaging:

The standard MSF track fields "name", "packaging", "isLive", "width", "height", and "framerate" retain their definitions from and are REQUIRED for NVC tracks.

Catalog: NVC Metadata Object The optional "nvc" object within a track catalog entry carries codec-specific metadata that a subscriber may need to configure its decoder:

Subscribers that do not recognize the "codec" value or cannot satisfy the metadata requirements SHOULD NOT subscribe to the track.

Decoder Requirements This section specifies the behavior required of an NMSF decoder. These requirements ensure that encoder and decoder context buffers remain synchronized, preventing visual artifacts caused by state drift.

Context Buffer Management The decoder MUST maintain a context buffer containing the reconstructed output of the most recently decoded frame. This buffer is used as input to the synthesis transform when decoding Inter frames. The context buffer is uninitialized when the decoder starts. The decoder MUST NOT attempt to decode Inter frames until it has successfully decoded an Intra frame. After decoding each frame (Intra or Inter), the decoder MUST replace its context buffer with the newly reconstructed output. Failure to update the context buffer causes progressive drift between encoder and decoder state, manifesting as visual artifacts commonly described as "ghosting" or "smearing."

Two-Track Decode Sequence When operating in two-track mode, the decoder MUST process frames in the following order for each frame N in Group G:

1. Receive Object N from the hyperprior track (Group G).
2. Decode the hyperprior to obtain CDF parameters for the entropy decoder.
3. Receive Object N from the latent track (Group G).
4. Entropy-decode the latent tensor using the CDF parameters from step 2.
5. Run the synthesis transform (Intra or Inter path) to produce the reconstructed frame.
6. Update the context buffer with the reconstructed frame.

The decoder MAY begin receiving the latent Object before the hyperprior decode is complete, but MUST NOT begin entropy decoding the latent until the hyperprior CDF parameters are available.

Intra Frame Handling When the decoder receives an Intra frame (frame_type = 0x00):

1. Discard any existing context buffer.
2. Decode the frame using only the payload data (no context).
3. Store the reconstructed output as the new context buffer.
4. Output the reconstructed frame for display.

Inter Frame Handling When the decoder receives an Inter frame (frame_type = 0x01):

1. Verify that a context buffer exists (i.e., an Intra frame has been previously decoded). If not, discard the frame.
2. Decode the frame using the payload AND the current context buffer.
3. Replace the context buffer with the newly reconstructed output.
4. Output the reconstructed frame for display.

Stream Join and Recovery When a subscriber joins a stream mid-session or recovers from packet loss:

1. Wait for the start of the next MoQ Group.
2. The first Object in the Group is an Intra frame.
3. Decode the Intra frame to initialize the context buffer.
4. Continue decoding subsequent Inter frames normally.

Objects received before the first Intra frame MUST be discarded. In two-track mode, the subscriber MUST subscribe to both the hyperprior and latent tracks. Subscribing to the latent track alone is insufficient, as the latent cannot be decoded without the hyperprior CDF parameters.

Encoder Context Synchronization The encoder MUST maintain its own context buffer that mirrors the decoder's state. After encoding each frame, the encoder MUST run the synthesis (decoding) transform on the quantized latent representation to produce a reconstructed frame, and use that reconstructed frame as the context for encoding the next frame. This "encode-decode loop" ensures that the encoder's context buffer contains exactly the same data that a decoder would produce from the transmitted bitstream, preventing encoder-decoder state drift.

Codec Registration The "codec" field in the catalog identifies which NVC is used:

New NVC codecs are compatible with NMSF packaging if they produce distinct Intra and Inter frame types and use a single sequential context buffer. Registration requires choosing a unique codec identifier string and documenting the payload sub-format (). No changes to the wire format header () are required for new codecs.

Catalog Examples The following section provides non-normative JSON examples of catalogs compliant with this draft.

Two-track NVC video with LOC audio This example shows a catalog for a live broadcast with DCVC-RT neural video using the two-track model (hyperprior + latent) and one Opus audio track using MSF's native LOC packaging.

Single-track NVC video (compact mode) This example shows a catalog using single-track mode, where the hyperprior and latent are combined in a single payload. This mode is simpler but does not support independent priority control.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Security Considerations NMSF relies on the security properties of MoQ Transport , which provides confidentiality and integrity via QUIC's TLS 1.3 encryption. NMSF does not add its own integrity or authentication mechanisms. The "payload_len" field permits payloads up to 4 GiB. Decoders SHOULD enforce a maximum payload size appropriate for their deployment environment (e.g., 100 MiB for 4K video) and reject Objects exceeding that limit to mitigate resource exhaustion. A malicious publisher could craft Intra frames that cause the decoder's context buffer to enter a state producing misleading visual output on subsequent Inter frames. This is analogous to reference picture manipulation in traditional codecs and is mitigated by the same trust model: subscribers SHOULD only connect to authenticated and authorized publishers. Neural video codec model weights are typically large (tens to hundreds of megabytes) and are NOT transmitted via MoQ. Both publisher and subscriber must have compatible model weights pre-installed. The "nvc" catalog metadata () enables version negotiation, but the secure distribution of model weights is outside the scope of this document. The pts_ms timestamp reveals the publisher's wallclock time, which may be a privacy concern in some deployments. Publishers MAY set pts_ms to zero to suppress timestamp information.

IANA Considerations This document requests registration of a new packaging type value "nvc" in the MSF packaging registry defined by . This document requests creation of an "NVC Codec Identifiers" registry with the initial values defined in . New entries require Specification Required registration policy.

References Normative References Akamai This document specifies the MOQT Streaming Format, designed to operate on Media Over QUIC Transport. Akamai This document updates [MSF] by defining a new optional feature for the streaming format. It specifies the syntax and semantics for adding CMAF-packaged media [CMAF] to MSF. Cisco Google Google Meta This document defines the core behavior for Media over QUIC Transport (MOQT), a media transport protocol designed to operate over QUIC and WebTransport, which have similar functionality. MOQT allows a producer of media to publish data and have it consumed via subscription by a multiplicity of endpoints. It supports intermediate content distribution networks and is designed for high scale and low latency distribution. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References Microsoft Research InterDigital ITU-R

Acknowledgments The author would like to thank Will Law for the MSF and CMSF specifications which established the extension pattern that NMSF follows, and the MoQ working group for the transport protocol that makes this work possible.

Changes from draft-herz-moq-nmsf-00

• Added two-track model: separate hyperprior and latent MoQ tracks with dependency and priority signaling (Section 3.1).
• Added per-frame PTS timestamp (pts_ms) to the wire format header for end-to-end latency measurement (Section 3.2).
• Added per-frame quality parameter (qp) to the wire format header for rate-distortion signaling (Section 3.2).
• Wire format header expanded from 17 bytes to 26 bytes.
• Added nvcRole, depends, and priority catalog fields for two-track mode (Section 3.8).
• Added two-track decode sequence to decoder requirements (Section 4.2).
• Retained single-track mode as a simpler alternative.
• Added privacy consideration for pts_ms timestamp (Section 8).
• Normalized payload sub-format with explicit tensor shape metadata preceding each bitstream component (Section 3.6).