]> Technical University Munich

mathis.engelbart@tum.de

Technical University Munich

ott@in.tum.de

Nokia Technologies

lukasz.kondrad@nokia.com

Web and Internet Transport Audio/Video Transport Core Maintenance RTP RTCP ROI Point Cloud This document specifies RTCP messages and RTP header extensions for exchanging parameters of real-time streamed point clouds. A sender can notify receivers of the currently applied parameters, such as selected regions, and their parameters, such as the respective resolutions and included point attributes. A receiver can request updates to the same parameters using RTCP feedback messages. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the Audio/Video Transport Core Maintenance Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction A point cloud is a set of data points in a three-dimensional coordinate system where each point is defined by its three coordinates. Point clouds can represent three-dimensional environments, such as a vehicle's surroundings. Each point in a point cloud may optionally be associated with additional attributes. Attributes can, for example, be colors or reflectance of objects in the scene. Sequences of point clouds can be generated by sensors such as Lidar ("light detection and ranging"), Radar ("radio detection and ranging"), or a multi-camera setup. Due to the high number of points in a scene, the bandwidth requirements of transmitting point clouds over a network are often higher than those of streaming video. In video streaming, efficient codecs are used to reduce the bandwidth requirement. Similar codecs are being developed for point clouds. However, when streaming point cloud data, consumers of the data usually aren't equally interested in each point. For further processing, focusing on some regions of a point cloud stream is often sufficient, allowing the producer of a point cloud sequence to filter out points of lower interest to further reduce the bandwidth requirements and prioritize bandwidth usage for points of higher importance. When selecting the regions to prioritize and deciding which points can be excluded from transmission, producers and consumers need a mechanism to signal a) which regions of a scene are currently being prioritized and b) request updates to prioritize different regions in the future. This document describes such a mechanism for real-time transmission of point clouds building on the RTP Control protocol RTCP, which is part of the Real-time Transport Protocol RTP. Additionally, this document provides RTP header extensions for senders to inform receivers about the currently applied parameters. The information might be useful for receivers in determining if an RTCP message requesting an update was received and acted upon by the sender. In , this document first defines a set of shared encoding schemes to represent point cloud parameters. and define RTCP messages and RTP header extensions, respectively. The RTCP messages and RTP header extensions reference the encoding scheme in . defines the SDP parameters required to negotiate the usage of the presented RTCP messages and header extensions.

Conventions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Definitions and Abbreviations

• TODO: Add definitions or remove section

Region and Parameter Encodings This section introduces encodings for point cloud regions and their parameters, such as the included attributes and the level of detail. It also provides an encoding for the definition of a viewport, including a dynamically adaptable precision. The encodings described in the following subsections are reused in RTCP feedback messages and RTP header extension as described in and .

Octree Encoding of Point Cloud Regions This section defines an encoding that is reused in RTCP message types to index regions in a point cloud. The encoding uses a recursive octree encoding to signal the presence of certain regions of a point cloud. The encoding can be used to set parameters for the present regions, e.g., receivers can signal individual priority of every region or sets of attributes to include in every region. The root node in every tree is a single byte where every bit indicates whether a child node of an octant is present. Child nodes are appended in pre-order traversal order. A zero-byte encodes a leaf node. The order of octants by the signs of the points of the X-, Y-, and Z-coordinates in each octant is given in . Regions in the octree are considered present when a leaf node encoding that region is present. For example, a single zero-byte encodes a single leaf node, and the root region covering the complete space is present. An encoded value of the two bytes 0x4000 would indicate that only the octant with X<0, Y>=0, and Z>=0 is present. Octant to bit mapping order with bit 0 being the most significant bit and bit 7 being the least significant bit.

Bit	X	Y	Z
0	+	+	+
1	-	+	+
2	-	-	+
3	+	-	+
4	+	+	-
5	-	+	-
6	-	-	-
7	+	-	-

The octree encoding can be used in absolute and relative forms. In the absolute form, the bounding box of the octree is implicitly defined by the space covered by the point cloud-generating source. In the relative form, the bounding box can be explicitly defined by setting the min and max values of the X-, Y-, and Z-coordinates. The coordinates are prefixed before the octree encoding as four-byte integers each, as shown in .

Relative octree encoding using an explicitly defined bounding box

Region-Dependent Attributes Encoding Attributes are additional values optionally associated with every point in a point cloud. The available attributes depend on the context of an application and thus need to be configured out-of-band. The attribute encoding described in this section requires mapping attributes to a bitmask value. describes one option to negotiate the mapping when using SDP. To signal the presence of attributes per region, senders and receivers can use the octree encoding presented in . For every region presented in the octree encoding, N bytes will be used to indicate the presence of attributes by setting the bits of the bitmask of the respective attributes to true. The size of N depends on the number of attributes and is implicitly given by the smallest number of bytes that can represent the largest bitmask negotiated during signaling. Bitmask values can be shared among attributes to allow signaling of attribute sets that always occur in combination.

Format of RTCP Feedback Messages This document describes RTCP message types that can be used to signal interest in the prioritization of regions and their parameters. Receivers can signal an interest in receiving a region with a higher priority. Different regions can be requested in different resolutions or with different sets of attributes included. Additionally, receivers can request a viewport precision update. A sender can acknowledge a parameter update using the RTP header extensions described in . Receivers should not retransmit the same request multiple times to avoid unnecessary overhead, but if the receiver can assume that a request was lost, it may retransmit the request. The request should not be retransmitted earlier than at least one RTT after the first request was transmitted. The following sections define the available message types in detail. All RTCP feedback messages use the common header format shown in . The first eight bytes of the header follow the format of RTCP message headers defined in . The common header is followed by an additional byte consisting of flags describing the payload. The payload of the RTCP messages following the header contains one or more of the parameter encodings in the following order:

1. Absolute/Relative Octree Encoding
2. (optional) Priority
3. (optional) Attribute Encoding
4. (optional) Level-of-Detail Encoding

• Note: alternative form: <Header><{abs,rel}-octree>[<priority>][<attributes>][<level-of-detail>]

• TODO: In addition to the encodings already described here, one could come up with additional parameter requests such as viewport precision, and sender position and possibly others.

The semantics of the different payload formats are explained in the following subsections.

Common RTCP feedback message header

The fields V, P, SSRC and length are used as defined in the RTP specification . The respective meaning is summarized below:

version (V): 2 bits

This field identifies the RTP version. The current version is 2.

padding (P): 1 bit

If set, the padding bit indicates that the packet contains additional padding octets at the end that are not part of the control information but are included in the length field.

Feedback message type (FMT): 5 bits

The feedback message type is XX (TODO: Use correct value).

Payload type (PT): 8 bits

The RTCP packet type is PSFB (206).

Relative (R): 1 bit

This field distinguishes between absolute and relative region requests (see . If the bit is set, the message contains a relative octree encoding and the minimum and maximum fields described in are present.

Priority: (P): 1 bit

If set, the RTCP message contains a priority request and the octree encoding is followed by one byte indicating the requested priority for each region encoded as leaf node in the octree encoding.

Attributes (A): 1 bit

If this bit is set, the RTCP message contains an attribute request and the octree encoding is followed by an attribute encoding for every region encoded as leaf node in the octree encdoing.

Level-of-Detail (L): 1 bit

If this bit is set, the RTCP message contains a level-of-detail request and the octree encoding (or the attribute encoding, if it is present) is followed by a level-of-detail encoding for every region encoded as leaf node in the octree encoding.

Region Requests A receiver can use region of Interest signaling to indicate interest in some areas of a point cloud, and a sender can react by prioritizing the regions of interest when allocating bandwidth. The region interest request uses the standard header defined in .

Priority Requests Receivers can append a priority encoding to the octree to signal fine-grained priorities per region. A priority is a value encoded as a single byte where a higher value indicates a higher priority. A priority request contains a priority byte for every region encoded as a leaf node in the preceding octree encoding.

Attribute Requests By setting the header Flag A to 1, the receiver can include an attribute encoding as described in to the feedback message to request attribute sets for every region encoded as a leaf node of the octree. The attribute encoding length depends on the number of negotiated attributes and their associated bitmask values. A single byte can indicate up to eight attributes or attribute sets. For example, if the two attributes, color, and reflectance, are negotiated with bitmask values of 0x01 and 0x02, one byte will be appended to the octree for every leaf node where the bits 0x01 and 0x02 are set to one for every region, which should include these attributes.

TODO: Region-Dependen Resolution or Level-of-Detail Request

• TODO: Using the same mechanism as for attrbiutes, we can signal resolution per region, but we need to define how to express resolution.

Examples An example encoding of a relative octree encoding followed by attribute and level-of-detail encodings:

A RTCP feedback message containing a relative octree encoding, attribute encoding and level-of-detail encoding.

Format of RTP Header Extensions RTP senders use RTP header extensions to acknowledge the applied parameters to the receiver. The parameters chosen by the sender may differ from the ones requested by the receiver. The header extension uses the two-byte header form defined in . The payload of the header extension element uses the same format as the RTCP messages defined in . Each extension element begins with the same one-byte header () followed by an octree encoding and optional priority, attribute, or level-of-detail encodings.

Common RTP header extension payload header

SDP Parameters This section defines SDP parameters for negotiating usage of the RTCP messages and RTP header extension described in this document.

RTCP Feedback Messages The rtcp-fb attribute is extended to indicate the capability to send or receive the RTCP feedback defined in this document. This document adds a new parameter "oerr" to the "ccm" feedback value defined in . The "oerr" (Octree Encoded Region Request) parameter indicates support for the RTCP feedback message defined in . rtcp-fb-ccm-param =/ SP "oerr" ; Octree Encoded Region Request

RTP Header Extensions The URI for indicating support for the RTP Header extension described in is "urn:ietf:params:rtp-hdrext:octree-region".

IANA Considerations

RTCP Feedback Message The following value are requested to be registered as FMT value in the "FMT Values for PSFB Payload Types" Registry:

Name:

OERR

Long Name:

Octree Encoded Region Request

Value:

TODO

Reference:

TODO: This document

RTP Header Extension The following URI is requested to be added to the RTP Compact Header Extensions registry:

Extension URI:

urn:ietf:params:rtp-hdrext:octree-region

Description:

Octree Encoded Region Information

Contact:

mathis.engelbart@gmail.com

Reference:

TODO: This document

Security Considerations TODO

RFC Editor Considerations Note to RFC Editor: This section may be removed after carrying out all the instructions of this section. TODO: Consider

Normative References In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This memorandum describes RTP, the real-time transport protocol. RTP provides end-to-end network transport functions suitable for applications transmitting real-time data, such as audio, video or simulation data, over multicast or unicast network services. RTP does not address resource reservation and does not guarantee quality-of- service for real-time services. The data transport is augmented by a control protocol (RTCP) to allow monitoring of the data delivery in a manner scalable to large multicast networks, and to provide minimal control and identification functionality. RTP and RTCP are designed to be independent of the underlying transport and network layers. The protocol supports the use of RTP-level translators and mixers. Most of the text in this memorandum is identical to RFC 1889 which it obsoletes. There are no changes in the packet formats on the wire, only changes to the rules and algorithms governing how the protocol is used. The biggest change is an enhancement to the scalable timer algorithm for calculating when to send RTCP packets in order to minimize transmission in excess of the intended rate when many participants join a session simultaneously. [STANDARDS-TRACK] This document provides a general mechanism to use the header extension feature of RTP (the Real-time Transport Protocol). It provides the option to use a small number of small extensions in each RTP packet, where the universe of possible extensions is large and registration is decentralized. The actual extensions in use in a session are signaled in the setup information for that session. This document obsoletes RFC 5285. This document specifies a few extensions to the messages defined in the Audio-Visual Profile with Feedback (AVPF). They are helpful primarily in conversational multimedia scenarios where centralized multipoint functionalities are in use. However, some are also usable in smaller multicast environments and point-to-point calls. The extensions discussed are messages related to the ITU-T Rec. H.271 Video Back Channel, Full Intra Request, Temporary Maximum Media Stream Bit Rate, and Temporal-Spatial Trade-off. [STANDARDS-TRACK]

Acknowledgments TODO acknowledge.