]> CableLabs

US g.white@cablelabs.com

Ericsson

SE ingemar.s.johansson@ericsson.com

Intel

US dibakar.das@intel.com

Internet Internet Area Working Group reordering link subnetwork Several link technology standards mandate that equipment guarantee in-order delivery of layer 2 frames, apparently due to a belief that this is required by higher layer protocols. In addition, certain link types can introduce out-of-order arrivals at the end of the layer 2 link, which the receiving equipment is required to rectify by delaying higher sequenced frames until all lower sequenced frames can be delivered or are deemed lost. The delaying of higher sequenced frames is generally done without any knowledge of the higher layer protocols in use, let alone any knowledge of higher layer protocol contexts (e.g. TCP connections) in the case that the layer 2 link is carrying a multiplex of such contexts. It could, for example, be the case that all of the higher sequenced frames being delayed are carrying packets for different layer 4 contexts than a single lower-sequenced frame that triggered the delay. The result is that this "re-sequencing" operation can introduce delays that result in degradation of performance rather than improving it. Moreover, modern, performant TCP and QUIC implementations support features that significantly improve their tolerance to out-of-order delivery. This draft is intended to promote an analysis and discussion of the sensitivity of modern IETF protocols to out-of-order delivery, and to potentially develop new information for layer 2 technology standards regarding the need to assure in-order delivery to support IETF protocols.

Introduction provides advice to Internet subnetwork designers regarding packet reordering. The advice seems reasonable: "try to avoid packet reordering whenever possible, but not if doing so compromises efficiency, impairs reliability, or increases average packet delay." However, along with this guidance come a couple of warnings that appear to have driven subnetwork designers towards assuring in-order delivery even if efficiency is impaired or average packet delay is increased. These warnings relate to the potential performance cost for TCP (as it was then defined) that arises due to out-of-order arrivals, and to the fact that out-of-order delivery would cause problems for "every header compression scheme currently standardized for the Internet." The text of does mention that research on improving TCP behavior in the face of packet reordering was underway. In the intervening 20+ years, that research has resulted in the definition of RACK which significantly reduces TCP's sensitivity to out-of-order arrivals. In addition, discusses the usefulness of implementing similar mechanisms in QUIC, and provides the protocol tools to do so. The result is that having the link-layer delay packets to provide them in-order to the transport layer was helpful multiple decades ago when TCP implementations had more naive algorithms and were very resource-constrained. Given how modern implementations work, resequencing at the link-layer is almost always harmful. Modern TCP and QUIC stacks can handle reordering well, thanks to both protocol features and implementations having more memory to work with. For these implementations, the delay induced by L2 resequencing will generally cause more harm than good. Furthermore, one of the biggest motivators for QUIC was to break head-of-line blocking and allow out-of-order delivery to the application layer. At this point we have large bodies of data showing that this improves real-world performance. Moreover, older implementations of TCP that don't implement RACK don't fail in the presence of some out-of-order packets. They may see reduced performance, but they do have the ability to correctly handle the receipt of packets out of order. At the time of publication of , the "ROHC" header compression framework defined in stated an operating assumption that the network would provide in-order delivery. This assumption is discussed in detail in , which provides guidance on implementing ROHC over channels that can reorder header-compressed packets, and explains different ways of implementing the profiles found in over reordering channels. Subsequent to the publication of the issue of reordering intolerance by ROHC was addressed by the development of ROHCv2 , which lists tolerance to reordering as one of its significant improvements over ROHC. There may be other protocols or protocol implementations that are sensitive to reordering to some degree. For instance, discusses IPSec and L2TP as being sensitive to packet reordering. TODO: describe "resequencing" generally The current situation seems to be that several layer 2 technologies implement resequencing functions that increase cost and complexity of equipment, and may in many cases degrade network performance rather than improving it. Any benefits of resequencing may be primarily limited to older protocol implementations e.g. maintaining the performance of older TCP implementations that are no longer widely used and may not be particularly performant by modern standards.

Terminology In this document, we adopt the following terminology.

Reordering:

The process where the packet/frame sequence is made out-of-order from the originally transmitted order.

Resequencing:

The process where an out-of-order packet/frame sequence is returned to the originally transmitted order.

Reordering in L2 Links In current L2 link technologies, frame reordering comes from two primary sources: multi-link aggregation and layer-2 retransmission.

Multi-Link Aggregation Some link technologies support the aggregation (within L2) of multiple links between a pair of nodes for the purpose of increasing the total capacity of the link. In some cases the individual links within the aggregation could have different capacities and/or latencies from one another. When this is the case, the frame reception order can differ from the frame transmission order.

Layer 2 Retransmissions Some link technologies (particularly wireless), are subject to noise and interference that would result in an unacceptably high frame error rate if it were not corrected. These links commonly implement a method of acknowledgement and retransmission of lost frames, referred to as Automatic Repeat Request (ARQ). When a frame loss affects one or more frames within a transmission, and the frame(s) at the end are unaffected, the retransmitted frames will arrive out of sequence from the original ordering.

Resequencing in L2 Standards This section discusses functions and requirements for resequencing in exising layer 2 standards.

LTE/5G Packet reordering can occur on both the MAC and the RLC layer in LTE and 5G systems. Packet resequencing occurs on the PDCP layer. On the MAC layer: An RLC packet data unit (RLC-PDU) is transmitted in one of up to 16 Hybrid ARQ processes on the MAC layer. This avoids the stop and wait for one transmission to succed before a new transmission can begin. Transmission on the MAC layer can however have a high block error rate (BLER), 10% or more is common especially when the traffic pattern and radio channel varies. Each retransmission adds a few milliseconds extra delay. This results in that packets are delivered out of order up to the RLC layer on the receiver side because some transmissions on the MAC layer may succeed in one transmission attempt while other transmissions may need several attepts, up to some given limit given by system configuration. HARQ failure is triggered when the max number of retransmissions is reached. On the RLC Layer: The RLC layer is configured in two typical ways.

• RLC Unacknowledged mode (RLC-UM): Is typically used for VoIP services a.k.a VoLTE (Voice over LTE) or VoNR (Voice over NR) but can also be used for the transmission of low latency video. Data blocks are not retransmitted in this mode.
• RLC Acknowledged mode (RLC-AM): Is typically used for normal internet traffic, a.k.a Mobile Broadband (MBB). Retransmissions on this layer are less common than retransmissions on the MAC layer and they occur when HARQ failure is triggered on the MAC layer, or in the case that the MAC layer erroneously decodes a NACK as an ACK. Retransmission on the RLC layer causes 10s of milliseconds on extra delay, depending on how timers are configured.

Besides the causes for packet reordering listed above, packet reordering can occur with Dual Connectivity where 2 or more radio frequency bands are combined on the RLC layer, this because there can be a time skew of a few milliseconds over transport links between the PDCP layer and the RLC layer(s). The RLC layer does resequencing on the RLC layer in LTE systems. On 5G systems however, resequencing is left to the PDCP layer. Resequencing on the PDCP layer is however optional according to the 5G standard but is mostly enabled out of concern for sensitivity to packet reordering on the transport and application layer. In addition radio bearers with robust header compression (RoHC) should enable resequencing.

Wi-Fi The IEEE 802.11 specification (Wi-Fi) currently supports only in-order delivery. The 802.11 protocols inherit traffic delivery requirements from the 802.1Q specification (see section 6.5.3 in 802.1Q), which only permits only in-order delivery. To utilize the medium efficiently, the 802.11 MAC aggregates data frames of a given traffic identifier (TID) and transmits them as a block. However, due to link errors, not all frames in the block may be received. A re-order buffer at the receiver holds up delivery of data frames if a frame ahead of it is missing. A separate reorder buffer is maintained for each of the 8 TIDs. A later retransmission may deliver the missing frame at which point the missing frame and subsequent frames are delivered. The process of in-order delivery is also taken advantage of as part of the security protocol. Each data frame is sequentially numbered with a packet number. To mitigate against a replay attack, the receiving MAC simply checks if the packet number of a received data frames is higher than the highest received one for that TID. Allowing out-of-order delivery within the 802.11 specification for a given TID would require some adjustments to the 802.11 protocol. Some of the adjustments include the following:

• The reorder buffer would need to release packets without waiting for older packets to be received.
• The replay detection mechanism would need to be updated to not drop an older missing frame when it is successfully retransmitted.

DOCSIS The Data-Over-Cable Service Interface Specifications provide broadband access over hybrid fiber-coaxial networks. The DOCSIS protocol does not support layer 2 retransmissions, but since the introduction of "channel bonding" functionality (a form of multi-link aggregation) in DOCSIS 3.0 in 2006, reordering can occur due to differences in capacity or latency between the bonded channels. All equipment built to that and later versions of DOCSIS (including DOCSIS 3.1 and DOCSIS 4.0) is required to support specific resequencing features to guarantee in-order delivery. In the downstream direction (i.e. from the network to end-user) individual IP packets are encapsulated in layer 2 frames that generally include a 20-bit Downstream Service ID (DSID), a 1-bit Sequence Change Count, and a 16-bit Packet Sequence Number. The specifications include detailed, manadatory, requirements on the handling of these fields, including a requirement that cable modems support at least 16 simultaneous resequencing contexts, each of which having sufficient memory to hold packets for up to 13 ms (18 ms in older equipment) at the maximum forwarding rate of the device, mechanisms for rapid loss detection, and significant management and configuration mechanisms to support the feature. In the upstream direction (i.e. from the end-user to the network) individual IP packets are encapsulated in layer 2 frames, then concatenated together and fragmented into "segments" for transmission. The segment boundaries are determined by the size of the transmission opportunities (grants) and generally do not relate to the internal frame or packet boundaries. So, in general a segment could begin with the tail of one L2 frame, then have zero or more full L2 frames, then the head of a final frame. Each segment has a segment header which contains a 13-bit sequence number. The CMTS is required to reassemble the concatenated frame sequence, and forward the individual frames in order.

IANA Considerations This memo includes no request to IANA.

Security Considerations This document should not affect the security of the Internet.

References Informative References

Acknowledgements

Contributors Thanks to all of the contributors.