]> CableLabs

US g.white@cablelabs.com

Ericsson

SE ingemar.s.johansson@ericsson.com

Intel

US dibakar.das@intel.com

Internet Internet Area Working Group reordering link subnetwork Several link technology standards mandate that equipment guarantee in-order delivery of layer 2 frames, apparently due to a belief that this is required by higher layer protocols. In addition, certain link types can introduce out-of-order arrivals at the end of the layer 2 link, which the receiving equipment is required to rectify by delaying higher sequenced frames until all lower sequenced frames can be delivered or are deemed lost. The delaying of higher sequenced frames is generally done without any knowledge of the higher layer protocols in use, let alone any knowledge of higher layer protocol contexts (e.g. TCP connections) in the case that the layer 2 link is carrying a multiplex of such contexts. It could, for example, be the case that all of the higher sequenced frames being delayed are carrying packets for different layer 4 contexts than a single lower-sequenced frame that triggered the delay. The result is that this "re-sequencing" operation can introduce delays that result in degradation of performance rather than improving it. Moreover, modern, performant TCP and QUIC implementations support features that significantly improve their tolerance to out-of-order delivery. This draft is intended to promote an analysis and discussion of the sensitivity of modern protocols to out-of-order delivery, and to potentially develop new guidance to layer 2 technology standards regarding the need to assure in-order delivery.

Introduction provides advice to Internet subnetwork designers regarding packet reordering. The advice seems reasonable: "try to avoid packet reordering whenever possible, but not if doing so compromises efficiency, impairs reliability, or increases average packet delay." However, along with this guidance come a couple of warnings that appear to have driven subnetwork designers towards assuring in-order delivery even if efficiency is impaired or average packet delay is increased. These warnings relate to the potential performance cost for TCP (as it was then defined) that arises due to out-of-order arrivals, and to the fact that out-of-order delivery would cause problems for "every header compression scheme currently standardized for the Internet." The text of does mention that research on improving TCP behavior in the face of packet reordering was underway. In the intervening 20+ years, that research has resulted in the definition of RACK which significantly reduces TCP's sensitivity to out-of-order arrivals. In addition, discusses the usefulness of implementing similar mechanisms in QUIC, and provides the protocol tools to do so. Moreover, neither TCP nor QUIC fail in the presence of some out-of-order packets, even if they don't implement RACK. The may see reduced performance, but both protocols have the ability to correctly handle the receipt of packets out of order. TODO: discuss current state of header compression In addition to the warnings about TCP and header compression discussed in , there may be other protocols that are sensitive to reordering. For instance, discusses IPSec and L2TP as being sensitive to packet reordering. TODO: describe "resequencing" generally The current situation seems to be that several layer 2 technologies implement resequencing functions that increase cost and complexity of equipment, and may in many cases degrade network performance rather than improving it. Any benefits of resequencing may be primarily limited to older protocol implementations e.g. maintaining the performance of older TCP implementations that are no longer widely used and may not be particularly performant by modern standards.

Terminology In this document, we adopt the following terminology.

Reordering:

The process where the packet/frame sequence is made out-of-order from the originally transmitted order.

Resequencing:

The process where an out-of-order packet/frame sequence is returned to the originally transmitted order.

Reordering in L2 Links In current L2 link technologies, frame reordering comes from two primary sources: multi-link aggregation and layer-2 retransmission.

Multi-Link Aggregation Some link technologies support the aggregation (within L2) of multiple links between a pair of nodes for the purpose of increasing the total capacity of the link. In some cases the individual links within the aggregation could have different capacities and/or latencies from one another. When this is the case, the frame reception order can differ from the frame transmission order.

Layer 2 Retransmissions Some link technologies (particularly wireless), are subject to noise and interference that would result in an unacceptably high frame error rate if it were not corrected. These links commonly implement a method of acknowledgement and retransmission of lost frames, referred to as Automatic Repeat Request (ARQ). When a frame loss affects one or more frames within a transmission, and the frame(s) at the end are unaffected, the retransmitted frames will arrive out of sequence from the original ordering.

Resequencing in L2 Standards This section discusses functions and requirements for resequencing in exising layer 2 standards.

LTE/5G TBD

Wi-Fi TBD

DOCSIS The Data-Over-Cable Service Interface Specifications provide broadband access over hybrid fiber-coaxial networks. The DOCSIS protocol does not support layer 2 retransmissions, but since the introduction of "channel bonding" functionality (a form of multi-link aggregation) in DOCSIS 3.0 in 2006, reordering can occur due to differences in capacity or latency between the bonded channels. All equipment built to that and later versions of DOCSIS (including DOCSIS 3.1 and DOCSIS 4.0) is required to support specific resequencing features to guarantee in-order delivery. In the downstream direction (i.e. from the network to end-user) individual IP packets are encapsulated in layer 2 frames that generally include a 20-bit Downstream Service ID (DSID), a 1-bit Sequence Change Count, and a 16-bit Packet Sequence Number. The specifications include detailed, manadatory, requirements on the handling of these fields, including a requirement that cable modems support at least 16 simultaneous resequencing contexts, each of which having sufficient memory to hold packets for up to 13 ms (18 ms in older equipment) at the maximum forwarding rate of the device, mechanisms for rapid loss detection, and significant management and configuration mechanisms to support the feature. In the upstream direction (i.e. from the end-user to the network) individual IP packets are encapsulated in layer 2 frames, then concatenated together and fragmented into "segments" for transmission. The segment boundaries are determined by the size of the transmission opportunities (grants) and generally do not relate to the internal frame or packet boundaries. So, in general a segment could begin with the tail of one L2 frame, then have zero or more full L2 frames, then the head of a final frame. Each segment has a segment header which contains a 13-bit sequence number. The CMTS is required to reassemble the concatenated frame sequence, and forward the individual frames in order.

IANA Considerations This memo includes no request to IANA.

Security Considerations This document should not affect the security of the Internet.

References Informative References

Acknowledgements

Contributors Thanks to all of the contributors.