A RoCEv2 Flow-Level Load Balancing Method Based on the IPv6 Flow Label

Introduction The rapid advancement of Artificial Intelligence (AI) and High-Performance Computing (HPC) has driven the widespread adoption of Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCEv2) in data center and intelligent computing networks. RoCEv2 enables high-throughput, low-latency data transfers that are critical for distributed training and storage workloads. However, the effective operation of these networks is challenged by the inherent characteristics of RDMA traffic, particularly the "elephant flow" problem. Traditional load balancing mechanisms in IP networks typically rely on a 5-tuple (source/destination IP address, source/destination port, and protocol number) to identify and distribute traffic flows. In RoCEv2 networks, a significant limitation arises: multiple distinct RDMA sessions or flows generated by the same upper-layer application may share an identical 5-tuple. This is because the RDMA Queue Pair (QP) information, which uniquely identifies a session, is encapsulated within the InfiniBand Base Transport Header (IB BTH) and Datagram Extended Transport Header (IB DETH) of the RoCEv2 packet. Consequently, conventional 5-tuple-based hashing treats these distinct RDMA flows as a single entity and forwards them to the same network path, leading to severe congestion, packet loss, and a significant degradation in overall network throughput and performance. To address this problem, this document introduces a novel method for flow-level load balancing that leverages a standard IPv6 extension mechanism. The core idea is to enable an ingress network device (e.g., a top-of-rack switch or router at the edge of the network) to extract the QP pair information (source QP and destination QP) from the RoCEv2 packets. This extracted QP pair information is then used as input to a CRC32-based hash function to generate a unique per-flow identifier. This identifier is subsequently mapped into the Flow Label field of the IPv6 header. The sending host is not required to set the Flow Label; it may send packets with the Flow Label set to zero, as permitted by . By combining the traditional 5-tuple with this dynamically generated Flow Label, the proposed method creates a fine-grained "5-tuple + Flow Label" flow identification key. This allows network devices to effectively distinguish between different RDMA sessions that were previously indistinguishable, thereby achieving true flow-level load balancing. This approach minimizes path collisions, reduces congestion, and enhances the utilization of multi-path network topologies within RoCEv2 environments. This document outlines the concept, details the packet processing method, and describes the mapping of the QP pair to the IPv6 Flow Label field. The subsequent sections will cover the mechanism in detail, discuss its advantages over existing solutions, and present use cases for its implementation in intelligent computing and data center networks.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

AIDC: Artificial Intelligence Data Center RoCEv2: RDMA over Converged Ethernet version 2 RDMA: Remote Direct Memory Access QP: Queue Pair IB BTH: InfiniBand Base Transport Header IB DETH: InfiniBand Datagram Extended Transport Header CRC32: Cyclic Redundancy Check 32-bit algorithm. PRNG: Pseudo-Random Number Generator Elephant Flow: A data flow that transfers a large volume of data (e.g., gigabytes or more) over a sustained period, often associated with distributed training, checkpointing, and parameter synchronization in AI/HPC workloads.

Relation to Existing Standards and the Need for QP-based Entropy This method extends the principles established in and for using the IPv6 Flow Label. recommends source hosts set the Flow Label using a PRNG to provide entropy for load balancing. further specifies that this Flow Label can be used by intermediate routers for ECMP hashing. describes using the IPv6 Flow Label for layer 3/4 load balancing in large server farms, suggesting that {source address, flow label} can serve as a session key when the Flow Label is non-zero. The present document complements in two key aspects: 1. targets general HTTP/TCP traffic where the 5-tuple is sufficient to distinguish sessions, whereas this document addresses RoCEv2 networks where multiple RDMA flows share an identical 5-tuple; 2. assumes the source host sets the Flow Label per , while this document specifies that the ingress network device writes the Flow Label based on QP information from the RoCEv2 payload, with the host setting it to zero as a signal for relabeling. However, the above existing standards all rely on a fundamental assumption: that different transport layer flows are distinguishable by their 5-tuple (Source/Dest IP, Source/Dest Port, Protocol). In a RoCEv2 network, this assumption breaks down. A single host process (e.g., a GPU communicator library) often creates multiple parallel RDMA sessions, all of which share the same source and destination IPs (typically the host interface IPs) and port numbers (the standard UDP port for RoCEv2). Therefore, the 5-tuple is identical for all these concurrent flows. In this scenario, a standard PRNG, which has no knowledge of the internal RDMA session structure, would assign the same Flow Label to all packets, or a random value that cannot be correlated with the actual sessions. This would not solve the "elephant flow" problem. To overcome this, the proposed method extracts the actual source of session differentiation: the Queue Pair (QP) number. The QP number is the unique identifier for an RDMA connection and is present within the IB BTH header of the RoCEv2 packet. By hashing this information, the Flow Label becomes a direct function of the session itself, not just the network 5-tuple. Moreover, To ensure compatibility with the expectations of , particularly Section 2, the following operational rule is established for this scheme: 1. As specified in , a Flow Label of zero is used to indicate packets that have not been explicitly labeled. Hosts participating in this scheme SHOULD either set the Flow Label to zero or not set it at all (which results in a default value of zero). This zero value serves as a signal to the network that the packet is eligible for relabeling by the ingress network device. 2. If the arriving packet already carries a non-zero Flow Label value (e.g., set by the sending host or by another upstream device), the ingress network device MUST NOT overwrite it. This rule aligns with the core principle of that "once set to a non-zero value, the Flow Label is expected to be delivered unchanged to the destination node(s)."

Construction of the Hash Input Ensuring the generated Flow Label can uniquely identify an RDMA flow while possessing sufficient randomness to minimize collision probability is critical. The procedure for constructing the hash input is as follows: 1. Extract the QP Pair: * Dst_QP: Extracted from the IB BTH (Base Transport Header) header, 24 bits long (e.g., 0xABCDEF). The IB BTH is mandatory and always present in the RoCEv2 packet, so Dst_QP is always available. * Src_QP: If the IB DETH (Datagram Extended Transport Header) header is present, extract Src_QP from the DETH header, 24 bits long (e.g., 0x123456). The IB DETH header is optional and only present in unreliable datagram (UD) service. In reliable connection (RC) and other transport services commonly used in AI computing networks, the DETH header is absent, and consequently Src_QP is not available. 2. Generate the Entropy Source: To increase hash randomness, an entropy source is introduced. This scheme recommends using portions of the IPv6 addresses. * Take the lower 16 bits of the IPv6 source address as the first entropy source, Entropy_Src. * Take the lower 16 bits of the IPv6 destination address as the second entropy source, Entropy_Dst. When DETH is present (Src_QP available): Hash_Input = {Src_QP || Dst_QP || Entropy_Src || Entropy_Dst} (80 bits) When DETH is absent (Src_QP not available): Hash_Input = {Dst_QP || Entropy_Src || Entropy_Dst} (56 bits) In the latter case, since only Dst_QP is used as the QP entropy source, the overall entropy is reduced. However, this is a reasonable trade-off: Dst_QP alone is still sufficient to distinguish different RDMA flows in most practical scenarios, because different QP sessions are uniquely identified by their destination QP number within the receiver. The use of IPv6 address entropy (Entropy_Src and Entropy_Dst) further mitigates the risk of collision.

Hash by CRC32 Algorithm This draft uses CRC32 as the core hash algorithm and Initialize the CRC register to 0xFFFFFFFF. CRC32 offers advantages such as fast computation, hardware-friendly implementation, and a low collision rate, making it highly suitable for line-rate forwarding in network devices. First step is Byte-wise Split (using Hash_Input = 0x123456ABCDEF00010002): 0x12, 0x34, 0x56, 0xAB, 0xCD, 0xEF, 0x00, 0x01, 0x00, 0x02 Second step is iterative Processing per Byte (using the first byte 0x12 as an example): Step 1 (XOR): XOR the lower 8 bits of the CRC register with the byte 0x12. Step 2 (8-bit Shift-XOR Loop): Process the result from Step 1 bit-by-bit for 8 iterations. In each iteration: a. Check the least significant bit (LSB) of the CRC register. b. Shift the CRC register right by one bit (pad the high bit with 0). c. If the LSB was 1, XOR the result with the generator polynomial 0x04C11DB7. Repeat Steps 1 and 2 for all subsequent bytes. After processing all bytes, the value in the CRC register is the final 32-bit hash result (e.g., 0x8E4D7A2F).

Flow Label Field Population From the 32-bit CRC32 hash result, take the lower 20 bits as the Flow Label value and write this 20-bit value into the Flow Label field of the IPv6 header.

Updated IPv6 Header Structure Showing the Newly Populated Flow Label Field

Where the Flow Label is Modified In this method, the Flow Label field is modified by the ingress network device(e.g., a top-of-rack switch or router), not by the sending host. The sending host sends the RoCEv2 packet with the IPv6 Flow Label field SHOULD set to zero, as is the default behavior specified in . Upon receiving the packet, the ingress network device parses the QP information from the IB BTH and IB DETH headers, computes the CRC32 hash, and writes the resulting 20-bit value into the Flow Label field before forwarding the packet toward the destination.

IANA Considerations This document makes no request to IANA.

Security Considerations

Security issue This scheme only modifies the Flow Label field of the IPv6 header, which is performed by the ingress network device. It does not involve altering the packet payload and does not affect end-to-end application-layer security (e.g., IPsec). The modification does not change IP addresses or port numbers, thus imposing no additional processing burden on existing stateful firewalls or NAT devices. However, as noted in , the Flow Label is an unprotected field that could be modified en route. Since the ingress network device is a trusted infrastructure element within the network operator's domain, the risk of malicious Flow Label modification is confined to the same trust boundary as other network devices.

Compatibility issue End-to-End Protocol: The receiving device typically ignores the Flow Label field, making the scheme completely transparent to terminals that support standard IPv6. Intermediate Devices: All network devices supporting the IPv6 Flow Label field can benefit from this scheme. For legacy devices that do not support the Flow Label, they can still forward packets based on the traditional 5-tuple. The scheme will not cause connectivity issues, but the full performance benefits will not be realized. Hardware-Friendly Implementation: The CRC32 algorithm is widely supported in existing network ASICs. Implementing the required logic (parsing BTH/DETH headers, performing the hash, and modifying the Flow Label) is relatively straightforward and requires minimal changes to existing hardware.

References Normative References

Contributors Thanks to all the contributors.