IOAM Trace-Type Extensions for Path bandwidth

Traffic scheduling and optimization have become routine network operation and maintenance tasks for operators. The operators need to select a path that can accommodate the capacity of the traffic to be scheduled. Typically, large network operators possess two nationwide backbone networks: one positioned as the public basic Internet backbone, featuring high capacity, high throughput, and best-effort service; the other positioned as a high-quality multi-service transport backbone network, with QoS guarantees and traffic engineering capabilities. For the former, the link utilization parameter provided by the network reflects the usage of network resources; while for the latter, the available bandwidth parameter provided by the network represents the capacity of new service traffic to be accepted. In situ Operations, Administration, and Maintenance (IOAM) [RFC9197] can collect operational and telemetry information in the packet while the packet traverses a path between two points in the network. Specifically, IOAM Trace Option can determine a path composed of a specific sequence of nodes and links that a packet flow traverses between an IOAM encapsulating node and an IOAM decapsulating node in a network. Therefore, IOAM may also be a practical tool capable of quickly discovering the available bandwidth of a path. RFC9486 defines IOAM Data-Fields encapsulated in IPv6. IOAM Direct Export (DEX) Option [RFC9326] is used as a trigger for IOAM data to be directly exported to a collector. [I-D.ietf-6man-icmpv6-reflection] specifies the ICMPv6 Reflection utility, which could be used for collecting IOAM data in the forward way as well as the reverse way, and this Reflection utility allows this information to be sent back to the probing node. Similarly, [I-D.ietf-ippm-stamp-ext-hdr] could be also used for collecting IOAM data in the forward way as well as the reverse way, in which the Simple Two-Way Active Measurement Protocol (STAMP) test packets are transmitted along a path between a Session-Sender and a Session-Reflector. This document defines two bit flags within IOAM Trace-Type for carrying bandwidth information.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Abbreviations used in this document: AI: Artificial Intelligence AS: Autonomous System DEX: Direct Exporting DC: Data Center DCI: Data Center Interconnection ICMPv6: Internet Control Message Protocol version 6 IOAM: In situ Operation, Administration, and Maintenance MPLS: Multi-Protocol Label Switching QoS: Quality of Service P: Provider PE: Provider Edge PoP: Point of Presence SR-MPLS: Segment Routing over MPLS STAMP: Simple Two-Way Active Measurement Protocol

For many customers, ensuring service bandwidth is a basic requirement, so it is necessary to provide customers with bandwidth guaranteed delivery. Operators typically provide bearer services to such customers through SR-MPLS backbone networks with QoS guarantees and traffic engineering capabilities. In a single network domain scenario as depicted in Figure 1.

+-------------------------------------------------+ | +--------+ | | /------->|P1...Pn |------->\ | | / +--------+ \ | | / \ | +---------+ | +----/------+ +-------+ +-----\----+ | +--------+ |Customer |--|-->|Ingress PE |---->|P1...Pn|---->|Egress PE |--|---->|Customer| |PoP | | +----\------+ +-------+ +-----/----+ | |PoP | +---------+ | \ / | +--------+ | \ +--------+ / | | \------->|P1...Pn |------->/ | | +--------+ | | SR-MPLS backbone network | +-------------------------------------------------+ The customer' Point of Presences (PoPs) are directly connected to the ingress PE and the egress PE of SR-MPLS backbone network, and there exist multiple paths from the ingress PE and the egress PE. In order to select an appropriate path that satisfies the bandwidth reqiurement, the ingress PE sends IOAM probe packets to the egress PE through different paths, and then the egress PE can reflect these IOAM probe packets to the ingress PE, carrying per-hop available bandwidth parameters.

With the rapid growth in demand for computing and storage resources in AI big models and distributed storage, cloud computing centers are interconnected through backbone networks to provide multi-DCs collaboration to compensate for the limitations of insufficient computing and storage resources in a single DC, and improve resource utilization. In multiple network domain scenario as depicted in Figure2, the cloud Data Center (DC) Gateways are connected to multiple backbone networks to schedule traffic for load balancing.

+---------------------------------------------------+ | +----------+ +--------+ +----------+ | +---------|---|Ingress PE|-----|P1...Pn |------|Egress PE |---|--------+ | | +----------+ +--------+ +----------+ | | | | IP Internet backbone network | | | +---------------------------------------------------+ | | +---------------------------------------------------+ | +----+-----+ | +----------+ +--------+ +-----------+ | +----+-----+ | Cloud DC |---|---|Ingress PE|-----|P1...Pn |------|Egress PE |--|---| Cloud DC | | Gateway | | +----------+ +--------+ +-----------+ | | Gateway | +----+-----+ | Cloud backbone network | +----+-----+ | +---------------------------------------------------+ | | +---------------------------------------------------+ | | | +----------+ +--------+ +----------+ | | +---------|---|Ingress PE|-----|P1...Pn |------|Egress PE |---|--------+ | +----------+ +--------+ +----------+ | | SR-MPLS backbone network | +---------------------------------------------------+ Since these networks belong to different administrative domain with differnet AS number, it is difficult for the cloud DCs obtain the availabe bandwidth information of path from backbone networks, the source DC Gateway of cloud Data Center is required to send IOAM probe packets to the destination DC Gateway of cloud Data Center through multiple backbone networks, and then the destination DC Gateway can reflect these IOAM probe packets to the source DC Gateway, carrying per-hop available bandwidth parameters. Consequently, the source DC Gateway steers partial traffic into other backbone networks to mitigate the cloud backbone network. For IP Internet backbone as mentioned above, which can only provide best-effort service, the available bandwidth parameter cannot be calculated or configured by network node, thus it is not available, collecting per-hop link utilization parameters is preferable. In addition, multiple interface rates coexist on IP Internet backbone, including 100 Gb/s, 200 Gb/s, 400 Gb/s, etc. per-hop link bandwidth (i.e., interface rate) parameters are also required to be collected. Typically, the peak link utilization should not exceed 90%. Based on per-hop link utilization together with per-hop link bandwidth along the path, we can roughly calculate the amount of traffic to be added to the path. For example, an IOAM probe packet traverses a path with 3 nodes within IP Internet backbone: the first node has a link utilization of 50% together with a link bandwidth of 100 Gb/s, the second node has a link utilization of 60% together with a link bandwidth of 200 Gb/s, and the third node has a link utilization of 70% together with a link bandwidth of 400 Gb/s. We can calculate that the first node has the available bandwidth of 40 Gb/s, the second node has the available bandwidth of 60 Gb/s, and the third node has the available bandwidth of 80 Gb/s, thus we can determine that this path could still accommodate up to 40Gb/s of traffic.

IOAM Trace-Type defined in [RFC9197] is a 24-bit identifier that specifies which data types are used in the node data list. The IOAM Trace-Type value is a bit field. This document defines two bit flags within IOAM Trace-Type for carrying bandwidth information as follows: Bit 12: When set, indicates the presence of link utilization and link bandwidth in the node data. Bit 13: When set, indicates the presence of available bandwidth in the node data.

All the IOAM-Data-Fields MUST be aligned by 4 octets. If a node that is supposed to update an IOAM-Data-Field is not capable of populating the value of a field set in the IOAM Trace-Type, the field value MUST be set to 0xFFFFFFFF for 4-octet fields, indicating that the value is not populated. The link utilization and link bandwidth field is a 4-octet field that is defined as follows: 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | link utilization | Rsvd | link bandwidth | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Link utilization: 2-octet unsigned integer field as a percentage of the link utilization, restricted to [0x0000,0xffff], with value 0x0000 indicating a link utilization of 0%, and value 0xffff indicating a link utilization of 100%. The resolution is 0.0015%. Link bandwidth: 2-octet unsigned integer field. The lower 11 bits of this field are defined as the standardized interface rates. The higher 5 bits of this field are reversed for future use, and they MUST be all zero on transmission and ignored on receipt. The link bandwidth is defined as follows: Bit 0: When set, indicates the link bandwidth of 10Mb/s. Bit 1: When set, indicates the link bandwidth of 100Mb/s. Bit 2: When set, indicates the link bandwidth of 1Gb/s. Bit 3: When set, indicates the link bandwidth of 10Gb/s. Bit 4: When set, indicates the link bandwidth of 50Gb/s. Bit 5: When set, indicates the link bandwidth of 100Gb/s. Bit 6: When set, indicates the link bandwidth of 200Gb/s. Bit 7: When set, indicates the link bandwidth of 400Gb/s. Bit 8: When set, indicates the link bandwidth of 800Gb/s. Bit 9: When set, indicates the link bandwidth of 1.6Tb/s. Bit 10: When set, indicates the link bandwidth of 3.2Tb/s. The available bandwidth field is a 4-octet field that is defined as follows: 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | available bandwidth | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Available bandwidth: 4-octet unsigned integer field, defined in [RFC8570]. This field carries the available bandwidth on a link in IEEE floating-point format with units of bytes per second.

IANA has defined an "IOAM Trace-Type" Registry. This registry defines code points for each bit in the 24-bit IOAM Trace-Type field for the Pre-allocated Trace Option-Type and Incremental Trace Option-Type defined in [RFC9197]. Bit 0-11 are defined in RFC9197. Bit 12 and Bit 13 are defined in this document. IANA is requested to allocate the following code point from the "IOAM Trace-Type" registry: Bit 12: TBA. Description: link utilization and link bandwidth. Reference: This document. Bit 13: TBA. Description: available bandwidth. Reference: This document.

The security considerations of IOAM in general are discussed in [RFC9197], the security considerations of IOAM DEX Option-Type are discussed in [RFC9326]. The security considerations for using the ICMPv6 Reflection utility are discussed in [I-D.ietf-6man-icmpv6-reflection], and the security considerations for using STAMP are discussed in [I-D.ietf-ippm-stamp-ext-hdr] . There are not additional security considerations in this Extended IOAM Trace-Type.