]> Elastic Bandwidth-aware Routing Framework China Mobile
China chengweiqiang@chinamobile.com
Huawei
China zhangka@huawei.com
Huawei
China zhangli344@huawei.com
Telefonica
Spain luismiguel.contrerasmurillo@telefonica.com
Huawei
China jie.dong@huawei.com
Routing Area Routing Area Working Group Bandwidth IGP Load-balancing Routing IGP normally computes the shortest paths in a network for packet forwarding, without taking the traffic demands and available bandwidth into consideration. When there is a link degradation or partial link failure in a network which causes throughput reduction, or the volume of specific traffic flows increase dramatically, unexpected congestion may happen if only the shortest paths are used for IP forwarding. Conventional centralized Traffic Engineering (TE) focuses on long-term bandwidth and routes planning based on traffic demands, which can not react to the congestions in networks timely. This document describes a distributed path computation and load balancing mechanism named Elastic Bandwidth-aware Routing (EBR), which can alleviate congestions timely before TE finishes the global optimization. It allows IGP-enabled nodes which face congestion to distribute traffic among the shortest paths and load-balancing alternate paths through Segment Routing Traffic Engineering (SR-TE), with weights determined based on the bandwidth utilization and available bandwidth of these paths. It provides an efficient, accurate and backward compatible approach for dynamic link congestion avoidance.
Introduction IGP normally computes the shortest path in a network for packet forwarding, without taking the traffic demands and available bandwidth into consideration. Although IGP TE extensions allow to advertise link bandwidth related information in link state advertisements, such information is not used by IGP for path computation. When there is a link degradation or partial link failure (e.g. bundle member link failure) in the network which causes throughput reduction, or the volume of specific traffic flows increase dramatically, unexpected congestion may happen if only the shortest path is used for IP forwarding. As IGP itself usually does not react to link bandwidth changes or congestions, this means the congestion problem currently can only be solved by manual adjustment (e.g. adjusting the link metric) or network controller-based traffic steering, resulting in long (usually from minutes to hours) recovery time and large economic losses. Although traffic engineering (TE) technology has been widely deployed in networks, the TE paths are usually pre-calculated by the ingress nodes or a centralized controller based on the bandwidth requirements and the available bandwidth in the network. However, this information is not always predictable, when unexpected changes happen (either in the available bandwidth or the bandwidth requirement), conventional TE technology can't react to these changes timely. This document describes a distributed path computation and load balancing mechanism named Elastic Bandwidth-aware Routing (EBR), which can alleviate congestions timely before TE finishes the global optimization. It allows IGP-enabled nodes facing congestion to distribute traffic among the shortest paths and pre-calculated load-balancing alternate paths through Segment Routing Traffic Engineering (SR-TE), with weights determined based on the bandwidth utilization and available bandwidth of these paths. It can reduce the burden and dependency on the controller by reducing the involvement of global TE.
Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.
Terminology
Load-balancing Alternate path:
Alternate TE routing paths used for traffic load balancing when the primary path is congested.
Congestion Threshold:
A configured value, when the bandwidth utilization of a local link exceeds this value, then the traffic distribution is initiated among the primary path and load-balancing alternate paths.
Restore Threshold:
A configured value, when the bandwidth utilization of a local link falls below this value, then the traffic distribution among the primary path and load-balancing alternate paths is canceled for the local link, and the primary path forwarding is restored.
Local Load Balancing Node (LLBN):
An EBR-enabled network node, which performs traffic load balancing upon detecting congestion on one of its local links.
Use Cases EBR aims to alleviate the link congestion caused by unexpected events timely. The typical use cases include but are not limited to link congestions caused by link degradation and burst traffic.
Congestion Caused by Link Degradation Existing IGP protocols calculate the shortest paths for traffic forwarding, it usually does not consider the actual link bandwidth and the traffic rates on the links. As a consequence, congestion may occur when network failure results in capacity reduction on a network link. An example network topology is shown in .
An example network topology
There are four nodes A, B, C, and D in the network, between each pair of the adjacent nodes, two fibers are bound as one bundle link. The shortest path from A to C is A->B->C. However, one of the fibers between A and B is broken due to unexpected events, but IGP protocol does not perceive this change because the link state connectivity does not change. Then the shortest path from A to C is still A->B->C, and the traffic from A to C is still forwarded along A->B->C. As a consequence, congestion may occur between node A and B, since the link bandwidth has been reduced by half. However, there are alternate paths from A to C (A->D->C and A->D->B->C), which provide plenty of available bandwidth. A mechanism is needed to distribute the traffic among the primary path and the alternate paths to accommodate the traffic during link failure and avoid congestion on the shortest path.
Congestion Caused by Burst Traffic Another example is the congestion caused by bursts traffic. Considering the same network topology as described in . The shortest path from A to C is A->B->C. Generally, the bandwidth of the link from A to B is capable of carrying the traffic from A to B. However, there may be some burst traffic from A to C due to some unexpected events (such as a concert or a football match), and the traffic exceeds the available bandwidth of link A-B. As a consequence, congestion may occur on the link from A to B. However, there are alternate paths from A to C (A->D->C and A->D->B->C), which provide plenty of available bandwidth to accommodate the burst traffic and avoid congestion. A mechanism is needed to distribute the traffic among the primary path and the alternate paths to accommodate the traffic burst and avoid congestion on the shortest path.
Overview of EBR This document proposes a new mechanism called EBR for dynamic congestion alleviation. EBR integrates IGP with SR-TE traffic steering, allowing the distribution of traffic among the primary path and multiple load-balancing alternate paths based on the perception of link congestion and bandwidth information of the whole network. It can effectively alleviate the congestion caused by different network events. EBR consists of four major steps:
  1. Monitoring and advertisement of link bandwidth information: Each EBR-enabled network node monitors the available bandwidth and bandwidth utilization of its local links, and advertises the update of bandwidth related information to other nodes using IGP.
  2. Load-balancing alternate path calculation: Each EBR-enabled network node calculates both the shortest path and load-balancing alternate paths to a specific destination. The algorithm for load-balancing alternate paths should try to keep the shortest path and the load-balancing alternate paths disjoint.
  3. Traffic distribution upon congestion: Once a LLBN detects that the bandwidth utilization of one of its links exceeds the Congestion Threshold, it will distribute traffic whose shortest path is via that link to the load-balancing alternate paths based on Unequal Cost Multiple Path (UCMP). The traffic forwarding on load-balancing alternate paths should be based on SR-TE to avoid forwarding loops. The weight of each load-balancing alternate path is determined based on the available bandwidth and bandwidth utilization of the paths.
  4. Traffic fallback: When some conditions are met (e.g., bandwidth utilization drops below the Restore Threshold), the load balancing is stopped and all traffic is reverted back to shortest path forwarding.
EBR Procedures
Monitoring and Advertisement of Link Bandwidth Information LLBN needs to continuously monitor the bandwidth utilization and available bandwidth of its outbound links. The bandwidth utilization is the key information for determining whether a link is congested. The determination of a link congestion depends on a configurable Congestion Threshold (such as 80%, 90%, etc.), which can be configured by network operator according to the network conditions. The mechanism used to obtain the bandwidth utilization information of a local link is out of scope of this document. Upon local link congestion, distributing traffic blindly to alternate paths may lead to new congestion occurring on other links. Therefore, it is crucial to obtain the information about the available bandwidth and bandwidth utilization of each link in the network. With such information, the headend can distribute traffic based on the available bandwidth and bandwidth utilization of the links on each path, thereby avoiding the occurrence of new congestion. The advertisement of bandwidth information can be achieved through IGP advertisement. Existing TE metric extensions to IGPs already allow a node to advertise the bandwidth related information (e.g. maximum link bandwidth, available bandwidth, and utilized bandwidth, etc.) of its links (IS-IS, OSPF). The considerations for bandwidth information advertisement is introduced in .
Load-balancing Alternate Path Calculation The load-balancing alternate paths in EBR are used for load balancing when the primary path is congested. Although the algorithm used for load-balancing alternate calculation is implementation-specific, it should meet the following requirements:
  1. The load-balancing alternate paths should be calculated by LLBN in advance to allow quick triggering of load balancing when local link congestion is detected.
  2. The calculation should make that the load-balancing alternate paths and the primary path are disjoint as much as possible.
  3. The bandwidth utilization of each link may be considered during the calculation, to avoid using links with high bandwidth utilization for congestion traffic offloading.
  4. Depending on service flow needs, other metrics and constraints may be considered during calculation.
The number of alternate paths depends on the configuration and network topology. The algorithm for calculating load-balancing alternate paths is out of scope of this document.
Traffic Distribution Upon Congestion Once a LLBN detects the bandwidth utilization of one of its outbound links exceeds the Congestion Threshold, the load balancing mechanism will be triggered to distribute traffic among the primary path and load-balancing alternate paths. UCMP is recommended to distribute flows among the paths, the weight of each path is determined according to the available bandwidth and bandwidth utilization of the path. The available bandwidth of a path is the available bandwidth of the link with the smallest available bandwidth on the entire path. The bandwidth utilization of each path is the utilization of the link with highest utilization rate on the entire path. Once the weight of each path is determined, it will not change unless new congestions are detected on the links of load-balancing alternate paths. Traffic distributed to load-balancing alternate paths will be forwarded based on SR-TE mechanism, which ensures the traffic be forwarded without loops.
  • For SR-MPLS network, packets will be encapsulated with an ordered list of MPLS labels which represent the load-balancing alternate path.
  • For SRv6 network, packets will be encapsulated with an outer IPv6 header, together with an SRH which contains the SID list representing the load-balancing alternate path.
Policies may be used for determining which groups of flows (e.g., according to traffic class, IP prefixes, etc.) should be migrated from the primary path to the load-balancing alternate paths.
Traffic Fallback Traffic fallback means a LLBN migrate the traffic on load-balancing alternate paths back to the primary path. This process restores the network to the original state. There are several methods for triggering traffic fallback:
  1. Threshold-based traffic fallback: When a LLBN detects that the bandwidth utilization of a congested link falls below the Restore Threshold, then the traffic fallback is triggered.
  2. Dynamic traffic fallback: In this method, there is no static threshold for traffic fallback, the fallback is triggered dynamically upon the local link of the LLBN can bear all the traffic without congestion.
In both of the above methods, the traffic fallback should be triggered only when the conditions have been met for a configurable period of time. Method 2 is recommended as it can avoid oscillations in traffic distribution and traffic fallback.
Operational Considerations
Oscillation Suppression Micro-burst traffic or flapping bundle member link may cause frequent change of the link utilization and congestion state, and may result in oscillation in traffic distribution. The following oscillation suppression measures should be taken:
  • The determination of congestion and restoration should consider the statistical characteristics of bandwidth utilization over a period of time, rather than only bandwidth utilization in a short interval.
  • If threshold-based traffic fallback is used, then the Congestion Threshold should be sufficiently far from the Restore Threshold to avoid oscillation in the link's congestion status caused by small traffic fluctuation.
Alleviation of Possible New Congestions The available bandwidth and utilization of load-balancing alternate paths are considered in traffic distribution, which effectively reduces the possibility of secondary congestion on the alternate paths. While in some cases it is possible that distributing traffic from primary path to load-balancing alternate paths may cause new congestion for the following reasons:
  • The available bandwidth of an alternate path does not match the rate of assigned flows. Although UCMP is used in the distribution of flows to alternate paths, due to different size of flows, a big flow may cause new congestion on some links of an alternate path.
  • Simultaneous traffic distribution initiated by different nodes. Since network nodes which support EBR act independently, simultaneous traffic distribution is possible, which may cause the total diverted traffic rate exceeds the available bandwidth of some links of an alternative path.
There are two mechanisms to alleviate the new congestions.
  • When a LLBN which initiated the traffic distribution perceives that the bandwidth utilization of an in-use load-balancing alternate path exceeds the Congestion Threshold, it can adjust the traffic distribution weight on different alternate paths to reduce the flows on the congested paths to relieve the congestion.
  • The LLBN which is adjacent to the newly congested link can initiate traffic distribution and divert a portion of traffic to its load-balancing alternate paths to alleviate the congestion.
Considerations on Bandwidth Information Advertisement
Trigger of Bandwidth Information Advertisement Although the effectiveness of EBR relies on the accuracy of available bandwidth and bandwidth utilization information, the control plane overhead in the advertisement and processing of the bandwidth information update also needs to be considered. Some recommendations about bandwidth information advertisement are provided as follows:
  • The interval of two advertisement must not be less than the Minimum Advertisement Interval. The Minimum Advertisement Interval should be configurable, 30 seconds is recommended for it.
  • If the Minimum Advertisement Interval expires, and there is a x% change in link available bandwidth since the last advertisement, then an advertisement should be sent. The value of x should be configurable, and 20% is recommended for it.
  • If the Minimum Advertisement Interval expires, and there is a consumption of over y% the link’s physical bandwidth and that was not noted in the previous advertisement, then an advertisement should be sent. The value of y should be configurable, and 70% is recommended for it.
  • In order to prevent the clustering of IGP messages on the receiving nodes, a “jitter” can be introduced. It is recommended to set the jitter value to 1/3 of the Minimum Advertisement Interval.
Optimizations on Bandwidth Information Advertisement As the advertisement of bandwidth information in IGPs are mainly for RSVP-TE based TE path computation, there may be concerns that advertising such information more frequently may affect those TE applications. The mechanisms with Application-Specific Link Attributes as defined in could be used to limit the usage of more frequently advertised bandwidth information to EBR, so as to avoid the impact to other TE applications which may use bandwidth information as its input.
Compatibility EBR can be deployed incrementally in the network. Network nodes which support EBR can calculate the load-balancing alternate paths and initiate UCMP load-balancing upon local link congestion. Network nodes which do not support EBR do not calculate the load-balancing alternate paths, and will not initiate UCMP load-balancing, while they can forward the offloaded traffic according to the SR SID list in the packets. Author's note: more operational considerations will be added in future.
IANA Considerations This document has no IANA actions.
Security Considerations EBR relies on the bandwidth information advertised by IGP, incorrect bandwidth information may lead to new congestions on specific links. In most deployments, the EBR is used within a network domain entirely under the control of the same operator. However, it is worth considering that transporting link bandwidth information over insecure links could include a man-in-the-middle attacker modifying the value of bandwidth information, and causing congestions on specific links. Advertising which links are approaching congestion may give an attacker a good plan for how to destabilise the network. Destabilisation may simply involve injecting an edge-to-edge (i.e., no need to change anything inside the network) flow that will tip the identified link into congested state. The use of cryptographic authentication mechanisms of link state advertisement can mitigate the above risks.
References Normative References Key words for use in RFCs to Indicate Requirement Levels In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References IS-IS Traffic Engineering (TE) Metric Extensions In certain networks, such as, but not limited to, financial information networks (e.g., stock market data providers), network-performance criteria (e.g., latency) are becoming as critical to data-path selection as other metrics. This document describes extensions to IS-IS Traffic Engineering Extensions (RFC 5305). These extensions provide a way to distribute and collect network-performance information in a scalable fashion. The information distributed using IS-IS TE Metric Extensions can then be used to make path-selection decisions based on network performance. Note that this document only covers the mechanisms with which network-performance information is distributed. The mechanisms for measuring network performance or acting on that information, once distributed, are outside the scope of this document. This document obsoletes RFC 7810. OSPF Traffic Engineering (TE) Metric Extensions In certain networks, such as, but not limited to, financial information networks (e.g., stock market data providers), network performance information (e.g., link propagation delay) is becoming critical to data path selection. This document describes common extensions to RFC 3630 "Traffic Engineering (TE) Extensions to OSPF Version 2" and RFC 5329 "Traffic Engineering Extensions to OSPF Version 3" to enable network performance information to be distributed in a scalable fashion. The information distributed using OSPF TE Metric Extensions can then be used to make path selection decisions based on network performance. Note that this document only covers the mechanisms by which network performance information is distributed. The mechanisms for measuring network performance information or using that information, once distributed, are outside the scope of this document. IS-IS Application-Specific Link Attributes Existing traffic-engineering-related link attribute advertisements have been defined and are used in RSVP-TE deployments. Since the original RSVP-TE use case was defined, additional applications (e.g., Segment Routing Policy and Loop-Free Alternates) that also make use of the link attribute advertisements have been defined. In cases where multiple applications wish to make use of these link attributes, the current advertisements do not support application-specific values for a given attribute, nor do they support indication of which applications are using the advertised value for a given link. This document introduces new link attribute advertisements that address both of these shortcomings.
Acknowledgements TBD
Contributors Huawei
China wangyifan82@huawei.com
Huawei
China rainsword.wang@huawei.com
Huawei
China ryan.cheung@huawei.com