MPLS On-Path Telemetry Network Action Flag for OAM

To gain detailed data plane visibility to support effective network OAM, it is essential to be able to examine the trace of user packets along their forwarding paths. Such on-path flow data reflect the state and status of each user packet's real-time experience and provide valuable information for network monitoring, measurement, and diagnosis. The telemetry data include but not limited to the detailed forwarding path, the timestamp/latency at each network node, and, in case of packet drop, the drop location as well as the reason. The emerging programmable data plane devices allow user-defined data collection or conditional data collection based on trigger events. Such on-path flow data are from and about the live user traffic, which complements the data acquired through other passive and active OAM mechanisms such as IPFIX and ICMP. On-path telemetry was developed to cater to the need of collecting on-path flow data. There are two basic modes for on-path telemetry: the passport mode (e.g., IOAM trace option) and the postcard mode (e.g., IOAM direct export option (DEX) ). In MPLS networks, MPLS Network Action (MNA) extends the MPLS label stack by supporting extra in-stack network actions and ancillary data encoded in stack, the in-stack MNA Sub-Stack is described in . MNA also extends the MPLS payload by supporting extra post-stack network actions and ancillary data encoded post-stack, the post-stack MNA header is described in . This document describes the method to apply a new variation of the postcard mode on-path telemetry, PBT-M, to MPLS network using an MNA flag only. PBT-M does not require a telemetry instruction header but a single trigger bit in MNA flags. The similar mechanism has been adopted by SRv6 for SRv6 OAM, which uses the O-bit in SRH flags as the marking bit to trigger the on-path telemetry. The key benefits of PBT-M are its low overhead and high flexibility. However, PBT-M introduces some unique requirements that need to be considered. This document discusses the requirements and their solutions in MPLS networks.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

As the name suggests, PBT-M only needs a marking-bit in the existing headers of user packets to trigger the telemetry data collection and export. The sketch of PBT-M is as follows. If on-path data need to be collected, the user packet is marked at the path head node. At each PBT-M-aware node, if the mark is detected and data collection is enabled, a postcard packet (i.e., the dedicated OAM packet triggered by a marked user packet) is generated and sent to a collector. The postcard contains the data requested by the management plane. The requested data are configured by the management plane. Once the collector receives all the postcards for a single user packet, it can infer the packet's forwarding path and analyze the data set. The path end node is configured to un-mark the packets to its original format if necessary. The overall architecture of PBT-M is depicted in Figure 1.

+------------+ +-----------+ | Network | | Telemetry | | Management |(-------| Data | | | | Collector | +-----:------+ +-----------+ : ^ :configurations |postcards : |(OAM pkts) ...............:.....................|........ : : : | : : +---------:---+-----------:---+--+-------:---+ : | : | : | : | V | V | V | V | +------+-+ +-----+--+ +------+-+ +------+-+ usr pkts | Head | | Path | | Path | | End | ====>| Node |====>| Node |====>| Node |====>| Node |===> | | | A | | B | | | +--------+ +--------+ +--------+ +--------+ mark usr pkts gen postcards gen postcards gen postcards gen postcards unmark usr pkts The advantages of PBT-M are summarized as follows. 1: PBT-M avoids augmenting user packets with new headers and the signaling for telemetry data collection remains in the data plane. 2: PBT-M is extensible for collecting arbitrary new data to support possible future use cases. The data set to be collected can be configured through the management plane or control plane. 3: PBT-M can avoid interfering with the normal forwarding. The collected data are free to be transported independently through in-band or out-of-band channels. The data collecting, processing, assembly, encapsulation, and transport are, therefore, decoupled from the forwarding of the corresponding user packets and can be performed in data-plane slow-path if necessary. 4: For PBT-M, the types of data collected from each node can vary depending on application requirements and node capability. 5: PBT-M makes it easy to secure the collected data without exposing it to unnecessary entities. For example, both the configuration and the telemetry data can be encrypted and/or authenticated before being transported, so passive eavesdropping and a man-in-the-middle attack can both be deterred. 6: Even if a user packet under inspection is dropped at some node in the network, the postcards collected from the preceding nodes are still valid and can be used to diagnose the packet drop location and reason. 7: Raw data can be processed or aggregated in data plane to reduce the exporting traffic load.

Although PBT-M has some unique features, it also introduces a few new requirements. Req. 1 (Packet Marking Bit): A user packet needs to be marked to trigger the path-associated data collection. Since PBT-M aims to avoid the need to augment user packets with new headers, it needs to reserve or reuse a single bit from the existing header fields. Req. 2 (Configuration): Since the packet header will not carry telemetry instructions anymore, the data plane devices need to be configured to know what data to collect. However, in general, the forwarding path of a flow packet (due to ECMP or dynamic routing) is unknown beforehand (note that there are some notable exceptions, such as segment routing). If the per-flow customized data collection is required, configuring the data set for each flow at all data plane devices might be expensive in terms of configuration load and data plane resources. Req. 3 (Data Correlation): Due to the variable transport latency, the dedicated postcard packets for a single packet may arrive at the collector out of order or be dropped in networks for some reason. In order to infer the packet forwarding path, the collector needs some information from the postcard packets to identify the user packet affiliation and the order of path node traversal. Req. 4 (Overhead and Security): Since each postcard packet has its header, the overall network bandwidth overhead of PBT-M can be high. A large number of postcards could add processing pressure on data collecting servers. That can be used as an attack vector for DoS.

To address the above requirements, we propose several design details for applying PBT-M in MPLS networks.

To trigger the path-associated data collection, usually, a single bit from some header field is sufficient. The proposed action encoding is shown in using the MNA Sub-Stack formats defined in .

In the figure, three LSE formats defined in are used to form the Network Action Sub-Stack (NAS). Format A (labeled A) is the MNA Sub-Stack Indicator carrying the MNA bSPL (value 4). Format B (labeled B) is the Initial Opcode LSE; it uses the Flag-Based Network Action Indicators without Ancillary Data opcode (Opcode 1, Section 6.2 of ), sets the scope (IHS) for the whole NAS, and points (NAL=1) to one following Format D LSE. Format D (labeled D) carries the P-flag. No Format C LSE is needed, since a Format D LSE only requires a preceding Format A and Format B LSE (Section 4.4 of ). The PBT-M indicator (P-flag) is carried as a single flag in the Format D data field. If the bit is set to '1', a node is triggered to collect and export the telemetry data as configured by the control plane. Following Section 10 of , the PBT-M network action is defined as follows: Format: a Format D LSE following a Format A and a Format B LSE, where the Format B LSE uses Opcode 1. No Format C LSE and no Ancillary Data are used. Scope: Hop-by-Hop (IHS = 01), so that every on-path PBT-M-aware node processes the flag. The egress node is included in the HbH scope, and the penultimate node MUST NOT remove the NAS. Ancillary Data: None. The action is encoded as a single flag under Opcode 1. Processing: A PBT-M-aware node with data collection enabled generates and exports a postcard when the P-flag is set. A node that does not support this action skips it (U = 0). Interactions: PBT-M does not modify other network actions; it only triggers telemetry export. Because PBT-M marks only a subset of the packets in a flow, the value of the P-flag differs among packets of the same flow. Per Section 5.2 of , such mutable data MUST NOT be placed in the most significant 20 bits (the label value, which is always used for ECMP), nor in the most significant 23 bits when the TC field is also used for ECMP hashing, of a Format B, C, or D LSE; otherwise it may perturb ECMP load-balancing and cause out-of-order delivery within a flow. The P-flag is therefore placed at Bit Position 42 as numbered in Section 13.2.1 of , which falls in the trailing data bits of the Format D LSE (word bits 24-31). This position lies outside both the most significant 20 bits and the most significant 23 bits, so the marking bit does not affect ECMP load-balancing whether or not the TC field is used in the hash.

In case the path that a flow traverses is unknown in advance, all PBT-M-aware nodes should be configured to react to the marked packets by exporting some basic data, such as node ID and TTL before a data set template for that flow is configured. This way, the management plane can learn the flow path dynamically. If the management plane wants to collect the on-path data for some flow, it configures the head node with a probability or time interval for the flow packet marking. When the first marked packet is forwarded in the network, the PBT-M-aware nodes will export the basic data set to the collector. Hence, the flow path is identified. If other data types need to be collected, the management plane can further configure the data set's template to the target nodes on the flow's path. The PBT-M-aware nodes collect and export data accordingly if the packet is marked and a data set template is present. If the flow path is changed for any reason, the new path can be quickly learned by the collector. Consequently, the management plane controller can be directed to configure the nodes on the new path. The outdated configuration can be automatically timed out or explicitly revoked by the management plane controller.

The collector needs to correlate all the postcard packets for a single user packet. Once this is done, the TTL (or the timestamp, if the network time is synchronized) can be used to infer the flow forwarding path. The key issue here is to correlate all the postcards for the same user packet. The first possible approach includes the flow ID in the OAM packets. In case of MPLS, the MPLS label stack can serve as the flow ID. If the packet marking interval is large enough, the flow ID is enough to identify a user packet. As a result, it can be assumed that all the exported postcard packets for the same flow during a short time interval belong to the same user packet. Alternatively, if the network is synchronized, then the flow ID plus the timestamp at each node can also infer the postcard affiliation. However, some errors may occur under some circumstances. For example, two consecutive user packets from the same flow are marked, but one exported postcard from a node is lost. It is difficult for the collector to decide to which user packet the remaining postcard is related. In many cases, such a rare error has no catastrophic consequence. Therefore it is tolerable.

PBT-M should not be applied to all the packets all the time. It is better to be used in an interactive environment where the network telemetry applications dynamically decide which subset of traffic is under scrutiny. The network devices can limit the packet marking rate through sampling and metering. The postcard packets can be distributed to different servers to balance the processing load. Because PBT-M sends telemetry data by dedicated postcard packets, it allows data aggregation and compression. Each node can process the generated raw data according to the configured local data-export policies. Such policies may specify how raw data is used to calculate performance metrics, e.g., max, min, mean, percentile, etc.

Access lists with an optional sampler, , should be configured and attached at the ingress of the PBT-M encapsulation node to select the intended flows for PBT-M. Based on the requirements and node capability, the flow data could be exported at each transit node and at the end edge node with IPFIX .

The data decomposition can be achieved on the PBT-M-aware node exporting the data or on the IPFIX data collection. describes how data is being exported when decomposed at IPFIX data collection. When being decomposed on the PBT-M-aware node the data can be aggregated according to section 5 of .

In MPLS networks, Maximum Label Depth (MLD) is a great concern which limits the MNA size and in turn the OAM capability. Moreover, for SR-MPLS, Maximum SID Depth(MSD) as well as PMTU in SR Policy are critical issues for SR path instantiation by a controller. PBT-M can become a critical solution to ensure that flexible on-path telemetry can be viable for operators by eliminating telemetry data from being carried in-situ in the SR-TE policy path. This draft provides a critical optimization that fills the gaps with IOAM DEX related to packet marking triggers using existing mechanisms as well as flow path discovery mechanisms to avoid configuration of on path data plane node complexity and helps mitigate SR MSD and PMTU issues.

Only the ingress node is allowed to set these flag bits. The other on-path nodes can only react to the bit values. The tampering of these flag-based actions would result in DoS attack or unreliable measurements. Therefore, security measures must be taken to ensure the proper functioning of these actions. The security considerations of the MNA Sub-Stack in Section 11 of also apply.

This document requests IANA to allocate one bit position (TBA1, suggested value 42) for the PBT-M network action, with the description "PBT-M (Postcard-Based Telemetry with Packet Marking)", from the "Network Action Flags Without Ancillary Data" registry in the "MPLS Network Actions" registry group created by . Bit Position 42 falls in a Format D LSE (word bits 24-31), outside the most significant 23 bits of the LSE, so that the mutable marking bit does not affect ECMP load-balancing whether or not the TC field is used in the hash (see Section 5.2 of ). The registration procedure for this range (bit positions 20-439) is IETF Review.