]> JHU/APL

Edward.Birrane@jhuapl.edu

JHU/APL

Sabrina.Pellegrini@jhuapl.edu

JHU/APL

Alex.White@jhuapl.edu

int dtn Bundle Protocol Delay-Tolerant Networking Reliability The Delay-Tolerant Networking (DTN) architecture describes a type of challenged network in which communications may be significantly affected by long signal propagation delays, frequent link disruptions, or both. These unique and challenging characteristics require adapted approaches for data transport, security, management, routing, and other networking functions. Using new approaches, DTNs can offer data services in environments that would be considered neither reliable nor resilient as these concepts are understood for unchallenged networks. This document identifies adapted definitions and concepts for DTN reliability and resiliency that can be applied even when there is no simultaneous, reliable, end-to-end path between sources and destinations in the network. These definitions and concepts are suitable for any challenged environment but, in particular, those communicating using the DTN Bundle Protocol (BP). The ability to overlay BP across multiple discontinuous underlay networks, to store-and-forward data where and as necessary, and the structure and security of the BPv7 bundle itself provide novel ways to identify, implement, signal, and otherwise consider guarantees associated with data exchange in the DTN environment. About This Document Status information for this document may be found at . Discussion of this document takes place on the Delay/Disruption Tolerant Networking Working Group mailing list (), which is archived at . Subscribe at .

Introduction One benefit associated with in-network store-and-forward operations, such as those supported by the DTN architecture, is increased reliability of network operations in challenged environments. The intuition being that challenged networking environments may never provide a simultaneous, end-to-end path and data must either be stored in the network or dropped. Therefore, in-network data storage leads to more reliability than the alternative of dropping data. While intuitive, achieving this benefit in any particular network deployment is nuanced and would be informed by the overall networking architecture and individual networking device capabilities. For example, networking devices supporting store-and-forward operations require more resources for network processing and storage management. Networking architectures that use store-and-forward concepts in the overlay may see fewer benefits when individual underlay networks also provide store-and-forward services or alternative mechanisms to provide reliability. This document defines terms and concepts associated with networking reliability within challenged environments. These terms and concepts presume that DTN architectural capabilities and DTN protocol-specific features are present, because the DTN architecture is built to provide capabilities and features for challenged networking environments.

Terminology This section defines terminology that either is unique to the DTN architecture or is necessary for understanding unique concepts defined in this specification. In addition to the terms defined below, this document uses the following terms as they are defined in the DTN architecture and the Bundle Protocol specification : bundle, Bundle Protocol Agent (BPA), Application Agent (AA), Application Data Unit (ADU), Convergence Layer Adapter (CLA), and Convergence Layer Protocol (CLP). Delay-Tolerant Networking (DTN) :: (1) A networking architecture for tolerating expected and unexpected end-to-end delays and disruptions through the use of a networking layer and associated services built for that purpose. (2) Any network instantiation that conforms to the architecture, protocols, and mechanisms of the Delay-Tolerant Networking Architecture. In particular, any network implementing a bundle layer providing BP data units and BP-related services. DTN Availability :: The ability to access an ingress into a DTN. The time-varying nature of DTN topology does not require any end-to-end communications for the network to be available to a given user. Any time a user can pass data to the first hop of the DTN, the DTN is considered available for that user. DTN Data Durability :: The ability to prevent the loss or corruption of data in the network. This includes the preservation of data when otherwise impacted by failures (node loss, misconfiguration, and other impairments). DTN Reliability :: The extent to which the DTN can be relied upon to provide consistent, predictable data delivery. This is often a function of achieving data delivery deadlines and not maintaining instantaneous data rates. DTN reliability is achieved as a function of DTN resiliency, durability, and availability. DTN Resiliency :: (1) The extent to which a DTN continues to function in a variety of operating conditions. (2) The ability of the DTN to recover from interruptions caused by the loss of a supported operating condition. Convergence Layer Protocol Stack (CLPS) :: An integrated protocol stack consisting of exactly one CLP and zero or more supporting networking protocols, implemented by one or more CLAs. A given CLPS may share protocols with other CLPSs but may provide differing reliability and other guarantees as a function of protocol combinations or configurations.

Motivation for DTN Reliability The purpose of any data network is the reliable end-to-end exchange of user information in ways compatible with user-expected policies and deadlines. Therefore, the motivation for network reliability is to allow the network to perform the duties for which the network exists. In this way, the motivation for DTN reliability is no different. What is different is that DTNs are purpose-built to operate in environments that do not guarantee the assumptions upon which typical reliability mechanisms are predicated. Most data transport networks only allow data to exist in the network for a relatively short period of time. Data arriving at a network node must be dispositioned - usually within tens to hundreds of milliseconds - for forwarding, delivery, or deletion. In these networks, reliability is less about preserving any individual set of data than it is about detecting when data is lost or deleted and then retransmitting that data. Within the DTN architecture, data may exist in the network for longer periods of time because data arriving at a network node may be persistently stored for long durations. Within this architecture, reliability prioritizes preserving individual data because of the cost and risks associated with end-to-end retransmission. For this reason, more customized definitions and mechanisms for DTN reliability need to be provided when operating across networks or network segments that would otherwise never reliably communicate data.

User Benefits Beyond the standard benefit of end-to-end data exchange, a reliable DTN offers other user benefits that would otherwise be considered outside of the normal responsibilities of a networking layer. By incorporating store-and-forward behaviors, security-at-rest, and richer meta-data tagging, DTN nodes provide a central place to enforce certain beneficial behaviors that would otherwise be implemented on a per-application basis. Providing these behaviors in the network layer is both a motivation for using DTN and a motivation for defining DTN reliability to support these behaviors.

1. Keep Storage Available
2. Delegate Network Monitoring
3. Extend Platform Life
4. Preserve Data Policies

Keep Storage Available Storage is a limited resource for many devices operating in challenged environments. This may be due to the infrequency of network access, the difficulty of deploying devices in locations without access to regular power, and the cost of developing devices to function in extreme environments, among many other reasons. A reliable DTN allows users to manage their limited storage resources as a function of introducing data into the network, rather than confirming the receipt of data across the network. If a user application were able to rely on the fact that a network accepted data in accordance with some guaranteed reliability service, then the application might be comfortable removing that data from its own storage, even before the data crosses the network. In this way, the delays and disruptions incumbent in operating in challenged environments might not have the same negative impact on user storage as would be the case without reliable delivery.

Delegate Network Monitoring Challenged networks may experience significant topological evolution over time. This evolution might be based on predicted impairments, such as signal propagation delays, occultation, and power duty cycling. However, this evolution might also be based on unplanned occurrences such as device failures, emergency operating conditions, and unexpected changes to the operating environment. Users of these networks expect that any partitioning or other disconnectivity of the network remain hidden from their primary network interfaces and that data policies remain enforced while data is in either active transit or at rest. Unlike timely networks where node loss can be accounted for through over-provisioning resources and path diversity, a challenged environment might need to buffer data when message endpoints exist in separate network partitions. The ability for the network to wait on appropriate connectivity without requiring any action from the user is, itself, a user-desired behavior. In timely networks, by contrast, network partitioning is handled as a network error requiring users to attempt network communications again at a later time.

Extend Platform Life User platforms, especially those harvesting operating energy from their environment, can extend the operational life of their platform and otherwise increase the duty cycles of their platforms, by making more efficient use of their on-board resources. One of the more power-intensive applications on most sensing platforms is a telecommunications system. Keeping telecommunications systems powered and operating to account for delivery acknowledgements, handle retransmissions, and generally to support bi-directional, "chatty" communications protocols can be a significant energy drain on a user platform. For example, the telecommunications system on a deep-space spacecraft can be the single largest energy consumer on that spacecraft. A reliable DTN can minimize data retransmissions by removing from the user platform the need to remain the retransmission source for any data generated by that node. Similarly, a DTN can increase the overall goodput of a network by reducing the number of individual messages that would need to be communicated based on the ability of DTNs to carry multiple types of secured information in a single network PDU. By allowing more efficient use of telecommunications systems (and the processing, memory, and storage associated with those systems) a reliable DTN can help platforms make more efficient use of resources, extend overall platform life, and minimize the impact of duty cycles for the platform itself (or any components).

Preserve Data Policies Policy-based traffic engineering allows users to express complex behaviors associated with their data flows, to include which parts of the network may carry data and what network services are to be applied to that data while in transit. This policy enforcement is usually applied at the interface between a user device and some device representing the "edge" of the network. In cases where data might live within the network long enough for the network to undergo significant topological change, policy expression might need to be examined and applied by more than the network's edge node. A reliable DTN is one that not only preserves data transmission during periods of delay and disruption; it also ensures that data policies remain enforced even when the path through the network changes. This includes determining when user data is no longer considered valid in the network and expiring that data when it is considered stale.

DTN Architectural View A defining characteristic of the DTN architecture is the "bundle layer", which forms a delay- and disruption-tolerant overlay interconnecting one or more underlay networks. Services provided by the bundle layer include routing between and among underlay networks, implementing store-and-forward services where and as required, policy-based management, quality-of-service, and fine-grained, intra-bundle security mechanisms. The protocol data unit of the bundle layer is the "bundle" as described in the Bundle Protocol . The bundle layer is not the only layer presupposed by the DTN architecture and, therefore, not the only layer that has the opportunity to contribute to the overall reliability of the network. Understanding that the DTN architecture remains a layered architecture, and that layers beyond the bundle layer may provide reliability, is fundamental to determining when and how to use various reliability mechanisms. This section provides a brief definition of the layering responsibilities of the DTN architecture, with a focus on the services provided by these layers and the types of faults that can occur at these layers.

Architectural Layers Because the DTN architecture requires a bundle layer, the minimal set of layers in the architecture comprises the bundle layer and any additional layers required to support it. Generally, this leads to an architectural decomposition of at least four layers, as follows.

1. Application Layer: An application layer that uses bundle layer services for the exchange of application (user) data.
2. Bundle Layer: The bundle layer, implemented by BPAs, providing bundle services.
3. Convergence Layer: A convergence layer, using CLPs and associated services to adapt bundles for transport over an underlay network.
4. Underlay Layer: An underlay network layer providing transmission of bundles between and amongst BPAs.

A simplified layer view, and its mapping to the logical Bundle Node described in , is illustrated in .

Architectural Layers

The remainder of this section provides an overview of each of these layers, to include

1. A brief description of layer responsibilities
2. Services expected to be available
3. Faults that may be encountered

Application Layer The application layer consists of the set of entities serving as messaging sources and destinations for traffic in the bundle layer. This can include user applications that directly produce and consume bundles as well as bundle proxy applications that aggregate non-bundle user traffic for the purpose of communicating those data as bundles across the bundle layer. The design and purpose of applications in this layer are not relevant to the discussion of DTN reliability other than to note that applications must account for longer delays, out of order ADU delivery, and losing expired data. While the DTN architecture mitigates the risks of operating in a challenged networking environment, it does not (and cannot) ensure that (even temporally discontinuous) paths ever exist between a source and destination within some data lifetime.

Relevant Application Services The only relevant service of the application layer, with respect to DTN reliability, is the exchange of Application Data Units (ADUs) with the bundle layer.

ADU Transmission Every application in this layer, either directly or through a proxy, must create an ADU that can be communicated to the bundle layer for the purpose of having that ADU placed into one or more bundles and transmitted across the DTN. In doing so, the application provides information necessary for the creation of the bundle, to include information which either directly identifies, or indirectly supports the BPA derivation of, the following information.

1. The destination of the ADU and the application expected to receive it.
2. Any special policy designations that must be honored in the creation or processing of bundles carrying all or part of the ADU.
3. The useful lifetime of the data in the network, after which the data can be deleted.
4. Any security information necessary to validate the identity of the application to the BPA.
5. Any acknowledgements needed to confirm to the BPA that ADUs were successfully received.

ADU Receipt Every application in this layer, either directly or through a proxy, must also be able to receive ADUs as assembled and communicated from an underlying BPA. In doing so, the application provides information necessary to help a BPA determine how and when ADUs may be delivered, to include information which either directly identifies, or indirectly supports the BPA derivation of, the following information.

1. The identification of the application as a destination for ADUs.
2. Any security information necessary to validate the identity of the application to the BPA.
3. Any acknowledgements needed to confirm to the BPA that ADUs were successfully received.

Application Faults The logical entity within a bundle node that interacts directly with a BPA is known as the Application Agent (AA). The AA acts as an intermediary through which Sender and Receiver applications communicate with the BPA. Faults related to this mechanism include errors with the exchange of Application Data Units (ADUs) and associated status reporting. Besides the clear faults of loss of connection with either the user application or the BPA, the following types of faults can also be encountered:

1. Untrusted Application. Application connections are rejected due to authentication or configuration issues.
2. Malformed Request. The BPA rejects an ADU as a function of policy, implementation limitations, or misconfiguration of metadata associated with the ADU.
3. Offline Application. The AA is unable to deliver an ADU received from a BPA to an application because the application is not online.

The Bundle Layer The bundle layer is a unique and defining element of the DTN architecture. This section describes those bundle services that might increase the overall reliability of the networks where they are implemented.

Relevant Bundle Services The set of services expected to exist in the bundle layer are provided in detail in . However, certain capabilities of the BPv7 bundle (as a PDU) and the behaviors of a BPA in processing that PDU, enable reliability mechanisms. These services are store-and-forward behaviors, time-variant routing, and PDU extensibility and augmentation.

In-Network Store-and-Forward The store-and-forward behavior implemented in the bundle layer allows for the option to hold bundles at Bundle Protocol Agents (BPAs) in the network. This behavior allows bundles to be communicated across various underlay networks between BPAs and stored at BPAs pending forwarding opportunities or transmission acknowledgements. Storage, in this scenario, includes the ability to hold bundles through a reboot or other fault management that might occur at the BPA or any device hosting a BPA. The amount of storage provided by any given BPA is expected to vary as is the access to storage for any given bundle. Generally, how bundles are stored by a BPA is expected to be governed by network management. Store-and-forward behaviors increase the reliability of a network in cases where data would otherwise be dropped. The most likely causes of dropped data are instances where a BPA uses an underlay network to carry a bundle to its next BPA and the underlay network drops the data. It is expected that underlay networks will drop data if the underlay network is either disconnected or unreliable. In these cases, were the BPA to store the bundle, the bundle could later be retransmitted over the same underlay or over a different underlay. In extremely challenged networks that never have simultaneous, end-to-end paths the only option for data delivery is store-and-forward.

Time-Variant Routing There are multiple reasons why a preferred path for a bundle might change over time, including preserving platform resources, increasing operating efficiency, and reacting to dynamic reachability . In cases where these changes are unplanned, route computation at the bundle layer would neither have knowledge of the instantaneous topology of the network nor understand how the topology might change over time. To the extent that BPAs detect and react to changes in topology over time, the bundle layer will need to reason about (and recalculate) routes over the lifetime of the bundle. Time-variant routing behaviors increase the reliability of a network in cases where different paths exhibit different reliability. For example, a time-variant route might be calculated that avoids the use of a currently-available-but-unreliable underlay network connection to wait for a later-available-and-reliable underlay network connection. Alternatively, routes could be selected that mitigate forecasted congestion at BPAs or otherwise select routes that preserve some policy-mandated behavior (such as BPA storage availability).

PDU Extensibility and Augmentation BPv7 bundles carry varied information in various bundle block types. Networks may define private-use blocks for annotating bundles resident in their administrative domain and, generally, new standardized block types may be defined through normative standards body actions. In addition to defining new block types, BPAs may add, remove, or update certain extension blocks in a bundle while the bundle is being processed by the BPA. This block processing may happen at BPAs acting as the bundle source, the bundle destination, or any bundle waypoint in the network. Extending the information in a bundle through the definition of extension blocks increases the reliability of the bundle in the network because it allows for ways to signal (in-band) reliability information for both the bundle itself and the overall bundle layer. Augmenting existing bundles in the network allows BPAs to react to changes in bundle handling in ways that increase the likelihood for eventual bundle delivery - such as changes to policy, network status, or other processing information that might not have been known at the time of bundle creation. The ability of the bundle layer to secure individual blocks individually, as outlined in , also provides additional reliability in the sense that malicious or misconfigured data could be detected and removed from the bundle layer to avoid congestion or other resource contention for properly credentialed network traffic.

Bundle Layer Faults The BPA is responsible for creating bundles from a given ADU, delivering ADUs from a received bundle, and otherwise processing bundles that are transiting through the BPA itself. The BPA is the source of any bundle processing related faults as this is the sole logical entity tasked with bundle processing. Faults in this area can be organized into creation, processing, and delivery faults.

Bundle Creation Faults related to bundle creation impact the AA requesting bundle creation. This includes both application-based senders as well as the creation of administrative bundles from the administrative element within the AA. BPA faults preventing the creation of a bundle include the following:

1. Malformed Bundle. The bundle would violate some configured limits or policies and cannot be created.
2. Incomplete Information. The AA provided insufficient information to create the bundle, such as malformed destinations.
3. Untrusted AA. The AA is not validated as able to request bundle creation.
4. Unsupported Services. The BPA is unable to provide the services necessary for processing the bundle.

Bundle Processing Processing, in this context, refers to all actions taken by a BPA at a source BPA, some intermediate BPA, or destination BPA. This processing happens after creation of the bundle at a source, after reception of a bundle from the convergence layer, before sending the bundle to a next hop using the convergence layer, and before delivering the ADU of a bundle to the AA at the destination node. Categories of faults related to bundle processing include the following.

1. Resource Exhaustion. Bundles might be kept at a node longer than expected due to lack of available, appropriate next hops, and dropped when node storage is exhausted.
2. Bundle Mangling. Due to policy or other mis-configuration, bundles may have extension blocks or other modifications that make them difficult or impossible to process downstream. This includes errors related to incorrect fragmentation or application of security.
3. Platform Error Loss. Bundles might be removed from the node unexpectedly due to implementation error or technical fault on the BPA itself - to include the loss of the BPA or the loss of the processing node.
4. Bundle Duplication. Implementation errors in a BPA might result in the creation of bundles that violate rules for uniqueness in the primary block. This can happen, for example, on systems where clocks are being manipulated or where sequence counters/nonces are reset.
5. Wrong CLA. A BPA might transmit/forward a bundle using an inappropriate CLA, as a function of mis-configuration or other error on the BPA.
6. Unsupported Service Loss. A special case of bundle loss where elements of a bundle cannot be processed by a BPA and the BPA drops the bundle due to bundle processing flags.
7. Extension Processing Loss. The bundle contains extension blocks that are in violation of BPA policy, or otherwise cause extension blocks or the bundle itself to be removed from the network.

Bundle Delivery Delivery, in this context, applies to any bundle processing that occurs at the destination BPA of a bundle. Destination processing represents a slightly different circumstance than other bundle processing because it involves resources and status exchange local to the node that is also running the AA. In addition to the regular faults that can occur as part of bundle processing, the following types of faults are unique to the delivery process.

1. Reassembly Failure. A deliverable ADU cannot be extracted from a series of bundles each holding ADU fragments. This is separate from issues reassembling a bundle up to the bundle layer from the convergence layer.
2. ADU Handoff Loss. The BPA is unable to send an ADU to an AA. This can happen if the AA cannot resolve the service/application that is the recipient of the ADU, or if the AA generally rejects the ADU as a matter of policy or configuration.

The Convergence Layer The BPv7 specification summarizes "Services Required of the Convergence Layer" () in which the "Convergence Layer" is populated by "underlying protocols" accessed via one or more "Convergence Layer Adapters" (CLAs). Each CLA implements (at a minimum) a Convergence Layer Protocol (CLP) that defines how to accept bundles from a BPA, send received bundles to a BPA, and otherwise report delivery status and metrics via some other networking protocol. For example, CLPs are defined for TCP , LTP, UDP, QUIC, and others. Separate from a CLP, other networking protocols may be implemented within the convergence layer - particularly when the underlying network being used by the convergence layer does not provide certain desirable features such as rate limiting and congestion control. While presumes the existence of multiple CLPs it does not otherwise discuss the existence of entire protocol stacks within the convergence layer. This is an important consideration as reliability (and other features) are typically achieved through the combination of multiple protocols in a protocol stack. To clarify how we discuss expectations within the convergence layer, we define the concept of a Convergence Layer Protocol Stack (CLPS). The "top" of any CLPS is a CLP and the "bottom" of any CLPS is the protocol accepted by the underlay network used by the convergence layer. In addition to the CLP, the CLPS may include zero or more other networking protocols providing a variety of capabilities. This section seeks to clarify multiple aspects of the convergence layer that are otherwise not directly addressed by . Particularly the following:

1. A convergence layer may support multiple CLPSs.
2. Each CLPS is differentiated by the composition of protocols in the stack and/or the configuration of protocols in the stack.
3. Reliability and other characteristics in the convergence layer are properties of a CLPS, and not necessarily the property of the CLP or any other individual protocol in the stack. For example, a CLPS may provide congestion control and rate-limiting behavior. This should not be interpreted as a requirement that every protocol in the CLPS provides congestion control and rate-limiting.

A simplified view of these elements is illustrated in and discussed further below.

Adapters and Protocol Stacks Protocol Stacks +-------+ | +-------+ / | Pn | | | Pn | / +-------+ | +-------+_/ | | | +-------------------------------+ | Underlay Network Protocol | +-------------------------------+ ]]>

Relevant Convergence Services The BPv7 specification () outlines the following convergence layer services:

1. Sending a bundle
2. Notifying the BPA upon the disposition of data-sending procedures
3. Delivering a bundle

Convergence Faults Faults that can occur in this layer include the following:

1. Malformed Bundle CLA Rejection. The CLA might refuse to accept a bundle that is malformed.
2. Policy-Induced CLA Rejection. A CLA might refuse to accept a bundle as a function of its own policy, such as if the bundle exceeds maximum sizes, uses EID schemes unknown to the CLA, or includes sources or destinations unknown or unsupported by the CLA. This might also include cases where the BPA and CLA are unable to establish trust/authentication mechanisms between them.
3. CL Processing Loss. The CL stack might fail to encapsulate a bundle as it processes through a CL stack. This may be due to mis-configurations or policy violations of the individual CLPs in the CL stack, malformations due to implementation errors in any implementation of the CLAs in that stack, or general errors or loss of the platform(s) implementing these CLAs and CLPs. This loss may happen both when sending or receiving bundles through the CL.
4. Network Rejection. The CL may fail to transmit the adapted bundle if the underlying network rejects the adapted bundle for any reason.

The Underlay Layer Any network capable of transmitting BPv7 PDUs (or portions of BPv7 PDUs carried in CLP PDUs) can act as an underlay network for the purpose of communicating bundles between two or more BPAs. There are no restrictions on the types of underlay networks that can be used, and there is no restriction that such underlay networks only implement specific features or functions. Part of the power of DTN as an overlay across heterogeneous networks lies in its ability to work across vastly dissimilar underlay networks. For example, all of the following may be considered underlay networks within the DTN architecture.

1. The terrestrial Internet.
2. A single hop between a transmitter and a receiver (such as between a spacecraft and a ground station).
3. A CCSDS constellation network using only framing protocols.

Relevant Underlay Services Individual underlay networks may support a large number of network services and associated service level agreements. From the perspective of DTN reliability, certain characteristics of the underlay network are more important when selecting which underlay network should be used for transmission and which services from the bundle and convergence layers may need to be relied upon to cover gaps in underlay network services.

1. The overall security of the underlay.
2. The reliability (to include resiliency and availability) of the network.
3. Features such as delivery acknowledgements and in-order delivery.

Underlay Faults Several faults can occur at this layer while trying to communicate between two BPAs. From the point of view of the network, CLP PDUs containing portions of a bundle are simply network traffic. Therefore, the faults associated with the underlay network are faults associated with any of its network traffic and not specific to the fact that the traffic includes bundle information. These faults include the following.

1. Lost Traffic. The network loses traffic for a variety of reasons, to include use of non-reliable architectures, links, or protocols otherwise hidden from the BP layer. This can also include loss of network components necessary for network data exchange.
2. Unexpected Traffic Delay. Network traffic is delivered, but over a longer time period due to delays caused by retransmissions, security or other round-trip exchanges, or as a function of priority/service priority for the network.
3. Traffic Corruption. Traffic is received but corrupted due to processing while in the network. This includes both intentional and unintentional sources of data corruption. Corrupted traffic might lead to traffic loss when the corruption is detectable and unrecoverable.
4. Improper Delivery. At a receiving network node, traffic might be incorrectly delivered to the CL responsible for extracting a bundle which would cause the bundle to never be sent to a BPA.

DTN Reliability The most important characteristic of a network is that it can be relied upon to communicate user data within performance, security, timeliness, and other user requirements. The overarching networking characteristic capturing this behavior is the reliability of the network. Any operational network needs to be reliable with respect to network-specific and user-specific requirements. Reliability is not an absolute characteristic of a network for two reasons. First, it is defined and applied in the context of network users and their associated data requirements. Therefore, no one set of network behaviors might be "reliable" for all users. Second, as a practical matter, pathological topologies, error conditions, or other issues can be encountered that impede networked data exchange. Therefore, reliability is often considered in the context of some set of supported operating conditions. One way to describe this nuanced view of reliability is to consider other network characteristics that contribute to the overall ability to exchange user data. These other characteristics include network resiliency, availability, and durability. The concept of reliability in a DTN needs to be defined in a way that is independent of end-to-end latency.

Reliability Definitions This section elaborates on the resiliency, availability, and durability terms defined in the Terminology section, as they contribute to overall DTN reliability.

Resiliency Network resiliency is the extent to which a network continues to function in a variety of operating conditions. Networks with little resiliency may operate effectively well within nominal operating conditions, but rapidly fail as the environment changes, such as due to component failures, attacks, user congestion, and other impairments. Inherent in any networking design is an analysis of the potential operating conditions through which a network is expected to function. Not every network is expected to operate in every operating condition, simply as a matter of practicality. Mechanisms for achieving resiliency are numerous, and include designing redundancy and diversity into the networking architecture, increasing the number and performance of networking devices, and deploying various data, management, and control protocols and associated algorithms. Defining resiliency as tolerance for operating conditions provides a useful way to assess proposed resiliency mechanisms.

Availability Network availability is the extent to which the network is able to accept user traffic at an ingress. Networks with little uptime may only be able to accept user traffic at specific times or under specific conditions. Networks with little availability may be both resilient and reliable during those periods of availability. Availability can be defined both as an absolute measure or relative to known usage patterns. Absolute availability (often measured through network uptime) is used when user traffic is omnipresent or otherwise when user traffic patterns are unknown but possible at any time. Relative availability is used when it is understood that user traffic will not exist at certain periods, or when the network itself undergoes planned downtime.

Data Durability Network data durability is the extent to which information in the network is protected from loss and corruption. Networks with little data durability may be resilient and available, but if data is corruptible either during transmission or when being processed in the network, it will not be durable. Durability, as defined here, encompasses data integrity, which may or may not be tied to data security: data integrity involves the ability to detect (and, where possible, preserve data from) corruption in the network, while data security includes mechanisms for implementing integrity protection using cryptographic means.

Reliability Signaling Network reliability (and associated characteristics) are always considered in the context of network user outcomes. For networks purpose-built for a particular user community there is a direct correlation between network characteristics and user needs. For networks with diverse users, network characteristics are associated with user traffic through network configurations and policies. Therefore, discussions about what mechanisms are needed to implement what levels of reliability (and thus resiliency, availability, and durability) need to occur in the context of what user outcomes need to be preserved. A common approach to providing these indicators is to build networks whose end-to-end latency is smaller than the timing requirements for providing indicators back to the user. In such cases, the success (or failure) of a given data exchange can be reported with certainty as it has already happened. This approach cannot generally be used in networks conforming to the DTN architecture because the expected operating conditions of the network cannot guarantee end-to-end latency smaller than user timing requirements. Instead, DTNs might need to provide user indicators based on processing at the source node itself, and without an understanding of end-to-end performance across the DTN. This section defines user expectations of indicators and subsequent sections describe how these indicators may be different in the context of DTN deployments. Expected user outcomes take the form of indicators provided by the user to the network (and confirmed by the network to the user). Some indicators that impact decisions relating to network reliability include the following.

1. Expected services and user mapping
2. Traffic acceptance and rejection
3. Delivery acknowledgements

Expected Services and User Mapping User expectations for services are often captured in various agreements between user communities and network operators. These often take the form of Service Level Agreements (SLAs) where services are pre-negotiated as part of supporting user traffic in a network. In situations where a network is custom-built for a user community, these services may simply be as designed into the network architecture. In situations where networks service multiple users, this may be negotiated dynamically. Networks need to provide a way to map users to their expected service agreements, and may need to indicate to users that their traffic has been accepted at the expected service level.

Traffic Acceptance and Rejection User traffic, when provided to the network, needs to be either accepted or rejected, and users may expect to be informed of both. User data is usually accepted by the network if it can be associated with a known user, if the network is known to be in an operational state, and if there exists knowledge that the user data can be communicated over the network within various user requirements. Signaling acceptance is often an implementation matter (e.g. through the programming interfaces by which user traffic is presented to the network). User data is rejected by the network whenever it cannot be accepted. This often occurs for reasons such as failing to authenticate the user traffic or match it to known user service requirements, situations where the network is not operating, or otherwise when there is no known path to the user traffic destination. Similar to acceptance, how rejection is communicated to the user is an implementation matter, but often includes some reason for the rejection. These indicators are important, as rejection of user traffic from a reliable network requires some action by the user. Therefore, signaling acceptance and rejection in a timely manner is important for many application operational concepts.

Delivery Acknowledgements In certain cases positive acknowledgements (ACKs) or negative acknowledgements (NACKs) are expected by users from the network for certain types of traffic. This is separate from application-space protocols that exchange request-response messages above the scope of the networking layer. These acknowledgements are usually reserved for critical information flows.

Reliability and Layering The reliability of the DTN architecture does not come from a single layer and, therefore, no single layer should be expected to account for every possible reliability consideration. The total set of services available across the application, bundle, convergence, and underlay network layers form a vast set of capabilities from which network engineers, architects, and operators can select to build networks appropriate for their deployment environments. This section summarizes how each of these types of reliability mechanism can increase the reliability of the DTN architecture when viewed from the perspective of a given layer.

In-Band Exchanges In-band reliability information consists of diagnostic information exchanged with other peers at a particular layer in a network. In this viewpoint, in-band refers to the fact that diagnostic information is exchanged within the particular network layer, not that it must all be exchanged by a single protocol operating at that layer. The content of diagnostic information varies as a function of mechanism, but common types of such information include the following.

1. Acknowledgements (ACKs) indicating delivery of one or more messages.
2. Negative acknowledgements (NACKs) indicating missing messages.
3. Statistics relating to number of bytes of messages delivered.
4. General health of sessions exchanging messages that are encrypted or otherwise opaque.

Out-Of-Band Exchanges Separate from reliability information that comes from other peers within a given layer, information can be communicated across layer "boundaries" in the form of out-of-band information. The types of out-of-band information that can inform reliability mechanisms include the following.

1. Configuration and policy updates from operations and orchestration entities that consider cross-layer and other inputs.
2. Information provided from the management and control planes of protocols operating at other layers. This includes metrics, acknowledgements, and other signaling from different layers in the architecture.
3. Backpressure and other data plane information from other layers as a function of cross-layer data exchange.

Delegation In cases where there is implicit trust associated with a given integrated platform, it is possible that no positive reliability signaling is needed beyond (optional) indications that data was received by another layer in the network architecture. For example, it may be the case that if data is accepted by a lower layer (without rejection or negative acknowledgement) that the data is presumed to be handled in accordance with the capabilities of the accepting layer, without additional in-band or out-of-band signaling.

Sample DTN Deployment The concept of reliability as a function of the overall architecture - and not from a specific layer - is illustrated in . The example network consists of three Bundle Nodes (A, B, and C) with Bundle Node A representing a bundle source for some sending application (Sender), through an intermediate Bundle Node B, to the bundle destination at Bundle Node C which delivers the bundle to some receiving application (Receiver). Bundle Nodes are connected by two different underlay networks, with Networks 1 and 2 connecting nodes A and B and Networks 3 and 4 connecting nodes B and C. Networks 1 and 3 are considered reliable (meaning they provide guaranteed delivery of accepted traffic) whereas Networks 2 and 4 are considered unreliable. Altogether this network is suitable for discussing the end-to-end reliability as a function of characteristics and events present at bundle sources, bundle intermediate nodes, and bundle destinations across both reliable and unreliable networks.

Exemplar DTN | BPA || | | AA |<->| BPA | | || AA |<->| BPA || |+----+ +-----+| | +----+ +-----+ | |+----+ +-----+| | ^ | | ^ | | ^ | | | | | | | | | | | +------+ | | +-----+---+--+-----+ | | +------+ | | | | | | | | | | | | | | | | +-v-+ +-v-+ | | +-v-+ +-v-+ +-v-+ +-v-+| | +-v-+ +-v-+ | | |CL1| |CL2| | | |CL2| |CL1| |CL3| |CL4|| | |CL4| |CL3| | +--+-^-+--+-^-+--+ +-+-^-+-+-^-+--+-^-+-+-^-++ +--+-^-+--+-^-+--+ | | | | | | | | | +-v-----------v-+ | | +-v----------v-+ | | | Network 2 | | | | (Unreliable) | | | | (Unreliable) | | | | Network 4 | | | +---------------+ | | +--------------+ | | | | | +-v------------------------v-+ +-v-----------------------v-+ | Network 1 | | Network 3 | | (Reliable) | | (Reliable) | +----------------------------+ +---------------------------+ ]]>

The flow of information provided in between the network Sender and Receiver can be evaluated to identify various places where faults can occur. Each of the processing boxes in this illustration, from the AA associated with the Sender to the AA associated with the Receiver, can inject faults or errors in the system. This includes boxes associated with the "networks". As one example of reliability arising from multiple layers, consider a bundle transmitted from Node A to Node B over Network 2 that is lost in transit. If Node A retains the bundle in storage and the CLPS associated with CL2 reports the transmission failure, Node A may retransmit the bundle over Network 2 or select the reliable Network 1 instead. If, instead, Node A released the bundle from storage upon transmission, the loss is unrecoverable at the bundle layer, regardless of what services Network 2 provides. The reliability of the end-to-end exchange is, therefore, a joint function of bundle-layer retention behavior, CLPS status reporting, and underlay network characteristics. The reliability classes in the next section formalize the retention and release behaviors that make such recovery possible.

Reliability in the Bundle Layer Because DTNs will vary in their individual complexity and requirements, not all DTNs need the same levels of reliability. Some data service needs will require end-to-end reliability, more complexity, more resource utilization, or more mission support than others. Users will need to determine which mission-impacting behaviors need to be realized for which data flows and decide which reliable services at what layer best suit their networks. Allocating these options into different "reliability classes" at different layers of the network provides a natural way to express these needs within the service agreement construct. This is similar to what is already used in terrestrial networks to map user expectations to network mechanisms. These classes must be defined differently depending on the network layer being discussed.

Custodial Behaviors Custody transfer is one method that can be used to increase network reliability at the bundle layer. It describes the transfer of responsibility for retransmitting a message at intermediate nodes or "custodians" in a network. A custodian is the node that acquires the responsibility to ensure data delivery. In short, a source node transmits data to a next hop that accepts custody of the data, thereby becoming the new custodian for that data. As a result, the original data source may go offline or delete the data locally because the new custodian is now responsible for the retransmission of the data. It is assumed that a topology with just one custodian (typically the source node) is more reliable than one with no custodians. It is also assumed that in a topology with more than one custodian, the new custodian is more likely to deliver the data to the final destination than the previous custodian. This could be because the new custodian has more "reliable hops" to the destination than the previous custodian, or it has fewer hops in general - therefore decreasing the probability of disruption prior to delivery. By allowing for these assumptions, the reliability of data exchange across a network increases with the implementation of custody transfer and increases with the number of designated custodians. The set of actions which permit and enable custody transfer are custodial behaviors. There are two primary enabling custodial behaviors:

1. Storing a bundle.
2. Releasing a bundle when release conditions are met.

A custodian must be capable of storing a bundle until the data is expired, the data is delivered, or the custodian is relieved of its custodial duties. A custodian must then also recognize the conditions to be relieved and release or delete the bundle accordingly. For this release to happen, the current custodian must receive a signal of acknowledgement from the new custodian or the destination. Depending on the network, the number of signals required for release may vary. For example, if a network is set up to provide multiple, parallel custodians at one time, a custodian may not release bundles from storage until a certain number of new custodians have acknowledged responsibility. Due to the variable levels of reliability that may exist across a given network, networks can be categorized by the different reliability classes defined in the next section.

Reliability Classes at the Bundle Layer At the bundle layer, these classes are defined based on three characteristics inherent to bundle-layer reliability mechanisms and the custodial behaviors described above. This class structure extends the custody transfer analysis presented in . The following variables are used to summarize bundle layer reliability classes:

1. Minimum number of acknowledged downstream custodians necessary for data release (MAD). This minimum applies at each node currently holding custody of the data, including the source node.
2. Ability to store a bundle until there is an egress link (SAB - "Store A Bundle"). This value is TRUE or FALSE.
3. Ability for the destination node to hold data until the ADU is delivered (HAD - "Hold At Destination"). This value is TRUE or FALSE.

This section defines four classes of reliability services selectable by users (such as space missions) in the context of a data-flow-specific service agreement, as we would expect them to be applied at the bundle layer. These classes are enumerated (0-3).

Class 0: Best Effort The class 0 service expressly prohibits the use of store-and-forward or associated custodial behaviors for any traffic associated with this service. Defining a service class for the sole purpose of prohibiting reliability is useful because it allows network nodes to short-circuit a variety of evaluations associated with data storage, buffer management, security processing, time-variant route computations, and other policy-based considerations. DTN nodes may treat this type of data in the same way that terrestrial routers treat data, even when that data is carried in BPv7 bundles: data in this class can be delivered, forwarded, or dropped. Users benefit from this class of service in a variety of ways, such as:

1. Data associated with this class of service may be provided at a lower cost to users.
2. Overall throughput may be faster through a network because the network applies less inspection on the data, with less need to process storage.
3. User assets will not need to wait for the network to acknowledge that data have been accepted.

Key Summary of Class 0:

• MAD = 0 (no custodial behaviors)
• SAB = FALSE
• HAD = FALSE

Class 1: Store-Until-Forward The class 1 reliability service provides a best-effort, store-and-forward delivery of data similar to that described in BPv7. In this class of service, DTN nodes will accept data and buffer them within some constraints as a function of in-band and out-of-band policy waiting for an appropriate next transmission of the data. Within this class of service, the MAD variable is 0. In other words, there is no acknowledgement required by a node for the release of a bundle from its storage because there are no custodians. Instead, the term retention in this reliability class implies the persistence of user data at a node if necessary for the transmission of data to its next hop. This means the SAB (storing a bundle) parameter is set to TRUE. For example, when data is received by a node, a determination is made as to whether the data can be immediately transmitted or not. If data can be immediately transmitted, then there is no need to store it, and it can be sent directly to a transmitter. Alternatively, if it can't be immediately transmitted, the node will need to determine if there is storage available. If so, it can be stored pending a future transmission opportunity. If there is no storage, or if a bundle expires or is otherwise expunged from storage, then the bundle will be lost. Key Summary of Class 1:

• MAD = 0 (no custodial behaviors)
• SAB = TRUE
• HAD = TRUE or FALSE pending user requirements

Class 2: Guaranteed Custodian The class 2 reliability service provides the guarantee that there exists, within the network itself, at least one node configured to serve as a custodian of data. The implication for this class of service is that, unlike the best-effort retention of a Class 1 service, the network will provide storage. The amount and nature of storage provided by the network is expected to be customized to individual user data flows. The way in which a network provides this guarantee is expected to be opaque to the network user, but generally there is some serialized order in which one or more nodes will accept custody of data as it works its way through the network. This might mean that there exists only one node in the network that is configured to serve as a custodian, or it might mean that networks may implement progressively updating custodians along a user data path, where each hop in the network is configured to serve as a custodian. Alternatively, it is possible that network service orchestration tools choose common custodians as a function of shared destinations, where the network defines certain nodes as custodians as a function of topological significance, available storage, or other policy inputs. This implies the data paths can go through any part of the network between custodians as necessary. In all of these cases, class 2 describes a scenario where a custodian must only receive one acknowledgement from a new custodian to release stored data and be relieved of its custodial duties (MAD = 1). It does not assume there are parallel paths and parallel custodians for increased reliability. Additionally, in each of these example cases, the user source and destination nodes do not require knowledge of how or where custodians are placed on a network. Key Summary of Class 2:

• MAD = 1
• SAB = TRUE
• HAD = TRUE or FALSE pending user requirements

Class 3: Independent Custodians The class 3 reliability service provides a guarantee that the network will assign multiple, parallel custodians within the network so that if there is an issue with a single custodian, data may still be able to be delivered within its lifetime and other policy constraints. There are a variety of reasons why a particular path or single custodian might not deliver data for a particular user, including the following. 1. Custodian Loss: Space remains a harsh environment, and space-based custodians may be located in spacecraft or other assets that experience catastrophic failure. If a single custodian in the network fails, then all untransmitted data on that custodian is lost. 2. Network Partition: The time-variant nature of interplanetary topologies can be unpredictable, especially when nodes in the network experience recoverable errors or other unplanned outages. When this occurs, a custodian may be isolated or disconnected from the rest of the network for a long enough period that the stored data expires and is removed from the node. 3. Configuration Changes: Changes to configurations at a custodian (to include misconfigurations) can introduce processing errors, especially as they relate to the security processing of the data. Additionally, policy changes may impose policy-based routing and retention decisions on a node that can result in loss of retained data. In these cases (and many others), very critical data can be lost if there is only a single custodian for data at a time. To increase the likelihood of successful data transfer, parallel custodians can be required such that a node holding custody of user data (to include the user source node) would need to receive confirmation that two or more custodians have actively accepted user data prior to removing those data from storage (MAD = 2+). To provide parallel custodians, a network offering a class 3 service replicates data toward multiple candidate custodians, ideally along disjoint paths, so that at least MAD custodians hold the data simultaneously. Because replication (and not only the acknowledgement threshold) is what provides tolerance to custodian loss and, potentially, faster delivery over the first path to reach the destination, a class 3 service is a statement about both custody release conditions and network forwarding behavior. Key Summary of Class 3:

• MAD = 2+
• SAB = TRUE
• HAD = TRUE or FALSE pending user requirements

Tracing of Classes to Faults Understanding both the potential network faults and reliability classes allows users to also understand which types of faults may be avoidable or mitigated through particular reliability mechanisms within each class. This is shown in below, which traces the aforementioned network faults with the reliability class(es) with the potential to mitigate those faults.

Layer Faults to Reliability Classes

Security Considerations The reliability mechanisms described in this document introduce security considerations beyond those associated with the Bundle Protocol itself, because reliability signaling and custodial state become part of the attack surface of the network. Custody acknowledgements represent release conditions for stored data. An attacker able to forge a custody acknowledgement can cause a legitimate custodian to prematurely release a bundle from storage, resulting in silent, unrecoverable data loss that the network believes was a successful transfer. Conversely, an attacker able to suppress or delay custody acknowledgements can prevent custodians from releasing stored data, contributing to storage exhaustion at those nodes. For these reasons, custody and other reliability signaling exchanged at the bundle layer needs integrity protection and source authentication, such as that provided by the Bundle Protocol Security specification . Custodial storage itself is a target for denial-of-service. Because Class 2 and Class 3 services obligate the network to store data on behalf of users, an attacker able to inject traffic requesting these service classes can consume custodian storage that would otherwise be available to legitimate traffic. Networks offering custodial services need to authenticate and authorize requests for those services, apply per-user or per-flow storage quotas, and consider the interaction between bundle lifetimes and storage commitments. Class 3 services additionally replicate user data across multiple custodians and, potentially, multiple paths. This replication increases the number of nodes holding copies of user data (increasing exposure of data at rest) and can act as a traffic amplification vector if service class requests are not authenticated. Confidentiality protection of bundle payloads , applied at or before the bundle source, mitigates the data-at-rest exposure; admission control mitigates the amplification concern. Finally, reliability status information (acknowledgements, custody signals, and delivery reports) can reveal information about network topology, custodian placement, and traffic patterns to an observer. Deployments in which such information is sensitive should consider confidentiality protection for reliability signaling in addition to user data.

IANA Considerations This document has no IANA actions.

Informative References This document describes an architecture for delay-tolerant and disruption-tolerant networks, and is an evolution of the architecture originally designed for the Interplanetary Internet, a communication system envisioned to provide Internet-like services across interplanetary distances in support of deep space exploration. This document describes an architecture that addresses a variety of problems with internetworks having operational and performance characteristics that make conventional (Internet-like) networking approaches either unworkable or impractical. We define a message- oriented overlay that exists above the transport (or other) layers of the networks it interconnects. The document presents a motivation for the architecture, an architectural overview, review of state management required for its operation, and a discussion of application design issues. This document represents the consensus of the IRTF DTN research group and has been widely reviewed by that group. This memo provides information for the Internet community. This document describes the end-to-end protocol, block formats, and abstract service description for the exchange of messages (bundles) in Delay Tolerant Networking (DTN). This document was produced within the IRTF's Delay Tolerant Networking Research Group (DTNRG) and represents the consensus of all of the active contributors to this group. See http://www.dtnrg.org for more information. This memo defines an Experimental Protocol for the Internet community. This document presents a specification for the Bundle Protocol, adapted from the experimental Bundle Protocol specification developed by the Delay-Tolerant Networking Research Group of the Internet Research Task Force and documented in RFC 5050. This document defines a security protocol providing data integrity and confidentiality services for the Bundle Protocol (BP). This document describes a TCP convergence layer (TCPCL) for Delay-Tolerant Networking (DTN). This version of the TCPCL protocol resolves implementation issues in the earlier TCPCL version 3 as defined in RFC 7242 and provides updates to the Bundle Protocol (BP) contents, encodings, and convergence-layer requirements in BP version 7 (BPv7). Specifically, TCPCLv4 uses BPv7 bundles encoded by the Concise Binary Object Representation (CBOR) as its service data unit being transported and provides a reliable transport of such bundles. This TCPCL version also includes security and extensibility mechanisms. The Delay-Tolerant Networking (DTN) architecture describes a type of challenged network in which communications may be significantly affected by long signal propagation delays, frequent link disruptions, or both. The unique characteristics of this environment require a unique approach to network management that supports asynchronous transport, autonomous local control, and a small footprint (in both resources and dependencies) so as to deploy on constrained devices. This document describes a DTN Management Architecture (DTNMA) suitable for managing devices in any challenged environment but, in particular, those communicating using the DTN Bundle Protocol (BP). Operating over BP requires an architecture that neither presumes synchronized transport behavior nor relies on query-response mechanisms. Implementations compliant with this DTNMA should expect to successfully operate in extremely challenging conditions, such as over unidirectional links and other places where BP is the preferred transport. IPNGROUP Spatiam Corporation SAIC University of Bologna This document describes Bundle-in-Bundle Encapsulation (BIBE), a Delay-Tolerant Networking (DTN) Bundle Protocol (BP) "convergence layer" protocol that tunnels BP "bundles" through encapsulating bundles. The services provided by the BIBE convergence-layer protocol adapter encapsulate an outbound BP "bundle" in a BIBE convergence-layer protocol data unit for transmission as the payload of a bundle. Security measures applied to the encapsulating bundle may augment those applied to the encapsulated bundle. The protocol includes a mechanism for recovery from loss of an encapsulating bundle, called Bundle Retransmission Methods (BRM). This mechanism is adapted from the custody transfer procedures described in the experimental Bundle Protocol (version 6) specification developed by the Delay-Tolerant Networking Research Group of the Internet Research Task Force and documented in RFC 5050.

Acknowledgments The authors thank the participants of the IETF DTN Working Group for discussions that informed this document.