Use Cases and Requirements for Multi-Domain and Hybrid Overlay/Underlay BGP-SPF (LSVR)

Introduction Modern Internet applications -- including cross-region real-time communication, collaborative productivity tools, cloud gaming, and large-scale Software as a Service (SaaS) -- are increasingly delivered from globally distributed edge and cloud Points of Presence (PoPs). A user attaches to a nearby ingress node, and traffic is carried across a provider-operated backbone to an egress node close to the peer or to the application instance. These applications have explicit end-to-end performance targets (latency, jitter, loss) that depend on the quality of every link segment along the path. The provider network controls the path between the ingress and egress PoPs; how an endpoint chooses its attachment PoP is an application-layer concern, discussed with the other assumptions in . The underlying network has two distinguishing characteristics. First, it is multi-domain: the user's Autonomous System (AS), the ingress/egress PoPs, and the transit between them may belong to different administrative domains, so inter-domain forwarding is unavoidable. Second, it is increasingly built from overlay logical links (tunnels established over third-party transit that the operator does not control), sometimes in combination with underlay physical links that the operator does own. The BGP Link State (BGP-LS) Shortest Path First (SPF) routing (BGP-SPF), developed in the LSVR Working Group, combines link-state computation with BGP transport, but its base scope is a single SPF domain within a data center . The name "Link-State Vector Routing" reflects a hybrid that combines link-state and path-vector mechanisms: the base BGP-SPF realizes the link-state half within a single domain, while the path-vector half -- inter-domain topology abstraction (information hiding) and policy at domain boundaries -- is precisely what the multi-domain extension motivated here must supply. Just as describes the usage and applicability of BGP-SPF within the data center (Clos/Fat-Tree) setting it was first designed for, this document describes its use cases across multiple administrative domains and over hybrid overlay/underlay topologies, and derives the requirements they imply. The requirements are informed by an experimental design, referred to here as Multi-Domain Routing (MIDR), that the authors are developing; this document specifies use cases and requirements only, and leaves protocol mechanisms to a separate specification.

Conventions and Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document is Informational and does not specify a protocol. Where these key words appear, they state requirements on a prospective solution that would satisfy the use cases described here; they do not describe the behavior of any protocol defined by this document.

PoP:: Point of Presence; an edge or cloud site at which users attach or applications are hosted.
Ingress / Egress node:: The PoP node at which user traffic enters / leaves the provider-operated backbone.
Underlay link:: A physical or otherwise directly observable link whose state can be measured locally by the attached router.
Overlay link:: A logical link (for example a tunnel over third-party transit) whose state cannot be observed directly and must be inferred by active measurement.
Hybrid topology:: A topology containing both underlay and overlay links.
Administrative domain:: A set of nodes under a single administrative and commercial authority, typically one or more ASes operated by one entity.
Overlay domain:: A group of PoPs whose overlay links are flooded and computed together as one SPF domain, typically a geographic region or a separately operated network, interconnected with other overlay domains at border PoPs.

Motivation To guarantee end-to-end application quality, path computation needs link-state, fine-grained topology and performance awareness. To remain scalable and deployable across domains, the control plane must build on BGP. Neither family of existing protocols satisfies both at once. BGP is not link-state aware: its decision process does not account for per-link performance, so it cannot guarantee end-to-end performance. It also converges slowly; on node or link failure it relies on route withdrawal and update propagation, which can take minutes, far exceeding the needs of real-time interactive applications. OSPF and IS-IS assume full trust within a domain: all nodes flood state and hold an identical topology database. Across administrative domains, which belong to different commercial entities, an operator is unwilling to fully expose its internal physical topology to others (for example to a competitor). These protocols provide neither inter-domain information hiding/abstraction nor inter-domain isolation. Moreover, where link state changes frequently, global flooding triggers frequent network-wide recomputation and does not scale. Overlay performance routing is today most often realized with a centralized controller (for example an SD-WAN or a proprietary global acceleration platform) that collects measurements and computes and installs paths. A centralized design is attractive for a single operator, but it fits the scenarios of this document poorly in three respects. First, it is a scaling and availability bottleneck: a controller responsible for hundreds of PoPs is a single point of failure and a central load concentrator. Second, recovery depends on a round trip to the controller: on an overlay-link failure, traffic cannot move to an alternate path until the controller has been informed, recomputed, and pushed new state, which is often too slow for a real-time session; a distributed link-state design lets each node recompute and reroute locally (REQ-5). Third, a controller presumes a single locus of control and full visibility, which does not hold once a path crosses administrative or operator boundaries (Scenarios B and C), where no party may see or control the whole topology. A distributed, link-state control plane addresses all three, which is why this document pursues one rather than extending a centralized controller. BGP-SPF is a suitable base for resolving this tension. It reuses BGP's transport and distribution machinery -- reliable transport, incremental per-change updates, and the BGP-LS encoding -- so it is incrementally deployable alongside existing BGP, while replacing the BGP path-vector decision process with a link-state Shortest Path First (SPF) computation that provides the fine-grained topology and performance awareness the scenarios require. As a distributed link-state computation, it also lets every node recompute locally on failure, without a round trip to a central controller -- a resilience property the use cases below rely on (REQ-5). As specified, however, BGP-SPF operates within a single SPF domain, in which link-state NLRIs are distributed to, and used identically by, all participating speakers; it provides no topology aggregation or hiding, and its decision process deliberately ignores the BGP path-vector policy attributes (ORIGIN, MULTI_EXIT_DISC, and LOCAL_PREF) . Inter-domain information hiding and policy are therefore not inherited from the base; they are the "path-vector" half of the link-state vector approach that the multi-domain extension must add, by scoping detailed link-state distribution within a domain and advertising only aggregated, abstracted reachability across domain boundaries (see REQ-6, REQ-7, and REQ-10). Because BGP-SPF already runs within the BGP ecosystem, that ecosystem's inter-domain machinery can be leveraged to provide them. Extending BGP-SPF in this way reuses most of the base design and is the lowest-cost approach; the scenarios below () make the extension concrete and derive its requirements.

Scope and Assumptions The use cases below concern routing within the provider-operated network: the computation and maintenance of the path between the ingress PoP at which a user attaches and the egress PoP closest to the peer or application instance. The following points delimit what is in scope.

Endpoint attachment is an application-layer function and is out of scope. Which ingress or egress PoP a given user attaches to is decided by application-layer proximity steering (for example DNS-based global server load balancing, anycast, or client-side probing), not by the routing control plane. In interactive applications the two endpoints of a session are typically not known at the same time: one user attaches first and is steered to a nearby PoP, and the peer's location becomes known only after it, too, has attached. Selecting the attachment PoPs is thus performed independently per endpoint and is not addressed here.
The routing system optimizes the ingress-to-egress segment. Given the two attachment PoPs, the requirements in this document concern computing and maintaining an optimized path between them across the provider network, which may be multi-domain and may be a hybrid of overlay and underlay. Where this document refers to "end-to-end" targets, those are the user-perceived objectives; the segment the routing system actually controls is from ingress PoP to egress PoP. The first and last access hops (user to ingress PoP, egress PoP to peer) are not the subject of these requirements.
Topology is maintained continuously, not discovered per session. Because the endpoint pair of a session is not known in advance, and because real-time applications cannot tolerate per-session topology discovery, the control plane maintains the link-state topology of the PoP network -- including measured overlay link state (REQ-1) -- continuously and independently of session arrival, so that an optimized ingress-to-egress path can be produced as soon as both attachment PoPs are known. This always-on, link-state model is a natural fit for a BGP-SPF-based control plane and contrasts with on-demand, per-flow path setup.

Scenarios The scenarios share a common setting. Users attach to nearby ingress PoPs; traffic crosses one or more administrative domains over a provider-operated backbone; and the carried applications -- cross-region video conferencing and collaborative office, cloud gaming, and large-scale SaaS -- all have explicit end-to-end latency, jitter, and loss targets. The number of PoPs is large, node health changes dynamically (for example on failure), and link state changes frequently. The scenarios are organized as a ladder of increasing complexity along two orthogonal axes. The complexity axis selects the scenario: a single overlay domain (), the interconnection of multiple overlay domains (), and the addition of an owned underlay layer (). The trust axis is developed inside the last two scenarios: the domains being interconnected may belong to the same operator (partitioned for scalability) or to different operators (separated by policy and limited trust). Each rung adds a distinct group of requirements to the ones before it; they are derived within each scenario and consolidated in .

Scenario A: A Single Overlay Domain In the simplest case the operator runs one overlay network that, from the overlay's own point of view, is a single domain: all PoPs participate in one link-state flooding scope, every node holds the full overlay topology, and each computes paths locally. What lies beneath each overlay link (the underlay that carries the tunnel) is out of view and is not the operator's concern here. shows such a domain.

Scenario A: a single overlay domain -- one flooding scope, every node holds the full overlay link state and computes paths locally overlay link (measured: delay / loss / bw) every node: full overlay LSDB + local SPF P1 ------------- P2 | \ / | | \ / | | \ / | | \/ | | /\ | | / \ | | / \ | | / \ | P3 ------------- P4 Structurally this is the case BGP-SPF was designed for: a single SPF domain in which link state is flooded to all speakers and used identically. The base mechanism therefore fits without any inter-domain machinery. However, even here the base is not sufficient as-is, because the links are overlay links. Distinguishing challenges:

Overlay link state is not directly observable. Unlike an underlay link, whose state a router can measure locally, an overlay link's bandwidth, delay, and loss must be obtained by active, continuous measurement and fed into the routing system as link-state performance attributes.
Overlay links are volatile. Their performance varies more sharply than underlay links, so paths must be re-selected as conditions change, which makes link-state-driven routing both more necessary and more demanding.
Fast fault detection and recovery. Overlay link and node failures are not directly observable and must be detected by liveness and measurement; once detected, traffic must move onto an alternate overlay path within the time budget of a real-time session. Because every node holds the full link-state topology of the domain, it can recompute and reroute locally, without a round trip to a central controller -- a resilience advantage of distributed link-state routing that is especially valuable when overlay links fail more often than physical ones.

Routing behavior required: nodes must discover candidate neighbors automatically and select among them based on measured path properties (for example avoiding tier-1 transit, or preferring non-overlapping paths), building on the LSVR Layer-3 discovery work ; they must compute paths over measured overlay link state using multiple performance metrics; and they must recover quickly from failure. Requirements arising from this scenario:

REQ-1 (Overlay link-state sensing):: The solution MUST support obtaining, representing, and using the state (bandwidth, delay, loss) of overlay links that cannot be observed directly, via continuous active measurement, maintained independently of session arrival (see ).
REQ-2 (Automatic, measurement-driven neighbor selection):: The solution MUST support automatic discovery of candidate neighbors at scale and selection among them based on measured path properties, rather than relying solely on manual configuration.
REQ-3 (BGP reuse and incremental deployability):: The solution MUST reuse BGP transport and the BGP-LS encoding , and MUST be incrementally deployable alongside existing BGP, so that a wholly new non-BGP inter-domain protocol is not required.
REQ-4 (Multi-metric path computation):: The solution MUST support path computation over multiple performance metrics (for example latency, jitter, and loss), rather than over a single uniform metric.
REQ-5 (Fast fault detection and local reroute):: The solution MUST support timely detection of overlay link and node failures, which are not directly observable, and MUST support local recomputation and reroute onto an alternate path without requiring a round trip to a centralized controller, so that recovery is fast enough for real-time interactive applications and substantially faster than BGP path-vector withdrawal/update.

Scenario B: Interconnecting Multiple Overlay Domains At global scale a single flat overlay domain does not hold up: a link-state scope spanning hundreds of PoPs would flood every measurement change everywhere and could not confine churn or failures. The overlay is therefore partitioned into multiple overlay domains -- for example one per geographic region, or one per separately operated network -- interconnected at border PoPs. shows two overlay domains and what information crosses the boundary between them: only abstracted, aggregated reachability, while detailed link state stays within each domain.

Scenario B: a backbone partitioned into interconnected overlay domains; only abstracted reachability crosses a domain boundary Overlay domain A Overlay domain B .-------------------------. .-------------------------. | BJ --- SH --- GZ | | DEN --- NYC | | \ | | / | | TY >=====|=====|=> LA | | (border) handoff (border) | | | | | | full link state kept | | full link state kept | | inside A (all tunnels | | inside B | | + measured metrics) | | | '-------------------------' '-------------------------' ^ | | user@BJ app@NYC v This scenario spans a trust axis. At one end the interconnected domains belong to the same operator and are partitioned primarily for scalability and fault isolation; the domains trust one another, but detailed state is still kept local so that a measurement change or failure in one region does not trigger network-wide recomputation. At the other end the domains belong to different operators, and the boundary must additionally enforce policy and hide internal topology because the operators do not fully trust one another and may be competitors. The same interconnection mechanism serves both ends: across a domain boundary each domain advertises only abstracted reachability with an aggregate metric -- for example A advertises "via TY reach BJ ~35 ms, SH ~30 ms" -- while its internal tunnels and their measured state stay hidden. Distinguishing challenges (in addition to those of Scenario A):

Information hiding and policy at boundaries. Detailed link state must be scoped within a domain, and only aggregated, abstracted reachability advertised across a boundary, under the advertising domain's policy.
Scale and churn containment. With many PoPs and frequent performance changes, naive global flooding would trigger constant network-wide recomputation; advertisement must be scoped so that churn stays within a domain.
Interconnecting independently measured domains. Each domain measures its own links by its own methodology and calibration, so their metrics are not automatically comparable; yet an ingress-to-egress path is stitched from segments contributed by several domains, so a common or reconciled metric basis is needed for the composed path to be meaningful.
Inter-domain loop prevention. Independent per-domain computation over asymmetric, aggregated information creates a risk of inter-domain forwarding loops that must be prevented.

Requirements arising from this scenario (in addition to REQ-1..REQ-5):

REQ-6 (Policy and information hiding):: The solution MUST provide inter-domain policy expressiveness and MUST support information hiding / topology abstraction, so that an operator need not expose its internal topology to other domains.
REQ-7 (Scalable, scoped flooding):: The solution MUST scope link-state advertisement so that frequent link-state churn is contained and does not trigger global recomputation.
REQ-8 (Cross-domain metric normalization):: When independently operated or independently measured overlay domains interconnect, the solution MUST provide a means to express link-state performance metrics on a common or reconcilable basis, so that a path composed of segments from different domains can be compared and computed meaningfully.
REQ-9 (Inter-domain loop prevention and consistency):: The solution MUST prevent inter-domain forwarding loops that can arise when domains compute paths independently over aggregated, asymmetric reachability information, and MUST keep advertised aggregates consistent enough for loop-free composition of ingress-to-egress paths.

REQ-8 and REQ-9 are among the harder requirements: normalizing independently measured metrics and guaranteeing loop-free composition over aggregated inter-domain state are open design problems. This document states them as requirements; the mechanisms that satisfy them are left to the protocol specification.

REQ-10 (Hierarchical, scalable computation):: The solution MUST support path computation that scales hierarchically (for example intra-domain and inter-domain), so that detailed state is handled locally while only aggregated information is exchanged between domains.

Scenario C: Hybrid Overlay/Underlay Networks This scenario adds an owned underlay. In addition to overlay links, the operator owns and can observe part of the underlay (for example a regional physical backbone), and an ingress-to-egress path may traverse both layers. The hybrid case arises because neither layer alone is sufficient: an underlay-only path may not meet the performance target where it depends on uncontrolled third-party transit, while a purely overlay path may be more costly or limited. The routing system must be able to compose a path across both layers. illustrates a path composed from owned underlay and measured overlay segments.

Scenario C: a cross-layer path composed from operator-owned underlay segments and a measured overlay segment underlay (operator-owned, observed): ==== overlay (tunnel over 3rd-party transit, measured): ~~~~ BJ ==== SH ==== TY ~~~~~~~~~~~~~~~ LA ==== NYC '----- owned underlay ----' overlay '-underlay-' (3rd-party transit) Regional segments (BJ..TY and LA..NYC) run on the operator's own underlay, whose state is observed locally; the long-haul TY..LA segment exists only as a measured overlay. When no single-layer path meets the target, routing composes the cross-layer path BJ=SH=TY ~ LA = NYC over one unified link-state view (REQ-11, REQ-12). Like Scenario B, this scenario spans a trust axis. At one end a single operator owns both the overlay and the underlay, so the two layers can be combined freely. At the other end the layers belong to different operators -- an overlay operator interconnecting with a traditional underlay operator -- so composing a cross-layer path additionally crosses an inter-operator boundary and must apply the information hiding and policy of Scenario B (REQ-6) on top of the cross-layer requirements below. Distinguishing challenges (in addition to those of Scenarios A and B):

Unified cross-layer view. Underlay link state (directly observable) and overlay link state (measured) must be represented together so that computation can reason over both.
Joint cross-layer path selection. When a single-layer path does not meet the target, the solution must be able to select a combined overlay/underlay path, rather than being confined to one layer.
Realization in the data plane. A computed cross-layer path must be installed into the forwarding plane (for example using SRv6 or tunnels), including any handling needed for paths expressed at a coarser granularity than per-hop.

Requirements arising from this scenario (in addition to REQ-1..REQ-10):

REQ-11 (Unified overlay/underlay link state):: The solution MUST represent and use underlay and overlay link state together within a single link-state view.
REQ-12 (Joint cross-layer path computation):: The solution MUST be able to compute a combined overlay/underlay path when no single-layer path meets the performance target.
REQ-13 (Data-plane installation):: The solution MUST define how computed paths are installed into a data plane (for example SRv6 or tunnels), including any special handling required for paths expressed at a coarser-than-per-hop granularity.
REQ-14 (Traffic engineering):: The solution SHOULD support traffic engineering to influence path selection based on collected network state.

Open issue: expressing a coarse-granularity path in the data plane may not fit the standard SRv6 processing model and needs care to avoid forwarding loops. This data-plane concern is distinct from the control-plane loop prevention of REQ-9, and is noted for the future protocol specification.

Consolidated Requirements For convenience, the requirements derived above are listed together here; the authoritative statements are in , , and .

Req	Summary	From
REQ-1	Overlay link-state sensing (active measurement)	A
REQ-2	Automatic, measurement-driven neighbor selection	A
REQ-3	BGP reuse and incremental deployability	A
REQ-4	Multi-metric path computation	A
REQ-5	Fast fault detection and local reroute	A
REQ-6	Inter-domain policy and information hiding	B
REQ-7	Scalable, scoped flooding	B
REQ-8	Cross-domain metric normalization	B
REQ-9	Inter-domain loop prevention and consistency	B
REQ-10	Hierarchical, scalable computation	B
REQ-11	Unified overlay/underlay link state	C
REQ-12	Joint cross-layer path computation	C
REQ-13	Data-plane installation of computed paths	C
REQ-14	Traffic engineering	C

Gap Analysis This section maps the requirements to the current LSVR base, grouped by the three scenarios. Single overlay domain (Scenario A). Structurally BGP-SPF already fits a single SPF domain, but it relies on directly observable link state and a single uniform metric, and reacts to failures through BGP-style withdrawal and update. REQ-1 and REQ-4 require sensing overlay link state and computing over multiple metrics; REQ-2 builds on LSVR Layer-3 discovery but adds measurement-driven neighbor selection; REQ-5 requires fast detection of overlay failures and local reroute within a real-time budget; REQ-3 requires that all of this reuse BGP and be incrementally deployable. Interconnecting overlay domains (Scenario B). The base is confined to one SPF domain with no aggregation or hiding and no policy support. REQ-6, REQ-7, and REQ-10 require operation across multiple domains with scoped advertisement, information hiding, and hierarchical computation; REQ-8 and REQ-9 add cross-domain metric normalization and inter-domain loop prevention when independently measured domains interconnect. The BGP-LS encoding defines Node, Link, and Prefix NLRI; REQ-1, REQ-6, REQ-7, and REQ-11 imply additional attributes (for example performance and aggregation/scoping information) and control over advertisement scope. Hybrid overlay/underlay (Scenario C). The single-domain underlay base does not compose paths across layers or address their data-plane realization. REQ-11 and REQ-12 require a unified cross-layer view and joint computation; REQ-13 introduces data-plane installation considerations; REQ-14 adds traffic engineering. Where the two layers belong to different operators, the Scenario B mechanisms (notably REQ-6) apply on top.

Security Considerations Operating across administrative domains raises trust concerns beyond the single-domain base. The integrity and authenticity of advertised link state and performance attributes MUST be protected, since false advertisements could be used to attract or divert traffic. Because operators in different domains may be competitors, the solution must avoid exposing internal topology beyond what an operator chooses to advertise (see REQ-6). Detailed mechanisms are left to the protocol specification.

IANA Considerations This document has no IANA actions.