



Routing Working Group                                           F. Baker
Internet-Draft
Intended status: Informational                                 C. Bowers
Expires: February 1, 2020                               Juniper Networks
                                                              J. Linkova
                                                                  Google
                                                           July 31, 2019


 Enterprise Multihoming using Provider-Assigned IPv6 Addresses without
         Network Prefix Translation: Requirements and Solutions
             draft-ietf-rtgwg-enterprise-pa-multihoming-12

Abstract

   Connecting an enterprise site to multiple ISPs over IPv6 using
   provider-assigned addresses is difficult without the use of some form
   of Network Address Translation (NAT).  Much has been written on this
   topic over the last 10 to 15 years, but it still remains a problem
   without a clearly defined or widely implemented solution.  Any
   multihoming solution without NAT requires hosts at the site to have
   addresses from each ISP and to select the egress ISP by selecting a
   source address for outgoing packets.  It also requires routers at the
   site to take into account those source addresses when forwarding
   packets out towards the ISPs.

   This document examines currently available mechanisms for providing a
   solution to this problem for a broad range of enterprise topologies.
   It covers the behavior of routers to forward traffic taking into
   account source address, and it covers the behavior of hosts to select
   appropriate default source addresses.  It also covers any possible
   role that routers might play in providing information to hosts to
   help them select appropriate source addresses.  In the process of
   exploring potential solutions, this document also makes explicit
   requirements for how the solution would be expected to behave from
   the perspective of an enterprise site network administrator.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.





Baker, et al.           Expires February 1, 2020                [Page 1]

Internet-Draft          Enterprise PA Multihoming              July 2019


   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on February 1, 2020.

Copyright Notice

   Copyright (c) 2019 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   4
   2.  Requirements Language . . . . . . . . . . . . . . . . . . . .   6
   3.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   6
   4.  Enterprise Multihoming Use Cases  . . . . . . . . . . . . . .   8
     4.1.  Simple ISP Connectivity with Connected SERs . . . . . . .   8
     4.2.  Simple ISP Connectivity Where SERs Are Not Directly
           Connected . . . . . . . . . . . . . . . . . . . . . . . .  10
     4.3.  Enterprise Network Operator Expectations  . . . . . . . .  12
     4.4.  More complex ISP connectivity . . . . . . . . . . . . . .  14
     4.5.  ISPs and Provider-Assigned Prefixes . . . . . . . . . . .  16
     4.6.  Simplified Topologies . . . . . . . . . . . . . . . . . .  17
   5.  Generating  Source-Prefix-Scoped Forwarding Tables  . . . . .  17
   6.  Mechanisms For Hosts To Choose Good Default Source Addresses
       In A Multihomed Site  . . . . . . . . . . . . . . . . . . . .  24
     6.1.  Default Source Address Selection Algorithm on Hosts . . .  26
     6.2.  Selecting Default Source Address When Both Uplinks Are
           Working . . . . . . . . . . . . . . . . . . . . . . . . .  29
       6.2.1.  Distributing Default Address Selection Policy Table
               with DHCPv6 . . . . . . . . . . . . . . . . . . . . .  29
       6.2.2.  Controlling Default Source Address Selection With
               Router Advertisements . . . . . . . . . . . . . . . .  30
       6.2.3.  Controlling Default Source Address Selection With
               ICMPv6  . . . . . . . . . . . . . . . . . . . . . . .  32
       6.2.4.  Summary of Methods For Controlling Default Source



Baker, et al.           Expires February 1, 2020                [Page 2]

Internet-Draft          Enterprise PA Multihoming              July 2019


               Address Selection To Implement Routing Policy . . . .  33
     6.3.  Selecting Default Source Address When One Uplink Has
           Failed  . . . . . . . . . . . . . . . . . . . . . . . . .  34
       6.3.1.  Controlling Default Source Address Selection With
               DHCPv6  . . . . . . . . . . . . . . . . . . . . . . .  35
       6.3.2.  Controlling Default Source Address Selection With
               Router Advertisements . . . . . . . . . . . . . . . .  36
       6.3.3.  Controlling Default Source Address Selection With
               ICMPv6  . . . . . . . . . . . . . . . . . . . . . . .  37
       6.3.4.  Summary Of Methods For Controlling Default Source
               Address Selection On The Failure Of An Uplink . . . .  38
     6.4.  Selecting Default Source Address Upon Failed Uplink
           Recovery  . . . . . . . . . . . . . . . . . . . . . . . .  38
       6.4.1.  Controlling Default Source Address Selection With
               DHCPv6  . . . . . . . . . . . . . . . . . . . . . . .  38
       6.4.2.  Controlling Default Source Address Selection With
               Router Advertisements . . . . . . . . . . . . . . . .  39
       6.4.3.  Controlling Default Source Address Selection With
               ICMP  . . . . . . . . . . . . . . . . . . . . . . . .  39
       6.4.4.  Summary Of Methods For Controlling Default Source
               Address Selection Upon Failed Uplink Recovery . . . .  40
     6.5.  Selecting Default Source Address When All Uplinks Failed   40
       6.5.1.  Controlling Default Source Address Selection With
               DHCPv6  . . . . . . . . . . . . . . . . . . . . . . .  40
       6.5.2.  Controlling Default Source Address Selection With
               Router Advertisements . . . . . . . . . . . . . . . .  40
       6.5.3.  Controlling Default Source Address Selection With
               ICMPv6  . . . . . . . . . . . . . . . . . . . . . . .  41
       6.5.4.  Summary Of Methods For Controlling Default Source
               Address Selection When All Uplinks Failed . . . . . .  41
     6.6.  Summary Of Methods For Controlling Default Source Address
           Selection . . . . . . . . . . . . . . . . . . . . . . . .  41
     6.7.  Solution Limitations  . . . . . . . . . . . . . . . . . .  43
       6.7.1.  Connections Preservation  . . . . . . . . . . . . . .  43
     6.8.  Other Configuration Parameters  . . . . . . . . . . . . .  44
       6.8.1.  DNS Configuration . . . . . . . . . . . . . . . . . .  44
   7.  Deployment Considerations . . . . . . . . . . . . . . . . . .  45
     7.1.  Deploying SADR Domain . . . . . . . . . . . . . . . . . .  46
     7.2.  Hosts-Related Considerations  . . . . . . . . . . . . . .  46
   8.  Other Solutions . . . . . . . . . . . . . . . . . . . . . . .  47
     8.1.  Shim6 . . . . . . . . . . . . . . . . . . . . . . . . . .  47
     8.2.  IPv6-to-IPv6 Network Prefix Translation . . . . . . . . .  47
     8.3.  Multipath Transport . . . . . . . . . . . . . . . . . . .  47
   9.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  48
   10. Security Considerations . . . . . . . . . . . . . . . . . . .  48
   11. Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  49
   12. References  . . . . . . . . . . . . . . . . . . . . . . . . .  49
     12.1.  Normative References . . . . . . . . . . . . . . . . . .  49



Baker, et al.           Expires February 1, 2020                [Page 3]

Internet-Draft          Enterprise PA Multihoming              July 2019


     12.2.  Informative References . . . . . . . . . . . . . . . . .  51
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  52

1.  Introduction

   Site multihoming, the connection of a subscriber network to multiple
   upstream networks using redundant uplinks, is a common enterprise
   architecture for improving the reliability of its Internet
   connectivity.  If the site uses provider-independent (PI) addresses,
   all traffic originating from the enterprise can use source addresses
   from the PI address space.  Site multihoming with PI addresses is
   commonly used with both IPv4 and IPv6, and does not present any new
   technical challenges.

   It may be desirable for an enterprise site to connect to multiple
   ISPs using provider-assigned (PA) addresses, instead of PI addresses.
   Multihoming with provider-assigned addresses is typically less
   expensive for the enterprise relative to using provider-independent
   addresses as it does not require obtaining and maintaining PI address
   space as well as running BGP between the enterprise and the ISPs (for
   small/meduim networks running BGP might be not just undesirable but
   impossible, especially if residential-type ISP connections are used).
   PA multihoming is also a practice that should be facilitated and
   encouraged because it does not add to the size of the Internet
   routing table, whereas PI multihoming does.  Note that PA is also
   used to mean "provider-aggregatable".  In this document we assume
   that provider-assigned addresses are always provider-aggregatable.

   With PA multihoming, for each ISP connection, the site is assigned a
   prefix from within an address block allocated to that ISP by its
   National or Regional Internet Registry.  In the simple case of two
   ISPs (ISP-A and ISP-B), the site will have two different prefixes
   assigned to it (prefix-A and prefix-B).  This arrangement is
   problematic.  First, packets with the "wrong" source address may be
   dropped by one of the ISPs.  In order to limit denial of service
   attacks using spoofed source addresses, BCP38 [RFC2827] recommends
   that ISPs filter traffic from customer sites to only allow traffic
   with a source address that has been assigned by that ISP.  So a
   packet sent from a multihomed site on the uplink to ISP-B with a
   source address in prefix-A may be dropped by ISP-B.

   However, even if ISP-B does not implement BCP38 or ISP-B adds
   prefix-A to its list of allowed source addresses on the uplink from
   the multihomed site, two-way communication may still fail.  If the
   packet with source address in prefix-A was sent to ISP-B because the
   uplink to ISP-A failed, then if ISP-B does not drop the packet and
   the packet reaches its destination somewhere on the Internet, the
   return packet will be sent back with a destination address in prefix-



Baker, et al.           Expires February 1, 2020                [Page 4]

Internet-Draft          Enterprise PA Multihoming              July 2019


   A.  The return packet will be routed over the Internet to ISP-A, but
   it will not be delivered to the multihomed site because the site
   uplink with ISP-A has failed.  Two-way communication would require
   some arrangement for ISP-B to advertise prefix-A when the uplink to
   ISP-A fails.

   Note that the same may be true with a provider that does not
   implement BCP 38, if his upstream provider does, or has no
   corresponding route to deliver the ingress traffic to the multihomed
   site.  The issue is not that the immediate provider implements
   ingress filtering; it is that someone upstream does (so egress
   traffic is blocked), or lacks a route (causing blackholing of the
   ingress traffic).

   Another issue with asymmetric traffic flow (when the egress traffic
   leaves the site via one ISP but the return traffic enters the site
   via another uplink) is related to stateful firewalls/middleboxes.
   Keeping state in that case might be problematic, even impossible.

   With IPv4, this problem is commonly solved by using [RFC1918] private
   address space within the multi-homed site and Network Address
   Translation (NAT) or Network Address/Port Translation (NAPT) on the
   uplinks to the ISPs.  However, one of the goals of IPv6 is to
   eliminate the need for and the use of NAT or NAPT.  Therefore,
   requiring the use of NAT or NAPT for an enterprise site to multihome
   with provider-assigned addresses is not an attractive solution.

   [RFC6296] describes a translation solution specifically tailored to
   meet the requirements of multi-homing with provider-assigned IPv6
   addresses.  With the IPv6-to-IPv6 Network Prefix Translation (NPTv6)
   solution, within the site an enterprise can use Unique Local
   Addresses [RFC4193] or the prefix assigned by one of the ISPs.  As
   traffic leaves the site on an uplink to an ISP, the source address
   gets translated to an address within the prefix assigned by the ISP
   on that uplink in a predictable and reversible manner.  [RFC6296] is
   currently classified as Experimental, and it has been implemented by
   several vendors.  See Section 8.2, for more discussion of NPTv6.

   This document defines routing requirements for enterprise multihoming
   This document focuses on the following general class of solutions.

   Each host at the enterprise has multiple addresses, at least one from
   each ISP-assigned prefix.  Each host, as discussed in Section 6.1 and
   [RFC6724], is responsible for choosing the source address applied to
   each packet it sends.  A host is expected to be able respond
   dynamically to the failure of an uplink to a given ISP by no longer
   sending packets with the source address corresponding to that ISP.
   Potential mechanisms for the communication of changes in the network



Baker, et al.           Expires February 1, 2020                [Page 5]

Internet-Draft          Enterprise PA Multihoming              July 2019


   to the host are Neighbor Discovery Router Advertisements ([RFC4861]),
   DHCPv6 ([RFC8415]), and ICMPv6 ([RFC4443]).

   The routers in the enterprise network are responsible for ensuring
   that packets are delivered to the "correct" ISP uplink based on
   source address.  This requires that at least some routers in the site
   network are able to take into account the source address of a packet
   when deciding how to route it.  That is, some routers must be capable
   of some form of Source Address Dependent Routing (SADR), if only as
   described in the section 4.3 of [RFC3704].  At a minimum, the routers
   connected to the ISP uplinks (the site exit routers or SERs) must be
   capable of Source Address Dependent Routing.  Expanding the connected
   domain of routers capable of SADR from the site exit routers deeper
   into the site network will generally result in more efficient routing
   of traffic with external destinations.

   This document is organized as follows.  Section 4 looks in more
   detail at the enterprise networking environments in which this
   solution is expected to operate.  The discussion of Section 4 uses
   the concepts of source-prefix-scoped routing advertisements and
   forwarding tables and provides a description of how source-prefix-
   scoped routing advertisements are used to generate source-prefix-
   scoped forwarding tables.  Instead, this detailed description is
   provided in Section 5.  Section 6 discusses existing and proposed
   mechanisms for hosts to select the default source address to be used
   by applications.  It also discusses the requirements for routing that
   are needed to support these enterprise network scenarios and the
   mechanisms by which hosts are expected to update default source
   addresses based on network state.  Section 7 discusses deployment
   considerations, while Section 8 discusses other solutions.

2.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

3.  Terminology

   PA (provider-assigned or provider-aggregatable) address space: a
   block of IP addresses assigned by an Regional Internet Registry (RIR)
   to a Local Internet Registry (LIR), used to create allocations to end
   sites.  Can be aggregated and present in the routing table as one
   route.





Baker, et al.           Expires February 1, 2020                [Page 6]

Internet-Draft          Enterprise PA Multihoming              July 2019


   PI (provider-independent) address space: a block of IP addresses
   assigned by an Regional Internet Registry (RIR) directly to end site/
   end customer.

   ISP: Internet Service Provider.

   LIR (Local Internet Registry): an organisation (usually an ISP or an
   enterprise/academic) which receives IP addresses allocation from its
   Regional Internet Regsitry, then assign parts of that allocation to
   its customers.

   RIR (Regional Internet Registry): an organization which manages the
   Internet number resources (such as IP addresses and AS numbers)
   within a geographical region of the world.

   SADR (Source Address Dependent Routing): Routing which takes into
   account the source address of a packet in addition to the packet
   destination address.

   SADR domain: a routing domain where some (or all) routers exchange
   source-dependent routing information.

   Source-Prefix-Scoped Routing/Forwarding Table: a routing (or
   forwarding) table which contains routing (or forwarding) information
   which is applicable to packets with source addresses from the
   specific prefix only.

   Unscoped Routing/Forwarding Table: a routing (or forwarding) table
   which can be used to route/forward packets with any source addresses.

   SER (Site Edge Router): a router which connects the site to an ISP
   (terminates an ISP uplink)..

   LLA (Link-Local Address): IPv6 Unicast Address from fe80::/10 prefix
   ([RFC4291]).

   ULA (Unique Local IPv6 Unicast Address): IPv6 unicast addresses from
   FC00::/7 prefix.  They are globally unique and intended for local
   communications ([RFC4193]).

   GUA (Global Unicast Address): globally routable IPv6 addresses of the
   global scope ([RFC4291]).

   SLAAC (IPv6 Stateless Address Autoconfiguration): a stateless process
   of configuring network stack on IPv6 hosts ([RFC4862]).






Baker, et al.           Expires February 1, 2020                [Page 7]

Internet-Draft          Enterprise PA Multihoming              July 2019


   RA (Router Advertisement): a message sent by an IPv6 router to
   advertise its presence to hosts together with various network-related
   parameters required for hosts to perform SLAAC ([RFC4861]).

   PIO (Prefix Information Option): a part of RA message containing
   information about IPv6 prefixes which could be used by hosts to
   generate global IPv6 addresses ([RFC4862]).

   RIO (Route Information Option): a part of RA message containing
   information about more specific IPv6 prefixes reachable via the
   advertising router ([RFC4191]).

4.  Enterprise Multihoming Use Cases

4.1.  Simple ISP Connectivity with Connected SERs

   We start by looking at a scenario in which a site has connections to
   two ISPs, as shown in Figure 1.  The site is assigned the prefix
   2001:db8:0:a000::/52 by ISP-A and prefix 2001:db8:0:b000::/52 by ISP-
   B.  We consider three hosts in the site.  H31 and H32 are on a LAN
   that has been assigned subnets 2001:db8:0:a010::/64 and
   2001:db8:0:b010::/64.  H31 has been assigned the addresses
   2001:db8:0:a010::31 and 2001:db8:0:b010::31.  H32 has been assigned
   2001:db8:0:a010::32 and 2001:db8:0:b010::32.  H41 is on a different
   subnet that has been assigned 2001:db8:0:a020::/64 and
   2001:db8:0:b020::/64.

























Baker, et al.           Expires February 1, 2020                [Page 8]

Internet-Draft          Enterprise PA Multihoming              July 2019


                                         2001:db8:0:1234::101   H101
                                                                  |
                                                                  |
 2001:db8:0:a010::31                                          --------
 2001:db8:0:b010::31                            ,-----.      /        \
                    +--+   +--+       +----+  ,'       `.   :          :
                +---|R1|---|R4|---+---|SERa|-+   ISP-A   +--+--        :
           H31--+   +--+   +--+   |   +----+  `.       ,'   :          :
                |                 |             `-----'     : Internet :
                |                 |                         :          :
                |                 |                         :          :
                |                 |                         :          :
                |                 |             ,-----.     :          :
           H32--+   +--+          |   +----+  ,'       `.   :          :
                +---|R2|----------+---|SERb|-+   ISP-B   +--+--        :
                    +--+          |   +----+  `.       ,'   :          :
                                  |             `-----'     :          :
                                  |                         :          :
                    +--+  +--+  +--+                         \        /
           H41------|R3|--|R5|--|R6|                          --------
                    +--+  +--+  +--+

 2001:db8:0:a020::41
 2001:db8:0:b020::41


           Figure 1: Simple ISP Connectivity With Connected SERs

   We refer to a router that connects the site to an ISP as a site edge
   router (SER).  Several other routers provide connectivity among the
   internal hosts (H31, H32, and H41), as well as connecting the
   internal hosts to the Internet through SERa and SERb.  In this
   example SERa and SERb share a direct connection to each other.  In
   Section 4.2, we consider a scenario where this is not the case.

   For the moment, we assume that the hosts are able to make good
   choices about which source addresses through some mechanism that
   doesn't involve the routers in the site network.  Here, we focus on
   primary task of the routed site network, which is to get packets
   efficiently to their destinations, while sending a packet to the ISP
   that assigned the prefix that matches the source address of the
   packet.  In Section 6, we examine what role the routed network may
   play in helping hosts make good choices about source addresses for
   packets.

   With this solution, routers will need some form of Source Address
   Dependent Routing, which will be new functionality.  It would be
   useful if an enterprise site does not need to upgrade all routers to



Baker, et al.           Expires February 1, 2020                [Page 9]

Internet-Draft          Enterprise PA Multihoming              July 2019


   support the new SADR functionality in order to support PA multi-
   homing.  We consider if this is possible and what are the tradeoffs
   of not having all routers in the site support SADR functionality.

   In the topology in Figure 1, it is possible to support PA multihoming
   with only SERa and SERb being capable of SADR.  The other routers can
   continue to forward based only on destination address, and exchange
   routes that only consider destination address.  In this scenario,
   SERa and SERb communicate source-scoped routing information across
   their shared connection.  When SERa receives a packet with a source
   address matching prefix 2001:db8:0:b000::/52 , it forwards the packet
   to SERb, which forwards it on the uplink to ISP-B.  The analogous
   behaviour holds for traffic that SERb receives with a source address
   matching prefix 2001:db8:0:a000::/52.

   In Figure 1, when only SERa and SERb are capable of source address
   dependent routing, PA multi-homing will work.  However, the paths
   over which the packets are sent will generally not be the shortest
   paths.  The forwarding paths will generally be more efficient as more
   routers are capable of SADR.  For example, if R4, R2, and R6 are
   upgraded to support SADR, then can exchange source-scoped routes with
   SERa and SERb.  They will then know to send traffic with a source
   address matching prefix 2001:db8:0:b000::/52 directly to SERb,
   without sending it to SERa first.

4.2.  Simple ISP Connectivity Where SERs Are Not Directly Connected

   In Figure 2, we modify the topology slightly by inserting R7, so that
   SERa and SERb are no longer directly connected.  With this topology,
   it is not enough to just enable SADR routing on SERa and SERb to
   support PA multi-homing.  There are two solutions to enable PA
   multihoming in this topology.



















Baker, et al.           Expires February 1, 2020               [Page 10]

Internet-Draft          Enterprise PA Multihoming              July 2019


                                         2001:db8:0:1234::101    H101
                                                                  |
                                                                  |
 2001:db8:0:a010::31                                          --------
 2001:db8:0:b010::31                            ,-----.      /        \
                    +--+   +--+       +----+  ,'       `.   :          :
                +---|R1|---|R4|---+---|SERa|-+   ISP-A   +--+--        :
           H31--+   +--+   +--+   |   +----+  `.       ,'   :          :
                |                 |             `-----'     : Internet :
                |               +--+                        :          :
                |               |R7|                        :          :
                |               +--+                        :          :
                |                 |             ,-----.     :          :
           H32--+   +--+          |   +----+  ,'       `.   :          :
                +---|R2|----------+---|SERb|-+   ISP-B   +--+--        :
                    +--+          |   +----+  `.       ,'   :          :
                                  |             `-----'     :          :
                                  |                         :          :
                    +--+  +--+  +--+                         \        /
           H41------|R3|--|R5|--|R6|                          --------
                    +--+  +--+  +--+                              |
                                                                  |
 2001:db8:0:a020::41                     2001:db8:0:5678::501    H501
 2001:db8:0:b020::41


       Figure 2: Simple ISP Connectivity Where SERs Are Not Directly
                                 Connected

   One option is to effectively modify the topology by creating a
   logical tunnel between SERa and SERb, using GRE ([RFC7676]) for
   example.  Although SERa and SERb are not directly connected
   physically in this topology, they can be directly connected logically
   by a tunnel.

   The other option is to enable SADR functionality on R7.  In this way,
   R7 will exchange source-scoped routes with SERa and SERb, making the
   three routers act as a single SADR domain.  This illustrates the
   basic principle that the minimum requirement for the routed site
   network to support PA multi-homing is having all of the site exit
   routers be part of a connected SADR domain.  Extending the connected
   SADR domain beyond that point can produce more efficient forwarding
   paths.








Baker, et al.           Expires February 1, 2020               [Page 11]

Internet-Draft          Enterprise PA Multihoming              July 2019


4.3.  Enterprise Network Operator Expectations

   Before considering a more complex scenario, let's look in more detail
   at the reasonably simple multihoming scenario in Figure 2 to
   understand what can reasonably be expected from this solution.  As a
   general guiding principle, we assume an enterprise network operator
   will expect a multihomed network to behave as close as to a single-
   homed network as possible.  So a solution that meets those
   expectations where possible is a good thing.

   For traffic between internal hosts and traffic from outside the site
   to internal hosts, an enterprise network operator would expect there
   be no visible change in the path taken by this traffic, since this
   traffic does not need to be routed in a way that depends on source
   address.  It is also reasonable to expect that internal hosts should
   be able to communicate with each other using either of their source
   addresses without restriction.  For example, H31 should be able to
   communicate with H41 using a packet with S=2001:db8:0:a010::31,
   D=2001:db8:0:b020::41, regardless of the state of uplink to ISP-B.

   These goals can be accomplished by having all of the routers in the
   network continue to originate normal unscoped destination routes for
   their connected networks.  If we can arrange so that these unscoped
   destination routes get used for forwarding this traffic, then we will
   have accomplished the goal of keeping forwarding of traffic destined
   for internal hosts, unaffected by the multihoming solution.

   For traffic destined for external hosts, it is reasonable to expect
   that traffic with a source address from the prefix assigned by ISP-A
   to follow the path to that the traffic would follow if there is no
   connection to ISP-B.  This can be accomplished by having SERa
   originate a source-scoped route of the form (S=2001:db8:0:a000::/52,
   D=::/0) .  If all of the routers in the site support SADR, then the
   path of traffic exiting via ISP-A can match that expectation.  If
   some routers don't support SADR, then it is reasonable to expect that
   the path for traffic exiting via ISP-A may be different within the
   site.  This is a tradeoff that the enterprise network operator may
   decide to make.

   It is important to understand how this multihoming solution behaves
   when an uplink to one of the ISPs fails.  To simplify this
   discussion, we assume that all routers in the site support SADR.  We
   first start by looking at how the network operates when the uplinks
   to both ISP-A and ISP-B are functioning properly.  SERa originates a
   source-scoped route of the form (S=2001:db8:0:a000::/52, D=::/0), and
   SERb is originates a source-scoped route of the form
   (S=2001:db8:0:b000::/52, D=::/0).  These routes are distributed
   through the routers in the site, and they establish within the



Baker, et al.           Expires February 1, 2020               [Page 12]

Internet-Draft          Enterprise PA Multihoming              July 2019


   routers two set of forwarding paths for traffic leaving the site.
   One set of forwarding paths is for packets with source address in
   2001:db8:0:a000::/52.  The other set of forwarding paths is for
   packets with source address in 2001:db8:0:b000::/52.  The normal
   destination routes which are not scoped to these two source prefixes
   play no role in the forwarding.  Whether a packet exits the site via
   SERa or via SERb is completely determined by the source address
   applied to the packet by the host.  So for example, when host H31
   sends a packet to host H101 with (S=2001:db8:0:a010::31,
   D=2001:db8:0:1234::101), the packet will only be sent out the link
   from SERa to ISP-A.

   Now consider what happens when the uplink from SERa to ISP-A fails.
   The only way for the packets from H31 to reach H101 is for H31 to
   start using the source address for ISP-B.  H31 needs to send the
   following packet: (S=2001:db8:0:b010::31, D=2001:db8:0:1234::101).

   This behavior is very different from the behavior that occurs with
   site multihoming using PI addresses or with PA addresses using NAT.
   In these other multi-homing solutions, hosts do not need to react to
   network failures several hops away in order to regain Internet
   access.  Instead, a host can be largely unaware of the failure of an
   uplink to an ISP.  When multihoming with PA addresses and NAT,
   existing sessions generally need to be re-established after a failure
   since the external host will receive packets from the internal host
   with a new source address.  However, new sessions can be established
   without any action on the part of the hosts.  Multihoming with PA
   addresses and NAT has created the expectation of a fairly quick and
   simple recovery from network failures.  Alternatives should to be
   evaluated in terms of the speed and complexity of the recovery
   mechanism.

   Another example where the behavior of this multihoming solution
   differs significantly from that of multihoming with PI address or
   with PA addresses using NAT is in the ability of the enterprise
   network operator to route traffic over different ISPs based on
   destination address.  We still consider the fairly simple network of
   Figure 2 and assume that uplinks to both ISPs are functioning.
   Assume that the site is multihomed using PA addresses and NAT, and
   that SERa and SERb each originate a normal destination route for
   D=::/0, with the route origination dependent on the state of the
   uplink to the respective ISP.

   Now suppose it is observed that an important application running
   between internal hosts and external host H101 experience much better
   performance when the traffic passes through ISP-A (perhaps because
   ISP-A provides lower latency to H101.)  When multihoming this site
   with PI addresses or with PA addresses and NAT, the enterprise



Baker, et al.           Expires February 1, 2020               [Page 13]

Internet-Draft          Enterprise PA Multihoming              July 2019


   network operator can configure SERa to originate into the site
   network a normal destination route for D=2001:db8:0:1234::/64 (the
   destination prefix to reach H101) that depends on the state of the
   uplink to ISP-A.  When the link to ISP-A is functioning, the
   destination route D=2001:db8:0:1234::/64 will be originated by SERa,
   so traffic from all hosts will use ISP-A to reach H101 based on the
   longest destination prefix match in the route lookup.

   Implementing the same routing policy is more difficult with the PA
   multihoming solution described in this document since it doesn't use
   NAT.  By design, the only way to control where a packet exits this
   network is by setting the source address of the packet.  Since the
   network cannot modify the source address without NAT, the host must
   set it.  To implement this routing policy, each host needs to use the
   source address from the prefix assigned by ISP-A to send traffic
   destined for H101.  Mechanisms have been proposed to allow hosts to
   choose the source address for packets in a fine grained manner.  We
   will discuss these proposals in Section 6.  However, interacting with
   host operating systems in some manner to ensure a particular source
   address is chosen for a particular destination prefix is not what an
   enterprise network administrator would expect to have to do to
   implement this routing policy.

4.4.  More complex ISP connectivity

   The previous sections considered two variations of a simple
   multihoming scenario where the site is connected to two ISPs offering
   only Internet connectivity.  It is likely that many actual enterprise
   multihoming scenarios will be similar to this simple example.
   However, there are more complex multihoming scenarios that we would
   like this solution to address as well.

   It is fairly common for an ISP to offer a service in addition to
   Internet access over the same uplink.  Two variations of this are
   reflected in Figure 3.  In addition to Internet access, ISP-A offers
   a service which requires the site to access host H51 at
   2001:db8:0:5555::51.  The site has a single physical and logical
   connection with ISP-A, and ISP-A only allows access to H51 over that
   connection.  So when H32 needs to access the service at H51 it needs
   to send packets with (S=2001:db8:0:a010::32, D=2001:db8:0:5555::51)
   and those packets need to be forward out the link from SERa to ISP-A.










Baker, et al.           Expires February 1, 2020               [Page 14]

Internet-Draft          Enterprise PA Multihoming              July 2019


                                         2001:db8:0:1234::101    H101
                                                                  |
                                                                  |
 2001:db8:0:a010::31                                          --------
 2001:db8:0:b010::31                            ,-----.      /        \
                    +--+   +--+       +----+  ,'       `.   :          :
                +---|R1|---|R4|---+---|SERa|-+   ISP-A   +--+--        :
           H31--+   +--+   +--+   |   +----+  `.       ,'   :          :
                |                 |             `-----'     : Internet :
                |                 |                |        :          :
                |                 |               H51       :          :
                |                 |     2001:db8:0:5555::51 :          :
                |               +--+                        :          :
                |               |R7|                        :          :
                |               +--+                        :          :
                |                 |                         :          :
                |                 |             ,-----.     :          :
           H32--+   +--+          |  +-----+  ,'       `.   :          :
                +---|R2|-----+----+--|SERb1|-+   ISP-B   +--+--        :
                    +--+     |       +-----+  `.       ,'   :          :
                           +--+                 `--|--'     :          :
  2001:db8:0:a010::32      |R8|                    |         \        /
                           +--+                 ,--|--.       --------
                             |       +-----+  ,'       `.         |
                             +-------|SERb2|-+   ISP-B   |        |
                             |       +-----+  `.       ,'       H501
                             |                  `-----'  2001:db8:0:5678
                             |                     |               ::501
                     +--+  +--+                   H61
            H41------|R3|--|R5|           2001:db8:0:6666::61
                     +--+  +--+

 2001:db8:0:a020::41
 2001:db8:0:b020::41

     Figure 3: Internet access and services offered by ISP-A and ISP-B

   ISP-B illustrates a variation on this scenario.  In addition to
   Internet access, ISP-B also offers a service which requires the site
   to access host H61.  The site has two connections to two different
   parts of ISP-B (shown as SERb1 and SERb2 in Figure 3).  ISP-B expects
   Internet traffic to use the uplink from SERb1, while it expects
   traffic destined for the service at H61 to use the uplink from SERb2.
   For either uplink, ISP-B expects the ingress traffic to have a source
   address matching the prefix it assigned to the site,
   2001:db8:0:b000::/52.





Baker, et al.           Expires February 1, 2020               [Page 15]

Internet-Draft          Enterprise PA Multihoming              July 2019


   As discussed before, we rely completely on the internal host to set
   the source address of the packet properly.  In the case of a packet
   sent by H31 to access the service in ISP-B at H61, we expect the
   packet to have the following addresses: (S=2001:db8:0:b010::31,
   D=2001:db8:0:6666::61).  The routed network has two potential ways of
   distributing routes so that this packet exits the site on the uplink
   at SERb2.

   We could just rely on normal destination routes, without using
   source-prefix scoped routes.  If we have SERb2 originate a normal
   unscoped destination route for D=2001:db8:0:6666::/64, the packets
   from H31 to H61 will exit the site at SERb2 as desired.  We should
   not have to worry about SERa needing to originate the same route,
   because ISP-B should choose a globally unique prefix for the service
   at H61.

   The alternative is to have SERb2 originate a source-prefix-scoped
   destination route of the form (S=2001:db8:0:b000::/52,
   D=2001:db8:0:6666::/64).  From a forwarding point of view, the use of
   the source-prefix-scoped destination route would result in traffic
   with source addresses corresponding only to ISP-B being sent to
   SERb2.  Instead, the use of the unscoped destination route would
   result in traffic with source addresses corresponding to ISP-A and
   ISP-B being sent to SERb2, as long as the destination address matches
   the destination prefix.  It seems like either forwarding behavior
   would be acceptable.

   However, from the point of view of the enterprise network
   administrator trying to configure, maintain, and trouble-shoot this
   multihoming solution, it seems much clearer to have SERb2 originate
   the source-prefix-scoped destination route correspond to the service
   offered by ISP-B.  In this way, all of the traffic leaving the site
   is determined by the source-prefix-scoped routes, and all of the
   traffic within the site or arriving from external hosts is determined
   by the unscoped destination routes.  Therefore, for this multihoming
   solution we choose to originate source-prefix-scoped routes for all
   traffic leaving the site.

4.5.  ISPs and Provider-Assigned Prefixes

   While we expect that most site multihoming involves connecting to
   only two ISPs, this solution allows for connections to an arbitrary
   number of ISPs to be supported.  However, when evaluating scalable
   implementations of the solution, it would be reasonable to assume
   that the maximum number of ISPs that a site would connect to is five
   (topologies with two redundant routers each having two uplinks to
   different ISPs plus a tunnel to a headoffice acting as fifth one are
   not unheard of).



Baker, et al.           Expires February 1, 2020               [Page 16]

Internet-Draft          Enterprise PA Multihoming              July 2019


   It is also useful to note that the prefixes assigned to the site by
   different ISPs will not overlap.  This must be the case, since the
   provider-assigned addresses have to be globally unique.

4.6.  Simplified Topologies

   The topologies of many enterprise sites using this multihoming
   solution may in practice be simpler than the examples that we have
   used.  The topology in Figure 1 could be further simplified by having
   all hosts directly connected to the LAN connecting the two site exit
   routers, SERa and SERb.  The topology could also be simplified by
   having the uplinks to ISP-A and ISP-B both connected to the same site
   exit router.  However, it is the aim of this document to provide a
   solution that applies to a broad a range of enterprise site network
   topologies, so this document focuses on providing a solution to the
   more general case.  The simplified cases will also be supported by
   this solution, and there may even be optimizations that can be made
   for simplified cases.  This solution however needs to support more
   complex topologies.

   We are starting with the basic assumption that enterprise site
   networks can be quite complex from a routing perspective.  However,
   even a complex site network can be multihomed to different ISPs with
   PA addresses using IPv4 and NAT.  It is not reasonable to expect an
   enterprise network operator to change the routing topology of the
   site in order to deploy IPv6.

5.  Generating Source-Prefix-Scoped Forwarding Tables

   So far we have described in general terms how the routers in this
   solution that are capable of Source Address Dependent Routing will
   forward traffic using both normal unscoped destination routes and
   source-prefix-scoped destination routes.  Here we give a precise
   method for generating a source-prefix-scoped forwarding table on a
   router that supports SADR.

   1.  Compute the next-hops for the source-prefix-scoped destination
       prefixes using only routers in the connected SADR domain.  These
       are the initial source-prefix-scoped forwarding table entries.

   2.  Compute the next-hops for the unscoped destination prefixes using
       all routers in the IGP.  This is the unscoped forwarding table.

   3.  For a given source-prefix-scoped forwarding table T (scoped to
       source prefix P), consider a source-prefix-scoped forwarding
       table T', whose source prefix P' contains P.  We call T the more
       specific source-prefix-scoped forwarding table, and T' the less
       specific source-prefix-scoped forwarding table.  We select



Baker, et al.           Expires February 1, 2020               [Page 17]

Internet-Draft          Enterprise PA Multihoming              July 2019


       entries in the less specific source-prefix-scoped forwarding
       table to augment the more specific source-prefix-scoped
       forwarding table based on the following rules.  If a destination
       prefix of an entry in the less specific source-prefix-scoped
       forwarding table exactly matches the destination prefix of an
       existing entry in the more specific source-prefix-scoped
       forwarding table (including destination prefix length), then do
       not add the entry to the more specific source-prefix-scoped
       forwarding table.  If the destination prefix does NOT match an
       existing entry, then add the entry to the more specific source-
       prefix-scoped forwarding table.  As the unscoped forwarding table
       is considered to be scoped to ::/0, this process will propagate
       routes from the unscoped forwarding table to the more specific
       source-prefix-scoped forwarding table.  If there exist multiple
       source-prefix-scoped forwarding tables whose source prefixes
       contain P, these source-prefix-scoped forwarding tables should be
       processed in order from most specific to least specific.

   The forwarding tables produced by this process are used in the
   following way to forward packets.

   1.  Select the most specific (longest prefix match) source-prefix-
       scoped forwarding table that matches the source address of the
       packet (again, the unscoped forwarding table is considered to be
       scoped to ::/0).

   2.  Look up the destination address of the packet in the selected
       forwarding table to determine the next-hop for the packet.

   The following example illustrates how this process is used to create
   a forwarding table for each provider-assigned source prefix.  We
   consider the multihomed site network in Figure 3.  Initially we
   assume that all of the routers in the site network support SADR.
   Figure 4 shows the routes that are originated by the routers in the
   site network.
















Baker, et al.           Expires February 1, 2020               [Page 18]

Internet-Draft          Enterprise PA Multihoming              July 2019


   Routes originated by SERa:
   (S=2001:db8:0:a000::/52, D=2001:db8:0:5555/64)
   (S=2001:db8:0:a000::/52, D=::/0)
   (D=2001:db8:0:5555::/64)
   (D=::/0)

   Routes originated by SERb1:
   (S=2001:db8:0:b000::/52, D=::/0)
   (D=::/0)

   Routes originated by SERb2:
   (S=2001:db8:0:b000::/52, D=2001:db8:0:6666::/64)
   (D=2001:db8:0:6666::/64)

   Routes originated by R1:
   (D=2001:db8:0:a010::/64)
   (D=2001:db8:0:b010::/64)

   Routes originated by R2:
   (D=2001:db8:0:a010::/64)
   (D=2001:db8:0:b010::/64)

   Routes originated by R3:
   (D=2001:db8:0:a020::/64)
   (D=2001:db8:0:b020::/64)

        Figure 4: Routes Originated by Routers in the Site Network

   Each SER originates destination routes which are scoped to the source
   prefix assigned by the ISP that the SER connects to.  Note that the
   SERs also originate the corresponding unscoped destination route.
   This is not needed when all of the routers in the site support SADR.
   However, it is required when some routers do not support SADR.  This
   will be discussed in more detail later.

   We focus on how R8 constructs its source-prefix-scoped forwarding
   tables from these route advertisements.  R8 computes the next hops
   for destination routes which are scoped to the source prefix
   2001:db8:0:a000::/52.  The results are shown in the first table in
   Figure 5.  (In this example, the next hops are computed assuming that
   all links have the same metric.)  Then, R8 computes the next hops for
   destination routes which are scoped to the source prefix
   2001:db8:0:b000::/52.  The results are shown in the second table in
   Figure 5 . Finally, R8 computes the next hops for the unscoped
   destination prefixes.  The results are shown in the third table in
   Figure 5.





Baker, et al.           Expires February 1, 2020               [Page 19]

Internet-Draft          Enterprise PA Multihoming              July 2019


   forwarding entries scoped to
   source prefix = 2001:db8:0:a000::/52
   ============================================
   D=2001:db8:0:5555/64      NH=R7
   D=::/0                    NH=R7

   forwarding entries scoped to
   source prefix = 2001:db8:0:b000::/52
   ============================================
   D=2001:db8:0:6666/64      NH=SERb2
   D=::/0                    NH=SERb1

   unscoped forwarding entries
   ============================================
   D=2001:db8:0:a010::/64    NH=R2
   D=2001:db8:0:b010::/64    NH=R2
   D=2001:db8:0:a020::/64    NH=R5
   D=2001:db8:0:b020::/64    NH=R5
   D=2001:db8:0:5555::/64    NH=R7
   D=2001:db8:0:6666::/64    NH=SERb2
   D=::/0                    NH=SERb1

                Figure 5: Forwarding Entries Computed at R8

   The final step is for R8 to augment the more specific source-prefix-
   scoped forwarding tables with entries from less specific source-
   prefix-scoped forwarding tables.  The unscoped forwarding table is
   considered as being scoped to ::/0, so both 2001:db8:0:a000::/52 and
   2001:db8:0:b000::/52 are more specific prefixes of ::/0.  Therefore,
   entries in the unscoped forwarding table will be evaluated to be
   added to these two more specific source-prefix-scoped forwarding
   tables.  If a forwarding entry from the less specific source-prefix-
   scoped forwarding table has the exact same destination prefix
   (including destination prefix length) as the forwarding entry from
   the more specific source-prefix-scoped forwarding table, then the
   existing forwarding entry in the more specific source-prefix-scoped
   forwarding table wins.

   As an example of how the source scoped forwarding entries are
   augmented, we consider how the two entries in the first table in
   Figure 5 (the table for source prefix = 2001:db8:0:a000::/52) are
   augmented with entries from the third table in Figure 5 (the table of
   unscoped or scoped to ::/0 forwarding entries).  The first four
   unscoped forwarding entries (D=2001:db8:0:a010::/64,
   D=2001:db8:0:b010::/64, D=2001:db8:0:a020::/64, and
   D=2001:db8:0:b020::/64) are not an exact match for any of the
   existing entries in the forwarding table for source prefix
   2001:db8:0:a000::/52.  Therefore, these four entries are added to the



Baker, et al.           Expires February 1, 2020               [Page 20]

Internet-Draft          Enterprise PA Multihoming              July 2019


   final forwarding table for source prefix 2001:db8:0:a000::/52.  The
   result of adding these entries is reflected in the first four entries
   the first table in Figure 6.

   The next less specific scoped (scope is ::/0) forwarding table entry
   is for D=2001:db8:0:5555::/64.  This entry is an exact match for the
   existing entry in the forwarding table for the more specific source
   prefix 2001:db8:0:a000::/52.  Therefore, we do not replace the
   existing entry with the entry from the unscoped forwarding table.
   This is reflected in the fifth entry in the first table in Figure 6.
   (Note that since both scoped and unscoped entries have R7 as the next
   hop, the result of applying this rule is not visible.)

   The next less specific prefix scoped (scope is ::/0) forwarding table
   entry is for D=2001:db8:0:6666::/64.  This entry is not an exact
   match for any existing entries in the forwarding table for source
   prefix 2001:db8:0:a000::/52.  Therefore, we add this entry.  This is
   reflected in the sixth entry in the first table in Figure 6.

   The next less specific prefix scoped (scope is ::/0) forwarding table
   entry is for D=::/0.  This entry is an exact match for the existing
   entry in the forwarding table for more specific source prefix
   2001:db8:0:a000::/52.  Therefore, we do not overwrite the existing
   source-prefix-scoped entry, as can be seen in the last entry in the
   first table in Figure 6.


























Baker, et al.           Expires February 1, 2020               [Page 21]

Internet-Draft          Enterprise PA Multihoming              July 2019


   if source address matches 2001:db8:0:a000::/52
   then use this forwarding table
   ============================================
   D=2001:db8:0:a010::/64    NH=R2
   D=2001:db8:0:b010::/64    NH=R2
   D=2001:db8:0:a020::/64    NH=R5
   D=2001:db8:0:b020::/64    NH=R5
   D=2001:db8:0:5555::/64    NH=R7
   D=2001:db8:0:6666::/64    NH=SERb2
   D=::/0                    NH=R7

   else if source address matches 2001:db8:0:b000::/52
   then use this forwarding table
   ============================================
   D=2001:db8:0:a010::/64    NH=R2
   D=2001:db8:0:b010::/64    NH=R2
   D=2001:db8:0:a020::/64    NH=R5
   D=2001:db8:0:b020::/64    NH=R5
   D=2001:db8:0:5555::/64    NH=R7
   D=2001:db8:0:6666::/64    NH=SERb2
   D=::/0                    NH=SERb1

   else if source address matches ::/0 use this forwarding table
   ============================================
   D=2001:db8:0:a010::/64    NH=R2
   D=2001:db8:0:b010::/64    NH=R2
   D=2001:db8:0:a020::/64    NH=R5
   D=2001:db8:0:b020::/64    NH=R5
   D=2001:db8:0:5555::/64    NH=R7
   D=2001:db8:0:6666::/64    NH=SERb2
   D=::/0                    NH=SERb1

            Figure 6: Complete Forwarding Tables Computed at R8

   The forwarding tables produced by this process at R8 have the desired
   properties.  A packet with a source address in 2001:db8:0:a000::/52
   will be forwarded based on the first table in Figure 6.  If the
   packet is destined for the Internet at large or the service at
   D=2001:db8:0:5555/64, it will be sent to R7 in the direction of SERa.
   If the packet is destined for an internal host, then the first four
   entries will send it to R2 or R5 as expected.  Note that if this
   packet has a destination address corresponding to the service offered
   by ISP-B (D=2001:db8:0:5555::/64), then it will get forwarded to
   SERb2.  It will be dropped by SERb2 or by ISP-B, since the packet has
   a source address that was not assigned by ISP-B.  However, this is
   expected behavior.  In order to use the service offered by ISP-B, the
   host needs to originate the packet with a source address assigned by
   ISP-B.



Baker, et al.           Expires February 1, 2020               [Page 22]

Internet-Draft          Enterprise PA Multihoming              July 2019


   In this example, a packet with a source address that doesn't match
   2001:db8:0:a000::/52 or 2001:db8:0:b000::/52 must have originated
   from an external host.  Such a packet will use the unscoped
   forwarding table (the last table in Figure 6).  These packets will
   flow exactly as they would in absence of multihoming.

   We can also modify this example to illustrate how it supports
   deployments where not all routers in the site support SADR.
   Continuing with the topology shown in Figure 3, suppose that R3 and
   R5 do not support SADR.  Instead they are only capable of
   understanding unscoped route advertisements.  The SADR routers in the
   network will still originate the routes shown in Figure 4.  However,
   R3 and R5 will only understand the unscoped routes as shown in
   Figure 7.

   Routes originated by SERa:
   (D=2001:db8:0:5555::/64)
   (D=::/0)

   Routes originated by SERb1:
   (D=::/0)

   Routes originated by SERb2:
   (D=2001:db8:0:6666::/64)

   Routes originated by R1:
   (D=2001:db8:0:a010::/64)
   (D=2001:db8:0:b010::/64)

   Routes originated by R2:
   (D=2001:db8:0:a010::/64)
   (D=2001:db8:0:b010::/64)

   Routes originated by R3:
   (D=2001:db8:0:a020::/64)
   (D=2001:db8:0:b020::/64)

     Figure 7: Routes Advertisements Understood by Routers that do no
                               Support SADR

   With these unscoped route advertisements, R5 will produce the
   forwarding table shown in Figure 8.









Baker, et al.           Expires February 1, 2020               [Page 23]

Internet-Draft          Enterprise PA Multihoming              July 2019


   forwarding table
   ============================================
   D=2001:db8:0:a010::/64    NH=R8
   D=2001:db8:0:b010::/64    NH=R8
   D=2001:db8:0:a020::/64    NH=R3
   D=2001:db8:0:b020::/64    NH=R3
   D=2001:db8:0:5555::/64    NH=R8
   D=2001:db8:0:6666::/64    NH=SERb2
   D=::/0                    NH=R8

    Figure 8: Forwarding Table For R5, Which Doesn't Understand Source-
                           Prefix-Scoped Routes

   As all SERs belong to the SADR domain any traffic that needs to exit
   the site will eventually hit a SADR-capable router.  To prevent
   routing loops involving SADR-capable and non-SADR-capable routers,
   traffic that enters the SADR-capable domain does not leave the domain
   until it exits the site.  Therefore all SADR-capable routers within
   the domain MUST be logically connected.

   Note that the mechanism described here for converting source-prefix-
   scoped destination prefix routing advertisements into forwarding
   state is somewhat different from that proposed in
   [I-D.ietf-rtgwg-dst-src-routing].  The method described in the
   current document is functionally equivalent, but it is based on
   application of existing mechanisms for the described scenarios.

6.  Mechanisms For Hosts To Choose Good Default Source Addresses In A
    Multihomed Site

   Until this point, we have made the assumption that hosts are able to
   choose the correct source address using some unspecified mechanism.
   This has allowed us to just focus on what the routers in a multihomed
   site network need to do in order to forward packets to the correct
   ISP based on source address.  Now we look at possible mechanisms for
   hosts to choose the correct source address.  We also look at what
   role, if any, the routers may play in providing information that
   helps hosts to choose source addresses.

   It should be noted that this section discussed how hosts could select
   the default source address for new connections.  Any connection which
   already exists on a host is bound to the specific source address
   which can not be changed.  Section 6.7 discusses the connections
   preservation issue in more details.

   Any host that needs to be able to send traffic using the uplinks to a
   given ISP is expected to be configured with an address from the
   prefix assigned by that ISP.  The host will control which ISP is used



Baker, et al.           Expires February 1, 2020               [Page 24]

Internet-Draft          Enterprise PA Multihoming              July 2019


   for its traffic by selecting one of the addresses configured on the
   host as the source address for outgoing traffic.  It is the
   responsibility of the site network to ensure that a packet with the
   source address from an ISP is now sent on an uplink to that ISP.

   If all of the ISP uplinks are working, the choice of source address
   by the host may be driven by the desire to load share across ISP
   uplinks, or it may be driven by the desire to take advantage of
   certain properties of a particular uplink or ISP (if some information
   about various path properties has been made availabe to the host
   somehow - see [I-D.ietf-intarea-provisioning-domains] as an example).
   If any of the ISP uplinks is not working, then the choice of source
   address by the host can cause packets to get dropped.

   How a host should make good decisions about source address selection
   in a multihomed site is not a solved problem.  We do not attempt to
   solve this problem in this document.  Instead we discuss the current
   state of affairs with respect to standardized solutions and
   implementation of those solutions.  We also look at proposed
   solutions for this problem.

   An external host initiating communication with a host internal to a
   PA multihomed site will need to know multiple addresses for that host
   in order to communicate with it using different ISPs to the
   multihomed site (knowing just one address would undermine all
   benefits of redundant connectivity provided by multihoming).  These
   addresses are typically learned through DNS.  (For simplicity, we
   assume that the external host is single-homed.)  The external host
   chooses the ISP that will be used at the remote multihomed site by
   setting the destination address on the packets it transmits.  For a
   session originated from an external host to an internal host, the
   choice of source address used by the internal host is simple.  The
   internal host has no choice but to use the destination address in the
   received packet as the source address of the transmitted packet.

   For a session originated by a host inside the multi-homed site, the
   decision of what source address to select is more complicated.  We
   consider three main methods for hosts to get information about the
   network.  The two proactive methods are Neighbor Discovery Router
   Advertisements and DHCPv6.  The one reactive method we consider is
   ICMPv6.  Note that we are explicitly excluding the possibility of
   having hosts participate in or even listen directly to routing
   protocol advertisements.

   First we look at how a host is currently expected to select the
   default source and destination addresses to be used for a new
   connection.




Baker, et al.           Expires February 1, 2020               [Page 25]

Internet-Draft          Enterprise PA Multihoming              July 2019


6.1.  Default Source Address Selection Algorithm on Hosts

   [RFC6724] defines the algorithms that hosts are expected to use to
   select source and destination addresses for packets.  It defines an
   algorithm for selecting a source address and a separate algorithm for
   selecting a destination address.  Both of these algorithms depend on
   a policy table.  [RFC6724] defines a default policy which produces
   certain behavior.

   The rules in the two algorithms in [RFC6724] depend on many different
   properties of addresses.  While these are needed for understanding
   how a host should choose addresses in an arbitrary environment, most
   of the rules are not relevant for understanding how a host should
   choose among multiple source addresses in multihomed environment when
   sending a packet to a remote host.  Returning to the example in
   Figure 3, we look at what the default algorithms in [RFC6724] say
   about the source address that internal host H31 should use to send
   traffic to external host H101, somewhere on the Internet.

   There is no choice to be made with respect to destination address.
   H31 needs to send a packet with D=2001:db8:0:1234::101 in order to
   reach H101.  So H31 have to choose between using
   S=2001:db8:0:a010::31 or S=2001:db8:0:b010::31 as the source address
   for this packet.  We go through the rules for source address
   selection in Section 5 of [RFC6724].

   Rule 1 (Prefer same address) is not useful to break the tie between
   source addresses, because neither the candidate source addresses
   equals the destination address.

   Rule 2 (Prefer appropriate scope) is also not used in this scenario,
   because both source addresses and the destination address have global
   scope.

   Rule 3 (Avoid deprecated addresses) applies to an address that has
   been autoconfigured by a host using stateless address
   autoconfiguration as defined in [RFC4862].  An address autoconfigured
   by a host has a preferred lifetime and a valid lifetime.  The address
   is preferred until the preferred lifetime expires, after which it
   becomes deprecated.  A deprecated address is not used if there is a
   preferred address of the appropriate scope available.  When the valid
   lifetime expires, the address cannot be used at all.  The preferred
   and valid lifetimes for an autoconfigured address are set based on
   the corresponding lifetimes in the Prefix Information Option in
   Neighbor Discovery Router Advertisements.  So a possible tool to
   control source address selection in this scenario would be for a host
   to make an address deprecated by having routers on that link, R1 and
   R2 in Figure 3, send a Router Advertisement message containing a



Baker, et al.           Expires February 1, 2020               [Page 26]

Internet-Draft          Enterprise PA Multihoming              July 2019


   Prefix Information Option for the source prefix to be discouraged (or
   prohibited) with the preferred lifetime set to zero.  This is a
   rather blunt tool, because it discourages or prohibits the use of
   that source prefix for all destinations.  However, it may be useful
   in some scenarios.  For example, if all uplinks to a particular ISP
   fail, it is desirable to prevent hosts from using source addresses
   from that ISP address space.

   Rule 4 (Avoid home addresses) does not apply here because we are not
   considering Mobile IP.

   Rule 5 (Prefer outgoing interface) is not useful in this scenario,
   because both source addresses are assigned to the same interface.

   Rule 5.5 (Prefer addresses in a prefix advertised by the next-hop) is
   not useful in the scenario when both R1 and R2 will advertise both
   source prefixes.  However potentially this rule may allow a host to
   select the correct source prefix by selecting a next-hop.  The most
   obvious way would be to make R1 to advertise itself as a default
   router and send PIO for 2001:db8:0:a010::/64, while R2 is advertising
   itself as a default router and sending PIO for 2001:db8:0:b010::/64.
   We'll discuss later how Rule 5.5 can be used to influence a source
   address selection in single-router topologies (e.g. when H41 is
   sending traffic using R3 as a default gateway).

   Rule 6 (Prefer matching label) refers to the Label value determined
   for each source and destination prefix as a result of applying the
   policy table to the prefix.  With the default policy table defined in
   Section 2.1 of [RFC6724], Label(2001:db8:0:a010::31) = 5,
   Label(2001:db8:0:b010::31) = 5, and Label(2001:db8:0:1234::101) = 5.
   So with the default policy, Rule 6 does not break the tie.  However,
   the algorithms in [RFC6724] are defined in such a way that non-
   default address selection policy tables can be used.  [RFC7078]
   defines a way to distribute a non-default address selection policy
   table to hosts using DHCPv6.  So even though the application of rule
   6 to this scenario using the default policy table is not useful, rule
   6 may still be a useful tool.

   Rule 7 (Prefer temporary addresses) has to do with the technique
   described in [RFC4941] to periodically randomize the interface
   portion of an IPv6 address that has been generated using stateless
   address autoconfiguration.  In general, if H31 were using this
   technique, it would use it for both source addresses, for example
   creating temporary addresses 2001:db8:0:a010:2839:9938:ab58:830f and
   2001:db8:0:b010:4838:f483:8384:3208, in addition to
   2001:db8:0:a010::31 and 2001:db8:0:b010::31.  So this rule would
   prefer the two temporary addresses, but it would not break the tie
   between the two source prefixes from ISP-A and ISP-B.



Baker, et al.           Expires February 1, 2020               [Page 27]

Internet-Draft          Enterprise PA Multihoming              July 2019


   Rule 8 (Use longest matching prefix) dictates that between two
   candidate source addresses the one which has longest common prefix
   length with the destination address.  For example, if H31 were
   selecting the source address for sending packets to H101, this rule
   would not be a tie breaker as for both candidate source addresses
   2001:db8:0:a101::31 and 2001:db8:0:b101::31 the common prefix length
   with the destination is 48.  However if H31 were selecting the source
   address for sending packets H41 address 2001:db8:0:a020::41, then
   this rule would result in using 2001:db8:0:a101::31 as a source
   (2001:db8:0:a101::31 and 2001:db8:0:a020::41 share the common prefix
   2001:db8:0:a000::/58, while for 2001:db8:0:b101::31 and
   2001:db8:0:a020::41 the common prefix is 2001:db8:0:a000::/51).
   Therefore rule 8 might be useful for selecting the correct source
   address in some but not all scenarios (for example if ISP-B services
   belong to 2001:db8:0:b000::/59 then H31 would always use
   2001:db8:0:b010::31 to access those destinations).

   So we can see that of the 8 source selection address rules from
   [RFC6724], four actually apply to our basic site multihoming
   scenario.  The rules that are relevant to this scenario are
   summarized below.

   o  Rule 3: Avoid deprecated addresses.

   o  Rule 5.5: Prefer addresses in a prefix advertised by the next-hop.

   o  Rule 6: Prefer matching label.

   o  Rule 8: Prefer longest matching prefix.

   The two methods that we discuss for controlling the source address
   selection through the four relevant rules above are SLAAC Router
   Advertisement messages and DHCPv6.

   We also consider a possible role for ICMPv6 for getting traffic-
   driven feedback from the network.  With the source address selection
   algorithm discussed above, the goal is to choose the correct source
   address on the first try, before any traffic is sent.  However,
   another strategy is to choose a source address, send the packet, get
   feedback from the network about whether or not the source address is
   correct, and try another source address if it is not.

   We consider four scenarios where a host needs to select the correct
   source address.  The first is when both uplinks are working.  The
   second is when one uplink has failed.  The third one is a situation
   when one failed uplink has recovered.  The last one is failure of
   both (all) uplinks.




Baker, et al.           Expires February 1, 2020               [Page 28]

Internet-Draft          Enterprise PA Multihoming              July 2019


   It should be noted that [RFC6724] only defines the behavior of IPv6
   hosts to select default addresses that applications and upper-layer
   protocols can use.  Applications and upper-layer protocols can make
   their own choices on selecting source addresses.  The mechanism
   proposed in this document attempts to ensure that the subset of
   source addresses available for applications and upper-layer protocols
   is selected with the up-to-date network state in mind.  The rest of
   the document discusses various aspects of the default source address
   selection defined in [RFC6724], calling it for the sake of brevity
   "the source address selection".

6.2.  Selecting Default Source Address When Both Uplinks Are Working

   Again we return to the topology in Figure 3.  Suppose that the site
   administrator wants to implement a policy by which all hosts need to
   use ISP-A to reach H101 at D=2001:db8:0:1234::101.  So for example,
   H31 needs to select S=2001:db8:0:a010::31.

6.2.1.  Distributing Default Address Selection Policy Table with DHCPv6

   This policy can be implemented by using DHCPv6 to distribute an
   address selection policy table that assigns the same label to
   destination address that match 2001:db8:0:1234::/64 as it does to
   source addresses that match 2001:db8:0:a000::/52.  The following two
   entries accomplish this.

               Prefix                 Precedence       Label
               2001:db8:0:1234::/64   50               33
               2001:db8:0:a000::/52   50               33

       Figure 9: Policy table entries to implement a routing policy

   This requires that the hosts implement [RFC6724], the basic source
   and destination address framework, along with [RFC7078], the DHCPv6
   extension for distributing a non-default policy table.  Note that it
   does NOT require that the hosts use DHCPv6 for address assignment.
   The hosts could still use stateless address autoconfiguration for
   address configuration, while using DHCPv6 only for policy table
   distribution (see [RFC8415]).  However this method has a number of
   disadvantages:

   o  DHCPv6 support is not a mandatory requirement for IPv6 hosts
      ([RFC6434]), so this method might not work for all devices.

   o  Network administrators are required to explicitly configure the
      desired network access policies on DHCPv6 servers.  While it might
      be feasible in the scenario of a single multihomed network, such
      approach might have some scalability issues, especially if the



Baker, et al.           Expires February 1, 2020               [Page 29]

Internet-Draft          Enterprise PA Multihoming              July 2019


      centralized DHCPv6 solution is deployed to serve a large number of
      multiomed sites.

6.2.2.  Controlling Default Source Address Selection With Router
        Advertisements

   Neighbor Discovery currently has two mechanisms to communicate prefix
   information to hosts.  The base specification for Neighbor Discovery
   (see [RFC4861]) defines the Prefix Information Option (PIO) in the
   Router Advertisement (RA) message.  When a host receives a PIO with
   the A-flag set, it can use the prefix in the PIO as source prefix
   from which it assigns itself an IP address using stateless address
   autoconfiguration (SLAAC) procedures described in [RFC4862].  In the
   example of Figure 3, if the site network is using SLAAC, we would
   expect both R1 and R2 to send RA messages with PIOs for both source
   prefixes 2001:db8:0:a010::/64 and 2001:db8:0:b010::/64 with the
   A-flag set.  H31 would then use the SLAAC procedure to configure
   itself with the 2001:db8:0:a010::31 and 2001:db8:0:b010::31.

   Whereas a host learns about source prefixes from PIO messages, hosts
   can learn about a destination prefix from a Router Advertisement
   containing Route Information Option (RIO), as specified in [RFC4191].
   The destination prefixes in RIOs are intended to allow a host to
   choose the router that it uses as its first hop to reach a particular
   destination prefix.

   As currently standardized, neither PIO nor RIO options contained in
   Neighbor Discovery Router Advertisements can communicate the
   information needed to implement the desired routing policy.  PIO's
   communicate source prefixes, and RIO communicate destination
   prefixes.  However, there is currently no standardized way to
   directly associate a particular destination prefix with a particular
   source prefix.

   [I-D.pfister-6man-sadr-ra] proposes a Source Address Dependent Route
   Information option for Neighbor Discovery Router Advertisements which
   would associate a source prefix and with a destination prefix.  The
   details of [I-D.pfister-6man-sadr-ra] might need tweaking to address
   this use case.  However, in order to be able to use Neighbor
   Discovery Router Advertisements to implement this routing policy, an
   extension that allows R1 and R2 to explicitly communicate to H31 an
   association between S=2001:db8:0:a000::/52 D=2001:db8:0:1234::/64
   would be needed.

   However, Rule 5.5 of the default source address selection algorithm
   (discussed in Section 6.1 above), together with default router
   preference (specified in [RFC4191]) and RIO can be used to influence
   a source address selection on a host as described below.  Let's look



Baker, et al.           Expires February 1, 2020               [Page 30]

Internet-Draft          Enterprise PA Multihoming              July 2019


   at source address selection on the host H41.  It receives RAs from R3
   with PIOs for 2001:db8:0:a020::/64 and 2001:db8:0:b020::/64.  At that
   point all traffic would use the same next-hop (R3 link-local address)
   so Rule 5.5 does not apply.  Now let's assume that R3 supports SADR
   and has two scoped forwarding tables, one scoped to
   S=2001:db8:0:a000::/52 and another scoped to S=2001:db8:0:b000::/52.
   If R3 generates two different link-local addresses for its interface
   facing H41 (one for each scoped forwarding table, LLA_A and LLA_B)
   and starts sending two different RAs: one is sent from LLA_A and
   includes PIO for 2001:db8:0:a020::/64, another is sent from LLA_B and
   includes PIO for 2001:db8:0:b020::/64.  Now it is possible to
   influence H41 source address selection for destinations which follow
   the default route by setting default router preference in RAs.  If it
   is desired that H41 reaches H101 (or any destinations in the
   Internet) via ISP-A, then RAs sent from LLA_A should have default
   router preference set to 01 (high priority), while RAs sent from
   LLA_B should have preference set to 11 (low).  Then LLA_A would be
   chosen as a next-hop for H101 and therefore (as per rule 5.5)
   2001:db8:0:a020::41 would be selected as the source address.  If, at
   the same time, it is desired that H61 is accessible via ISP-B then R3
   should include a RIO for 2001:db8:0:6666::/64 to its RA sent from
   LLA_B.  H41 would chose LLA_B as a next-hop for all traffic to H61
   and then as per Rule 5.5, 2001:db8:0:b020::41 would be selected as a
   source address.

   If in the above mentioned scenario it is desirable that all Internet
   traffic leaves the network via ISP-A and the link to ISP-B is used
   for accessing ISP-B services only (not as ISP-A link backup), then
   RAs sent by R3 from LLA_B should have Router Lifetime set to 0 and
   should include RIOs for ISP-B address space.  It would instruct H41
   to use LLA_A for all Internet traffic but use LLA_B as a next-hop
   while sending traffic to ISP-B addresses.

   The description of the mechanism above assumes SADR support by the
   first-hop routers as well as SERs.  However, a first-hop router can
   still provide a less flexible version of this mechanism even without
   implementing SADR.  This could be done by providing configuration
   knobs on the first-hop router that allow it to generate different
   link-local addresses and to send individual RAs for each prefix.

   The mechanism described above relies on Rule 5.5 of the default
   source address selection algorithm defined in [RFC6724].  [RFC8028]
   states that "A host SHOULD select default routers for each prefix it
   is assigned an address in".  It also recommends that hosts should
   implement Rule 5.5. of [RFC6724].  Hosts following the
   recommendations specified in [RFC8028] therefore should be able to
   benefit from the solution described in this document.  No standards
   need to be updated in regards to host behavior.



Baker, et al.           Expires February 1, 2020               [Page 31]

Internet-Draft          Enterprise PA Multihoming              July 2019


6.2.3.  Controlling Default Source Address Selection With ICMPv6

   We now discuss how one might use ICMPv6 to implement the routing
   policy to send traffic destined for H101 out the uplink to ISP-A,
   even when uplinks to both ISPs are working.  If H31 started sending
   traffic to H101 with S=2001:db8:0:b010::31 and
   D=2001:db8:0:1234::101, it would be routed through SER-b1 and out the
   uplink to ISP-B.  SERb1 could recognize that this traffic is not
   following the desired routing policy and react by sending an ICMPv6
   message back to H31.

   In this example, we could arrange things so that SERb1 drops the
   packet with S=2001:db8:0:b010::31 and D=2001:db8:0:1234::101, and
   then sends to H31 an ICMPv6 Destination Unreachable message with Code
   5 (Source address failed ingress/egress policy).  When H31 receives
   this packet, it would then be expected to try another source address
   to reach the destination.  In this example, H31 would then send a
   packet with S=2001:db8:0:a010::31 and D=2001:db8:0:1234::101, which
   will reach SERa and be forwarded out the uplink to ISP-A.

   However, we would also want it to be the case that SERb1 does not
   enforce this routing policy when the uplink from SERa to ISP-A has
   failed.  This could be accomplished by having SERa originate a
   source-prefix-scoped route for (S=2001:db8:0:a000::/52,
   D=2001:db8:0:1234::/64) and have SERb1 monitor the presence of that
   route.  If that route is not present (because SERa has stopped
   originating it), then SERb1 will not enforce the routing policy, and
   it will forward packets with S=2001:db8:0:b010::31 and
   D=2001:db8:0:1234::101 out its uplink to ISP-B.

   We can also use this source-prefix-scoped route originated by SERa to
   communicate the desired routing policy to SERb1.  We can define an
   EXCLUSIVE flag to be advertised together with the IGP route for
   (S=2001:db8:0:a000::/52, D=2001:db8:0:1234::/64).  This would allow
   SERa to communicate to SERb that SERb should reject traffic for
   D=2001:db8:0:1234::/64 and respond with an ICMPv6 Destination
   Unreachable Code 5 message, as long as the route for
   (S=2001:db8:0:a000::/52, D=2001:db8:0:1234::/64) is present.  The
   definition of an EXCLUSIVE flag for SADR advertisements in IGPs would
   require future standardization work.

   Finally, if we are willing to extend ICMPv6 to support this solution,
   then we could create a mechanism for SERb1 to tell the host what
   source address it should be using to successfully forward packets
   that meet the policy.  In its current form, when SERb1 sends an
   ICMPv6 Destination Unreachable Code 5 message, it is basically
   saying, "This source address is wrong.  Try another source address."
   In the absence of a clear indication which address to try next, the



Baker, et al.           Expires February 1, 2020               [Page 32]

Internet-Draft          Enterprise PA Multihoming              July 2019


   host will iterate over all addresses assigned to the interface (e.g.
   various privacy addresses) which would lead to significant delays and
   degraded user experience.  It would be better is if the ICMPv6
   message could say, "This source address is wrong.  Instead use a
   source address in S=2001:db8:0:a000::/52.".

   However using ICMPv6 for signaling source address information back to
   hosts introduces new challenges.  Most routers currently have
   software or hardware limits on generating ICMP messages.  A site
   administrator deploying a solution that relies on the SERs generating
   ICMP messages could try to improve the performance of SERs for
   generating ICMP messages.  However, in a large network, it is still
   likely that ICMP message generation limits will be reached.  As a
   result hosts would not receive ICMPv6 back which in turn leads to
   traffic blackholing and poor user experience.  To improve the
   scalability of ICMPv6-based signaling hosts SHOULD cache the
   preferred source address (or prefix) for the given destination (which
   in turn might cause issues in case of the corresponding ISP uplinks
   failure - see Section 6.3).  In addition, the same source prefix
   SHOULD be used for other destinations in the same /64 as the original
   destination address.  The source prefix to the destination mapping
   SHOULD have a specific lifetime.  Expiration of the lifetime SHOULD
   trigger the source address selection algorithm again.

   Using ICMPv6 Destination Unreachable Messages with Code 5 to
   influence source address selection introduces some security
   challenges which are discussed in Section 10.

   As currently standardized in [RFC4443], the ICMPv6 Destination
   Unreachable Message with Code 5 would allow for the iterative
   approach to retransmitting packets using different source addresses.
   As currently defined, the ICMPv6 message does not provide a mechanism
   to communication information about which source prefix should be used
   for a retransmitted packet.  The current document does not define
   such a mechanism but it might be a useful extension to define in a
   different document.  However this approach has some security
   implications such as an ability for an attacker to send spoofed
   ICMPv6 messages to signal invalid/unreachable source prefix causing
   DoS-type attack.

6.2.4.  Summary of Methods For Controlling Default Source Address
        Selection To Implement Routing Policy

   So to summarize this section, we have looked at three methods for
   implementing a simple routing policy where all traffic for a given
   destination on the Internet needs to use a particular ISP, even when
   the uplinks to both ISPs are working.




Baker, et al.           Expires February 1, 2020               [Page 33]

Internet-Draft          Enterprise PA Multihoming              July 2019


   The default source address selection policy cannot distinguish
   between the source addresses needed to enforce this policy, so a non-
   default policy table using associating source and destination
   prefixes using Label values would need to be installed on each host.
   A mechanism exists for DHCPv6 to distribute a non-default policy
   table but such solution would heavily rely on DHCPv6 support by host
   operating system.  Moreover there is no mechanism to translate
   desired routing/traffic engineering policies into policy tables on
   DHCPv6 servers.  Therefore using DHCPv6 for controlling address
   selection policy table is not recommended and SHOULD NOT be used.

   At the same time Router Advertisements provide a reliable mechanism
   to influence source address selection process via PIO, RIO and
   default router preferences.  As all those options have been
   standardized by IETF and are supported by various operating systems
   no changes are required on hosts.  First-hop routers in the
   enterprise network need to be able of sending different RAs for
   different SLAAC prefixes (either based on scoped forwarding tables or
   based on pre-configured policies).

   SERs can enforce the routing policy by sending ICMPv6 Destination
   Unreachable messages with Code 5 (Source address failed ingress/
   egress policy) for traffic that is being sent with the wrong source
   address.  The policy distribution could be automated by defining an
   EXCLUSIVE flag for the source-prefix-scoped route which can be set on
   the SER that originates the route.  As ICMPv6 message generation can
   be rate-limited on routers, it SHOULD NOT be used as the only
   mechanism to influence source address selection on hosts.  While
   hosts SHOULD select the correct source address for a given
   destination the network SHOULD signal any source address issues back
   to hosts using ICMPv6 error messages.

6.3.  Selecting Default Source Address When One Uplink Has Failed

   Now we discuss if DHCPv6, Neighbor Discovery Router Advertisements,
   and ICMPv6 can help a host choose the right source address when an
   uplink to one of the ISPs has failed.  Again we look at the scenario
   in Figure 3.  This time we look at traffic from H31 destined for
   external host H501 at D=2001:db8:0:5678::501.  We initially assume
   that the uplink from SERa to ISP-A is working and that the uplink
   from SERb1 to ISP-B is working.

   We assume there is no particular routing policy desired, so H31 is
   free to send packets with S=2001:db8:0:a010::31 or
   S=2001:db8:0:b010::31 and have them delivered to H501.  For this
   example, we assume that H31 has chosen S=2001:db8:0:b010::31 so that
   the packets exit via SERb to ISP-B.  Now we see what happens when the
   link from SERb1 to ISP-B fails.  How should H31 learn that it needs



Baker, et al.           Expires February 1, 2020               [Page 34]

Internet-Draft          Enterprise PA Multihoming              July 2019


   to start sending the packet to H501 with S=2001:db8:0:a010::31 in
   order to start using the uplink to ISP-A?  We need to do this in a
   way that doesn't prevent H31 from still sending packets with
   S=2001:db8:0:b010::31 in order to reach H61 at D=2001:db8:0:6666::61.

6.3.1.  Controlling Default Source Address Selection With DHCPv6

   For this example we assume that the site network in Figure 3 has a
   centralized DHCP server and all routers act as DHCP relay agents.  We
   assume that both of the addresses assigned to H31 were assigned via
   DHCP.

   We could try to have the DHCP server monitor the state of the uplink
   from SERb1 to ISP-B in some manner and then tell H31 that it can no
   longer use S=2001:db8:0:b010::31 by settings its valid lifetime to
   zero.  The DHCP server could initiate this process by sending a
   Reconfigure Message to H31 as described in Section 18.3 of [RFC8415].
   Or the DHCP server can assign addresses with short lifetimes in order
   to force clients to renew them often.

   This approach would prevent H31 from using S=2001:db8:0:b010::31 to
   reach a host on the Internet.  However, it would also prevent H31
   from using S=2001:db8:0:b010::31 to reach H61 at
   D=2001:db8:0:6666::61, which is not desirable.

   Another potential approach is to have the DHCP server monitor the
   uplink from SERb1 to ISP-B and control the choice of source address
   on H31 by updating its address selection policy table via the
   mechanism in [RFC7078].  The DHCP server could initiate this process
   by sending a Reconfigure Message to H31.  Note that [RFC8415]
   requires that Reconfigure Message use DHCP authentication.  DHCP
   authentication could be avoided by using short address lifetimes to
   force clients to send Renew messages to the server often.  If the
   host is not obtaining its IP addresses from the DHCP server, then it
   would need to use the Information Refresh Time option defined in
   [RFC8415].

   If the following policy table can be installed on H31 after the
   failure of the uplink from SERb1, then the desired routing behavior
   should be achieved based on source and destination prefix being
   matched with label values.










Baker, et al.           Expires February 1, 2020               [Page 35]

Internet-Draft          Enterprise PA Multihoming              July 2019


               Prefix                 Precedence       Label
               ::/0                   50               44
               2001:db8:0:a000::/52   50               44
               2001:db8:0:6666::/64   50               55
               2001:db8:0:b000::/52   50               55


      Figure 10: Policy Table Needed On Failure Of Uplink From SERb1

   The described solution has a number of significant drawbacks, some of
   them already discussed in Section 6.2.1.

   o  DHCPv6 support is not required for an IPv6 host and there are
      operating systems which do not support DHCPv6.  Besides that, it
      does not appear that [RFC7078] has been widely implemented on host
      operating systems.

   o  [RFC7078] does not clearly specify this kind of a dynamic use case
      where address selection policy needs to be updated quickly in
      response to the failure of a link.  In a large network it would
      present scalability issues as many hosts need to be reconfigured
      in very short period of time.

   o  Updating DHCPv6 server configuration each time an ISP uplink
      changes its state introduces some scalability issues, especially
      for mid/large distributed scale enterprise networks.  In addition
      to that, the policy table needs to be manually configured by
      administrators which makes that solution prone to human error.

   o  No mechanism exists for making DHCPv6 servers aware of network
      topology/routing changes in the network.  In general DHCPv6
      servers monitoring network-related events sounds like a bad idea
      as completely new functionality beyond the scope of DHCPv6 role is
      required.

6.3.2.  Controlling Default Source Address Selection With Router
        Advertisements

   The same mechanism as discussed in Section 6.2.2 can be used to
   control the source address selection in the case of an uplink
   failure.  If a particular prefix should not be used as a source for
   any destinations, then the router needs to send RA with Preferred
   Lifetime field for that prefix set to 0.

   Let's consider a scenario when all uplinks are operational and H41
   receives two different RAs from R3: one from LLA_A with PIO for
   2001:db8:0:a020::/64, default router preference set to 11 (low) and
   another one from LLA_B with PIO for 2001:db8:0:a020::/64, default



Baker, et al.           Expires February 1, 2020               [Page 36]

Internet-Draft          Enterprise PA Multihoming              July 2019


   router preference set to 01 (high) and RIO for 2001:db8:0:6666::/64.
   As a result H41 is using 2001:db8:0:b020::41 as a source address for
   all Internet traffic and those packets are sent by SERs to ISP-B.  If
   SERb1 uplink to ISP-B failed, the desired behavior is that H41 stops
   using 2001:db8:0:b020::41 as a source address for all destinations
   but H61.  To achieve that R3 should react to SERb1 uplink failure
   (which could be detected as the scoped route (S=2001:db8:0:b000::/52,
   D=::/0) disappearance) by withdrawing itself as a default router.  R3
   sends a new RA from LLA_B with Router Lifetime value set to 0 (which
   means that it should not be used as default router).  That RA still
   contains PIO for 2001:db8:0:b020::/64 (for SLAAC purposes) and RIO
   for 2001:db8:0:6666::/64 so H41 can reach H61 using LLA_B as a next-
   hop and 2001:db8:0:b020::41 as a source address.  For all traffic
   following the default route, LLA_A will be used as a next-hop and
   2001:db8:0:a020::41 as a source address.

   If all uplinks to ISP-B have failed and therefore source addresses
   from ISP-B address space should not be used at all, the forwarding
   table scoped S=2001:db8:0:b000::/52 contains no entries.  Hosts can
   be instructed to stop using source addresses from that block by
   sending RAs containing PIO with Preferred Lifetime set to 0.

6.3.3.  Controlling Default Source Address Selection With ICMPv6

   Now we look at how ICMPv6 messages can provide information back to
   H31.  We assume again that at the time of the failure H31 is sending
   packets to H501 using (S=2001:db8:0:b010::31,
   D=2001:db8:0:5678::501).  When the uplink from SERb1 to ISP-B fails,
   SERb1 would stop originating its source-prefix-scoped route for the
   default destination (S=2001:db8:0:b000::/52, D=::/0) as well as its
   unscoped default destination route.  With these routes no longer in
   the IGP, traffic with (S=2001:db8:0:b010::31, D=2001:db8:0:5678::501)
   would end up at SERa based on the unscoped default destination route
   being originated by SERa.  Since that traffic has the wrong source
   address to be forwarded to ISP-A, SERa would drop it and send a
   Destination Unreachable message with Code 5 (Source address failed
   ingress/egress policy) back to H31.  H31 would then know to use
   another source address for that destination and would try with
   (S=2001:db8:0:a010::31, D=2001:db8:0:5678::501).  This would be
   forwarded to SERa based on the source-prefix-scoped default
   destination route still being originated by SERa, and SERa would
   forward it to ISP-A.  As discussed above, if we are willing to extend
   ICMPv6, SERa can even tell H31 what source address it should use to
   reach that destination.  The expected host behaviour has been
   discussed in Section 6.2.3.  Using ICMPv6 would have the same
   scalability/rate limiting issues discussed in Section 6.2.3.  ISP-B
   uplink failure immediately makes source addresses from
   2001:db8:0:b000::/52 unsuitable for external communication and might



Baker, et al.           Expires February 1, 2020               [Page 37]

Internet-Draft          Enterprise PA Multihoming              July 2019


   trigger a large number of ICMPv6 packets being sent to hosts in that
   subnet.

6.3.4.  Summary Of Methods For Controlling Default Source Address
        Selection On The Failure Of An Uplink

   It appears that DHCPv6 is not particularly well suited to quickly
   changing the source address used by a host in the event of the
   failure of an uplink, which eliminates DHCPv6 from the list of
   potential solutions.  On the other hand Router Advertisements
   provides a reliable mechanism to dynamically provide hosts with a
   list of valid prefixes to use as source addresses as well as prevent
   particular prefixes to be used.  While no additional new features are
   required to be implemented on hosts, routers need to be able to send
   RAs based on the state of scoped forwarding tables entries and to
   react to network topology changes by sending RAs with particular
   parameters set.

   The use of ICMPv6 Destination Unreachable messages generated by the
   SER (or any SADR-capable) routers seem like they have the potential
   to provide a support mechanism together with RAs to signal source
   address selection errors back to hosts, however scalability issues
   may arise in large networks in case of sudden topology change.
   Therefore it is highly desirable that hosts are able to select the
   correct source address in case of uplinks failure with ICMPv6 being
   an additional mechanism to signal unexpected failures back to hosts.

   The current behavior of different host operating system when
   receiving ICMPv6 Destination Unreachable message with code 5 (Source
   address failed ingress/egress policy) is not clear to the authors.
   Information from implementers, users, and testing would be quite
   helpful in evaluating this approach.

6.4.  Selecting Default Source Address Upon Failed Uplink Recovery

   The next logical step is to look at the scenario when a failed uplink
   on SERb1 to ISP-B is coming back up, so hosts can start using source
   addresses belonging to 2001:db8:0:b000::/52 again.

6.4.1.  Controlling Default Source Address Selection With DHCPv6

   The mechanism to use DHCPv6 to instruct the hosts (H31 in our
   example) to start using prefixes from ISP-B space (e.g.
   S=2001:db8:0:b010::31 for H31) to reach hosts on the Internet is
   quite similar to one discussed in Section 6.3.1 and shares the same
   drawbacks.





Baker, et al.           Expires February 1, 2020               [Page 38]

Internet-Draft          Enterprise PA Multihoming              July 2019


6.4.2.  Controlling Default Source Address Selection With Router
        Advertisements

   Let's look at the scenario discussed in Section 6.3.2.  If the
   uplink(s) failure caused the complete withdrawal of prefixes from
   2001:db8:0:b000::/52 address space by setting Preferred Lifetime
   value to 0, then the recovery of the link should just trigger new RA
   being sent with non-zero Preferred Lifetime.  In another scenario
   discussed in Section 6.3.2, the SERb1 uplink to ISP-B failure leads
   to disappearance of the (S=2001:db8:0:b000::/52, D=::/0) entry from
   the forwarding table scoped to S=2001:db8:0:b000::/52 and, in turn,
   caused R3 to send RAs from LLA_B with Router Lifetime set to 0.  The
   recovery of the SERb1 uplink to ISP-B leads to
   (S=2001:db8:0:b000::/52, D=::/0) scoped forwarding entry re-
   appearance and instructs R3 that it should advertise itself as a
   default router for ISP-B address space domain (send RAs from LLA_B
   with non-zero Router Lifetime).

6.4.3.  Controlling Default Source Address Selection With ICMP

   It looks like ICMPv6 provides a rather limited functionality to
   signal back to hosts that particular source addresses have become
   valid again.  Unless the changes in the uplink state a particular
   (S,D) pair, hosts can keep using the same source address even after
   an ISP uplink has come back up.  For example, after the uplink from
   SERb1 to ISP-B had failed, H31 received ICMPv6 Code 5 message (as
   described in Section 6.3.3) and allegedly started using
   (S=2001:db8:0:a010::31, D=2001:db8:0:5678::501) to reach H501.  Now
   when the SERb1 uplink comes back up, the packets with that (S,D) pair
   are still routed to SERa1 and sent to the Internet.  Therefore H31 is
   not informed that it should stop using 2001:db8:0:a010::31 and start
   using 2001:db8:0:b010::31 again.  Unless SERa has a policy configured
   to drop packets (S=2001:db8:0:a010::31, D=2001:db8:0:5678::501) and
   send ICMPv6 back if SERb1 uplink to ISP-B is up, H31 will be unaware
   of the network topology change and keep using S=2001:db8:0:a010::31
   for Internet destinations, including H51.

   One of the possible option may be using a scoped route with EXCLUSIVE
   flag as described in Section 6.2.3.  SERa1 uplink recovery would
   cause (S=2001:db8:0:a000::/52, D=2001:db8:0:1234::/64) route to
   reappear in the routing table.  In the absence of that route packets
   to H101 which were sent to ISP-B (as ISP-A uplink was down) with
   source addresses from 2001:db8:0:b000::/52.  When the route re-
   appears SERb1 would reject those packets and sends ICMPv6 back as
   discussed in Section 6.2.3.  Practically it might lead to scalability
   issues which have been already discussed in Section 6.2.3 and
   Section 6.4.3.




Baker, et al.           Expires February 1, 2020               [Page 39]

Internet-Draft          Enterprise PA Multihoming              July 2019


6.4.4.  Summary Of Methods For Controlling Default Source Address
        Selection Upon Failed Uplink Recovery

   Once again DHCPv6 does not look like reasonable choice to manipulate
   source address selection process on a host in the case of network
   topology changes.  Using Router Advertisement provides the flexible
   mechanism to dynamically react to network topology changes (if
   routers are able to use routing changes as a trigger for sending out
   RAs with specific parameters).  ICMPv6 could be considered as a
   supporting mechanism to signal incorrect source address back to hosts
   but should not be considered as the only mechanism to control the
   address selection in multihomed environments.

6.5.  Selecting Default Source Address When All Uplinks Failed

   One particular tricky case is a scenario when all uplinks have
   failed.  In that case there is no valid source address to be used for
   any external destinations while it might be desirable to have intra-
   site connectivity.

6.5.1.  Controlling Default Source Address Selection With DHCPv6

   From DHCPv6 perspective uplinks failure should be treated as two
   independent failures and processed as described in Section 6.3.1.  At
   this stage it is quite obvious that it would result in quite
   complicated policy table which needs to be explicitly configured by
   administrators and therefore seems to be impractical.

6.5.2.  Controlling Default Source Address Selection With Router
        Advertisements

   As discussed in Section 6.3.2 an uplink failure causes the scoped
   default entry to disappear from the scoped forwarding table and
   triggers RAs with zero Router Lifetime.  Complete disappearance of
   all scoped entries for a given source prefix would cause the prefix
   being withdrawn from hosts by setting Preferred Lifetime value to
   zero in PIO.  If all uplinks (SERa, SERb1 and SERb2) failed, hosts
   either lost their default routers and/or have no global IPv6
   addresses to use as a source.  (Note that 'uplink failure' might mean
   'IPv6 connectivity failure with IPv4 still being reachable', in which
   case hosts might fall back to IPv4 if there is IPv4 connectivity to
   destinations).  As a result, intra-site connectivity is broken.  One
   of the possible way to solve it is to use ULAs.

   All hosts have ULA addresses assigned in addition to GUAs and used
   for intra-site communication even if there is no GUA assigned to a
   host.  To avoid accidental leaking of packets with ULA sources SADR-
   capable routers SHOULD have a scoped forwarding table for ULA source



Baker, et al.           Expires February 1, 2020               [Page 40]

Internet-Draft          Enterprise PA Multihoming              July 2019


   for internal routes but MUST NOT have an entry for D=::/0 in that
   table.  In the absence of (S=ULA_Prefix; D=::/0) first-hop routers
   will send dedicated RAs from a unique link-local source LLA_ULA with
   PIO from ULA address space, RIO for the ULA prefix and Router
   Lifetime set to zero.  The behaviour is consistent with the situation
   when SERb1 lost the uplink to ISP-B (so there is no Internet
   connectivity from 2001:db8:0:b000::/52 sources) but those sources can
   be used to reach some specific destinations.  In the case of ULA
   there is no Internet connectivity from ULA sources but they can be
   used to reach another ULA destinations.  Note that ULA usage could be
   particularly useful if all ISPs assign prefixes via DHCP-PD.  In the
   absence of ULAs, upon the all uplinks failure hosts would lost all
   their GUAs upon prefix lifetime expiration which again makes intra-
   site communication impossible.

   It should be noted that the Rule 5.5 (prefer a prefix advertised by
   the selected next-hop) takes precedence over the Rule 6 (prefer
   matching label, which ensures that GUA source addresses are preferred
   over ULAs for GUA destinations).  Therefore if ULAs are used, the
   network administrator needs to ensure that while the site has an
   Internet connectivity, hosts do not select a router which advertises
   ULA prefixes as their default router.

6.5.3.  Controlling Default Source Address Selection With ICMPv6

   In case of all uplinks failure all SERs will drop outgoing IPv6
   traffic and respond with ICMPv6 error message.  In the large network
   when many hosts are trying to reach Internet destinations it means
   that SERs need to generate an ICMPv6 error to every packet they
   receive from hosts which presents the same scalability issues
   discussed in Section 6.3.3

6.5.4.  Summary Of Methods For Controlling Default Source Address
        Selection When All Uplinks Failed

   Again, combining SADR with Router Advertisements seems to be the most
   flexible and scalable way to control the source address selection on
   hosts.

6.6.  Summary Of Methods For Controlling Default Source Address
      Selection

   To summarize the scenarios and options discussed above:

   While DHCPv6 allows administrators to manipulate source address
   selection policy tables, this method has a number of significant
   disadvantages which eliminates DHCPv6 from a list of potential
   solutions:



Baker, et al.           Expires February 1, 2020               [Page 41]

Internet-Draft          Enterprise PA Multihoming              July 2019


   1.  It required hosts to support DHCPv6 and its extension (RFC7078);

   2.  DHCPv6 server needs to monitor network state and detect routing
       changes.

   3.  The use of policy tables requires manual configuration and might
       be extremely complicated, especially in the case of distributed
       network when large number of remote sites are being served by
       centralized DHCPv6 servers.

   4.  Network topology/routing policy changes could trigger
       simultaneous re-configuration of large number of hosts which
       present serious scalability issues.

   The use of Router Advertisements to influence the source address
   selection on hosts seem to be the most reliable, flexible and
   scalable solution.  It has the following benefits:

   1.  no new (non-standard) functionality needs to be implemented on
       hosts (except for [RFC4191] RIO support, which remains at the
       time of this writing not widely implemented);

   2.  no changes in RA format;

   3.  routers can react to routing table changes by sending RAs which
       would minimize the failover time in the case of network topology
       changes;

   4.  information required for source address selection is broadcast to
       all affected hosts in case of topology change event which
       improves the scalability of the solution (comparing to DHCPv6
       reconfiguration or ICMPv6 error messages).

   To fully benefit from the RA-based solution, first-hop routers need
   to implement SADR, belong to the SADR domain and be able to send
   dedicated RAs per scoped forwarding table as discussed above,
   reacting to network changes with sending new RAs.  It should be noted
   that the proposed solution would work even if first-hop routers are
   not SADR-capable but still able to send individual RAs for each ISP
   prefix and react to topology changes as discussed above (e.g. via
   configuration knobs).

   The RA-based solution relies heavily on hosts correctly implementing
   default address selection algorithm as defined in [RFC6724].  While
   the basic (and most common) multihoming scenario (two or more
   Internet uplinks, no 'walled gardens') would work for any host
   supporting the minimal implementation of [RFC6724], more complex use
   cases (such as "walled garden" and other scenarios when some ISP



Baker, et al.           Expires February 1, 2020               [Page 42]

Internet-Draft          Enterprise PA Multihoming              July 2019


   resources can only be reached from that ISP address space) require
   that hosts support Rule 5.5 of the default address selection
   algorithm.  There is some evidence that not all host OSes have that
   rule implemented currently.  However it should be noted that
   [RFC8028] states that Rule 5.5 should be implemented.

   ICMPv6 Code 5 error message SHOULD be used to complement RA-based
   solution to signal incorrect source address selection back to hosts,
   but it SHOULD NOT be considered as the stand-alone solution.  To
   prevent scenarios when hosts in multihomed envinronments incorrectly
   identify onlink/offlink destinations, hosts SHOULD treat ICMPv6
   Redirects as discussed in [RFC8028].

6.7.  Solution Limitations

6.7.1.  Connections Preservation

   The proposed solution is not designed to preserve connection state in
   case of an uplink failure.  When all uplinks to an ISP go down all
   transport connections established to/from that ISP address space will
   be interrupted (unless the transport protocol has specific
   multihoming support).  That behaviour is similar to the scenario of
   IPv4 multihoming with NAT when an uplink failure causes all
   connections to be NATed to completely different public IPv4
   addresses.  While it does sound suboptimal, it is determined by the
   nature of PA address space: if all uplinks to the particular ISP have
   failed, there is no path for the ingress traffic to reach the site
   and the egress traffic is supposed to be dropped by the BCP38
   [RFC2827] ingress filters.  The only potential way to overcome this
   limitation would be running BGP with all ISPs and advertise all site
   prefixes to all uplinks - a solution which shares all drawbacks of
   using PI address space without having its benefits.  Networks willing
   and capable of running BGP and using PI are out of scope of this
   document.

   It should be noted that in case of IPv4 NAT-based multihoming uplink
   recovery could cause connection interruptions as well (unless packet
   forwarding is integrated with existing NAT sessions tracking so the
   egress interface for the existing sessions is not changed).  However
   the proposed solution has a benefit of preserving the existing
   sessions during/after the failed uplink restoration.  Unlike the
   uplink failure event which causes all addresses from the affected
   prefix to be deprecated the recovery would just add new preferred
   addresses to a host without making any addresses unavailable.
   Therefore connections estavlished to/from those addresses do not have
   to be interrupted.





Baker, et al.           Expires February 1, 2020               [Page 43]

Internet-Draft          Enterprise PA Multihoming              July 2019


   While it's desirable for active connections to survive ISP failover
   events, for sites using PA address space such events affect the
   reachability of IP addresses assigned to hosts.  Unless the transport
   (or even higher level protocols) are capable of suviving the host
   renumbering, the active connections will be broken.  The proposed
   solution focuses on minimizing the impact of failover for new
   connections and for multipath-aware protocols.

   Another way to preserve connection state would be using multipath
   transport as discussed in Section 8.3.

6.8.  Other Configuration Parameters

6.8.1.  DNS Configuration

   In mutihomed envinronment each ISP might provide their own list of
   DNS servers.  For example, in the topology shown in Figure 3, ISP-A
   might provide recursive DNS server H51 2001:db8:0:5555::51, while
   ISP-B might provide H61 2001:db8:0:6666::61 as a recursive DNS
   server.  [RFC8106] defines IPv6 Router Advertisement options to allow
   IPv6 routers to advertise a list of DNS recursive server addresses
   and a DNS Search List to IPv6 hosts.  Using RDNSS together with
   'scoped' RAs as described above would allow a first-hop router (R3 in
   the Figure 3) to send DNS server addresses and search lists provided
   by each ISP (or the corporate DNS servers addresses if the enterprise
   is running its own DNS servers - as discussed below DNS split-horizon
   problem is to hard to solve without running a local DNS server).

   As discussed in Section 6.5.2, failure of all ISP uplinks would cause
   deprecation of all addresses assigned to a host from the address
   space of all ISPs.  If any intra-site IPv6 connectivity is still
   desirable (most likely to be the case for any mid/large scare
   network), then ULAs should be used as discussed in Section 6.5.2.  In
   such a scenario, the enterprise network should run its own recursive
   DNS server(s) and provide its ULA addresses to hosts via RDNSS in RAs
   send for ULA-scoped forwarding table as described in Section 6.5.2.

   There are some scenarios when the final outcome of the name
   resolution might be different depending on:

   o  which DNS server is used;

   o  which source address the client uses to send a DNS query to the
      server (DNS split horizon).

   There is no way currently to instruct a host to use a particular DNS
   server out of the configured servers list for resolving a particular
   name.  Therefore it does not seem feasible to solve the problem of



Baker, et al.           Expires February 1, 2020               [Page 44]

Internet-Draft          Enterprise PA Multihoming              July 2019


   DNS server selection on the host (it should be noted that this
   particular issue is protocol-agnostic and happens for IPv4 as well).
   In such a scenario it is recommended that the enterprise runs its own
   local recursive DNS server.

   To influence host source address selection for packets sent to a
   particular DNS server the following requirements must be met:

   o  the host supports RIO as defined in [RFC4191];

   o  the routers send RIO for routes to DNS server addresses.

   For example, if it is desirable that host H31 reaches the ISP-A DNS
   server H51 2001:db8:0:5555::51 using its source address
   2001:db8:0:a010::31, then both R1 and R2 should send the RIO
   containing the route to 2001:db8:0:5555::51 (or covering route) in
   their 'scoped' RAs, containing LLA_A as the default router address
   and the PO for SLAAC prefix 2001:db8:0:a010::/64.  In that case the
   host H31 (if it supports the Rule 5.5) would select LLA_A as a next-
   hop and then chose 2001:db8:0:a010::31 as the source address for
   packets to the DNS server.

   It should be noted that [RFC6106] explicitly prohibits using DNS
   information if the RA router Lifetime expired: "An RDNSS address or a
   DNSSL domain name MUST be used only as long as both the RA router
   Lifetime (advertised by a Router Advertisement message) and the
   corresponding option Lifetime have not expired.".  Therefore hosts
   might ignore RDNSS information provided in ULA-scoped RAs as those
   RAs would have router lifetime set to 0.  However the updated version
   of RFC6106 ([RFC8106]) has that requirement removed.

   As discussed above the DNS split-horizon problem and selecting the
   correct DNS server in a multihomed envinroment is not an easy one to
   solve.  The proper solution would require hosts to support the
   concept of multiple Provisioning Domains (PvD, a set of configuration
   information associated with a network, [RFC7556]).

7.  Deployment Considerations

   The solution described in this document requires certain mechanisms
   to be supported by the network infrastructure and hosts.  It requires
   some routers in the enterprise site to support some form of Source
   Address Dependent Routing (SADR).  It also requires hosts to be able
   to learn when the uplink to an ISP changes its state so the
   corresponding source addresses should (or should not) be used.
   Ongoing work to create mechanisms to accomplish this are discussed in
   this document, but they are still a work in progress.




Baker, et al.           Expires February 1, 2020               [Page 45]

Internet-Draft          Enterprise PA Multihoming              July 2019


7.1.  Deploying SADR Domain

   The proposed solution provides does not prescribe particular details
   regarding deploying an SADR domain within a multihomed enterprise
   network.  However the following guidelines could be applied:

   o  The SADR domain is usually limited by the multihomed site border.

   o  The minimal deployable scenario requires enabling SADR on all SERs
      and including them into a single SADR domain.

   o  As discussed in Section 4.2, extending the connected SADR domain
      beyond that point down to the first-hop routers can produce more
      efficient forwarding paths and allow the network to fully benefit
      from SADR. it would also simplify the operation of the SADR
      domain.

   o  During the incremental SADR domain expansion from the SERs down
      towards first-hop routers it's important to ensure that at any
      moment of time all SADR-capable routers within the domain are
      logically connected (see Section 5).

7.2.  Hosts-Related Considerations

   The solution discussed in this document relies on the default address
   selection algorithm ([RFC6724]) Rule 5.5.  While [RFC6724] considers
   this rule as optional, the recent [RFC8028] states that "A host
   SHOULD select default routers for each prefix it is assigned an
   address in".  It also recommends that hosts should implement Rule
   5.5. of [RFC6724].  Therefore while RFC8028-compliant hosts already
   have mechanism to learn about ISP uplinks state changes and selecting
   the source addresses accordingly, many hosts do not have such
   mechanism supported yet.

   It should be noted that multihomed enterprise network utilizing
   multiple ISP prefixes can be considered as a typical multiple
   provisioning domain (mPVD) scenario, as described in [RFC7556].  This
   document defines a way for the network to provide the PVD information
   to hosts indirectly, using the existing mechanisms.  At the same time
   [I-D.ietf-intarea-provisioning-domains] takes one step further and
   describes a comprehensive mechanism for hosts to discover the whole
   set of configuration information associated with different PVD/ISPs.
   [I-D.ietf-intarea-provisioning-domains] complements this document in
   terms of making hosts being able to learn about ISP uplink states and
   selecting the corresponding source addresses.






Baker, et al.           Expires February 1, 2020               [Page 46]

Internet-Draft          Enterprise PA Multihoming              July 2019


8.  Other Solutions

8.1.  Shim6

   The Shim6 working group specified the Shim6 protocol [RFC5533] which
   allows a host at a multihomed site to communicate with an external
   host and exchange information about possible source and destination
   address pairs that they can use to communicate.  It also specified
   the REAP protocol [RFC5534] to detect failures in the path between
   working address pairs and find new working address pairs.  A
   fundamental requirement for Shim6 is that both internal and external
   hosts need to support Shim6.  That is, both the host internal to the
   multihomed site and the host external to the multihomed site need to
   support Shim6 in order for there to be any benefit for the internal
   host to run Shim6.  The Shim6 protocol specification was published in
   2009, but it has not been widely implemented.  Therefore Shim6 is not
   considered as a viable solution for enterprise multihoming.

8.2.  IPv6-to-IPv6 Network Prefix Translation

   IPv6-to-IPv6 Network Prefix Translation (NPTv6) [RFC6296] is not the
   focus of this document.  NPTv6 suffers from the same fundamental
   issue as any other address translation approaches: it breaks end-to-
   end connectivity.  Therefore NPTv6 is not considered as desirable
   solution and this document intentionally focuses on solving
   enterprise multihoming problem without any form of address
   translations.

   With increasing interest and ongoing work in bringing path awareness
   to transport and application layer protocols hosts might be able to
   determine the properties of the various network paths and choose
   among paths available to them.  As selecting the correct source
   address is one of the possible mechanisms path-aware hosts may
   utilize, address translation negatively affects hosts path-awareness
   which makes NTPv6 even more undesirable solution.

8.3.  Multipath Transport

   Using multipath transport (such as MPTCP, [RFC6824] or multipath
   capabilities in QUIC) might solve the problems discussed in Section 6
   since it would allow hosts to use multiple source addresses for a
   single connection and switch between source addresses when a
   particular address becomes unavailable or a new address gets assigned
   to the host interface.  Therefore if all hosts in the enterprise
   network are only using multipath transport for all connections, the
   signaling solution described in Section 6 might not be needed (it
   should be noted that the Source Address Dependent Routing would still
   be required to deliver packets to the correct uplinks).  At the time



Baker, et al.           Expires February 1, 2020               [Page 47]

Internet-Draft          Enterprise PA Multihoming              July 2019


   this document was written, multipath transport alone could not be
   considered a solution for the problem of selecting the source address
   in a multihomed environment.  There are significant number of hosts
   which do not use multipath transport currently and it seems unlikely
   that the situation is going to change in any foreseeable future (even
   if new releases of operatin systems get multipath protocols support
   there will be a long tail of legacy hosts).  The solution for
   enterprise multihoming needs to work for the least common
   denominator: hosts without multipath transport support.  In addition,
   not all protocols are using multipath transport.  While multipath
   transport would complement the solution described in Section 6, it
   could not be considered as a sole solution to the problem of source
   address selection in multihomed environments.

   On the other hand PA-based multihoming could provide additional
   benefits for multipath protocol, should those protocols be deployed
   in the network.  Multipath protocols could leverage source address
   selection to achieve maximum path diversity (and potentially improved
   performance).

   Therefore deploying multipath protocols could not be considered as an
   alternative to the approach proposed in this document.  Instead both
   solutions complement each other so deploying multipath protocols in
   PA-based multihomed network proves mutually beneficial.

9.  IANA Considerations

   This memo asks the IANA for no new parameters.

10.  Security Considerations

   Section 6.2.3 discusses a mechanism for controlling source address
   selection on hosts using ICMPv6 messages.  Using ICMPv6 to influence
   source address selection allows an attacker to exhaust the list of
   candidate source addresses on the host by sending spoofed ICMPv6 Code
   5 for all prefixes known on the network (therefore preventing a
   victim from establishing a communication with the destination host).
   Another possible attack vector is using ICMPv6 Destination
   Unreachable Messages with Code 5 to steer the egress tra ffic towards
   the particular ISP (for example if the attacker has the ability of
   doing traffic sniffing or man-in-the-middle attack in that ISP
   network).

   To prevent those attacks hosts SHOULD verify that the original packet
   header included into ICMPv6 error message was actually sent by the
   host (to ensure that the ICMPv6 message was triggered by a packet
   sent by the host).




Baker, et al.           Expires February 1, 2020               [Page 48]

Internet-Draft          Enterprise PA Multihoming              July 2019


   As ICMPv6 Destination Unreachable Messages with Code 5 could be
   originated by any SADR-capable router within the domain (or even come
   from the Internet), GTSM ([RFC5082]) can not be applied.  Filtering
   such ICMOv6 messages at the site border can not be recommended as it
   would break the legitimate end2end error signalling mechanism ICMPv6
   is designed for.

   The security considerations of using stateless address
   autoconfiguration are discussed in [RFC4862].

11.  Acknowledgements

   The original outline was suggested by Ole Troan.

   The authors would like to thank the following people (in alphabetical
   order) for their review and feedback: Olivier Bonaventure, Deborah
   Brungard, Brian E Carpenter, Lorenzo Colitti, Roman Danyliw, Benjamin
   Kaduk, Suresh Krishnan, Mirja Kuhlewind, David Lamparter, Nicolai
   Leymann, Acee Lindem, Philip Matthewsu, Robert Raszuk, Alvaro Retana,
   Pete Resnick, Dave Thaler, Michael Tuxen, Martin Vigoureux, Eric
   Vyncke, Magnus Westerlund.

12.  References

12.1.  Normative References

   [RFC1918]  Rekhter, Y., Moskowitz, B., Karrenberg, D., de Groot, G.,
              and E. Lear, "Address Allocation for Private Internets",
              BCP 5, RFC 1918, DOI 10.17487/RFC1918, February 1996,
              <https://www.rfc-editor.org/info/rfc1918>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC2827]  Ferguson, P. and D. Senie, "Network Ingress Filtering:
              Defeating Denial of Service Attacks which employ IP Source
              Address Spoofing", BCP 38, RFC 2827, DOI 10.17487/RFC2827,
              May 2000, <https://www.rfc-editor.org/info/rfc2827>.

   [RFC4191]  Draves, R. and D. Thaler, "Default Router Preferences and
              More-Specific Routes", RFC 4191, DOI 10.17487/RFC4191,
              November 2005, <https://www.rfc-editor.org/info/rfc4191>.

   [RFC4193]  Hinden, R. and B. Haberman, "Unique Local IPv6 Unicast
              Addresses", RFC 4193, DOI 10.17487/RFC4193, October 2005,
              <https://www.rfc-editor.org/info/rfc4193>.



Baker, et al.           Expires February 1, 2020               [Page 49]

Internet-Draft          Enterprise PA Multihoming              July 2019


   [RFC4291]  Hinden, R. and S. Deering, "IP Version 6 Addressing
              Architecture", RFC 4291, DOI 10.17487/RFC4291, February
              2006, <https://www.rfc-editor.org/info/rfc4291>.

   [RFC4443]  Conta, A., Deering, S., and M. Gupta, Ed., "Internet
              Control Message Protocol (ICMPv6) for the Internet
              Protocol Version 6 (IPv6) Specification", STD 89,
              RFC 4443, DOI 10.17487/RFC4443, March 2006,
              <https://www.rfc-editor.org/info/rfc4443>.

   [RFC4861]  Narten, T., Nordmark, E., Simpson, W., and H. Soliman,
              "Neighbor Discovery for IP version 6 (IPv6)", RFC 4861,
              DOI 10.17487/RFC4861, September 2007,
              <https://www.rfc-editor.org/info/rfc4861>.

   [RFC4862]  Thomson, S., Narten, T., and T. Jinmei, "IPv6 Stateless
              Address Autoconfiguration", RFC 4862,
              DOI 10.17487/RFC4862, September 2007,
              <https://www.rfc-editor.org/info/rfc4862>.

   [RFC6106]  Jeong, J., Park, S., Beloeil, L., and S. Madanapalli,
              "IPv6 Router Advertisement Options for DNS Configuration",
              RFC 6106, DOI 10.17487/RFC6106, November 2010,
              <https://www.rfc-editor.org/info/rfc6106>.

   [RFC6296]  Wasserman, M. and F. Baker, "IPv6-to-IPv6 Network Prefix
              Translation", RFC 6296, DOI 10.17487/RFC6296, June 2011,
              <https://www.rfc-editor.org/info/rfc6296>.

   [RFC6724]  Thaler, D., Ed., Draves, R., Matsumoto, A., and T. Chown,
              "Default Address Selection for Internet Protocol Version 6
              (IPv6)", RFC 6724, DOI 10.17487/RFC6724, September 2012,
              <https://www.rfc-editor.org/info/rfc6724>.

   [RFC7078]  Matsumoto, A., Fujisaki, T., and T. Chown, "Distributing
              Address Selection Policy Using DHCPv6", RFC 7078,
              DOI 10.17487/RFC7078, January 2014,
              <https://www.rfc-editor.org/info/rfc7078>.

   [RFC7556]  Anipko, D., Ed., "Multiple Provisioning Domain
              Architecture", RFC 7556, DOI 10.17487/RFC7556, June 2015,
              <https://www.rfc-editor.org/info/rfc7556>.

   [RFC8028]  Baker, F. and B. Carpenter, "First-Hop Router Selection by
              Hosts in a Multi-Prefix Network", RFC 8028,
              DOI 10.17487/RFC8028, November 2016,
              <https://www.rfc-editor.org/info/rfc8028>.




Baker, et al.           Expires February 1, 2020               [Page 50]

Internet-Draft          Enterprise PA Multihoming              July 2019


   [RFC8106]  Jeong, J., Park, S., Beloeil, L., and S. Madanapalli,
              "IPv6 Router Advertisement Options for DNS Configuration",
              RFC 8106, DOI 10.17487/RFC8106, March 2017,
              <https://www.rfc-editor.org/info/rfc8106>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

   [RFC8415]  Mrugalski, T., Siodelski, M., Volz, B., Yourtchenko, A.,
              Richardson, M., Jiang, S., Lemon, T., and T. Winters,
              "Dynamic Host Configuration Protocol for IPv6 (DHCPv6)",
              RFC 8415, DOI 10.17487/RFC8415, November 2018,
              <https://www.rfc-editor.org/info/rfc8415>.

12.2.  Informative References

   [I-D.ietf-intarea-provisioning-domains]
              Pfister, P., Vyncke, E., Pauly, T., Schinazi, D., and W.
              Shao, "Discovering Provisioning Domain Names and Data",
              draft-ietf-intarea-provisioning-domains-05 (work in
              progress), June 2019.

   [I-D.ietf-rtgwg-dst-src-routing]
              Lamparter, D. and A. Smirnov, "Destination/Source
              Routing", draft-ietf-rtgwg-dst-src-routing-07 (work in
              progress), March 2019.

   [I-D.pfister-6man-sadr-ra]
              Pfister, P., "Source Address Dependent Route Information
              Option for Router Advertisements", draft-pfister-6man-
              sadr-ra-01 (work in progress), June 2015.

   [RFC3704]  Baker, F. and P. Savola, "Ingress Filtering for Multihomed
              Networks", BCP 84, RFC 3704, DOI 10.17487/RFC3704, March
              2004, <https://www.rfc-editor.org/info/rfc3704>.

   [RFC4941]  Narten, T., Draves, R., and S. Krishnan, "Privacy
              Extensions for Stateless Address Autoconfiguration in
              IPv6", RFC 4941, DOI 10.17487/RFC4941, September 2007,
              <https://www.rfc-editor.org/info/rfc4941>.

   [RFC5082]  Gill, V., Heasley, J., Meyer, D., Savola, P., Ed., and C.
              Pignataro, "The Generalized TTL Security Mechanism
              (GTSM)", RFC 5082, DOI 10.17487/RFC5082, October 2007,
              <https://www.rfc-editor.org/info/rfc5082>.





Baker, et al.           Expires February 1, 2020               [Page 51]

Internet-Draft          Enterprise PA Multihoming              July 2019


   [RFC5533]  Nordmark, E. and M. Bagnulo, "Shim6: Level 3 Multihoming
              Shim Protocol for IPv6", RFC 5533, DOI 10.17487/RFC5533,
              June 2009, <https://www.rfc-editor.org/info/rfc5533>.

   [RFC5534]  Arkko, J. and I. van Beijnum, "Failure Detection and
              Locator Pair Exploration Protocol for IPv6 Multihoming",
              RFC 5534, DOI 10.17487/RFC5534, June 2009,
              <https://www.rfc-editor.org/info/rfc5534>.

   [RFC6434]  Jankiewicz, E., Loughney, J., and T. Narten, "IPv6 Node
              Requirements", RFC 6434, DOI 10.17487/RFC6434, December
              2011, <https://www.rfc-editor.org/info/rfc6434>.

   [RFC6824]  Ford, A., Raiciu, C., Handley, M., and O. Bonaventure,
              "TCP Extensions for Multipath Operation with Multiple
              Addresses", RFC 6824, DOI 10.17487/RFC6824, January 2013,
              <https://www.rfc-editor.org/info/rfc6824>.

   [RFC7676]  Pignataro, C., Bonica, R., and S. Krishnan, "IPv6 Support
              for Generic Routing Encapsulation (GRE)", RFC 7676,
              DOI 10.17487/RFC7676, October 2015,
              <https://www.rfc-editor.org/info/rfc7676>.

Authors' Addresses

   Fred Baker
   Santa Barbara, California  93117
   USA

   Email: FredBaker.IETF@gmail.com


   Chris Bowers
   Juniper Networks
   Sunnyvale, California  94089
   USA

   Email: cbowers@juniper.net


   Jen Linkova
   Google
   1 Darling Island Rd
   Pyrmont, NSW  2009
   AU

   Email: furry@google.com




Baker, et al.           Expires February 1, 2020               [Page 52]
