BGP UPDATE for SD-WAN Edge Discovery

3.1. The Objectives of SD-WAN Edge Discovery

The objectives of SD-WAN edge discovery for an SD-WAN edge node are to discover its authorized BGP peers and each peer's associated properities in order to establish secure overlay tunnels [Net2Cloud]. The attributes to be propagated include:¶

• the SD-WAN (client) VPNs information,¶

• the attached routes under the SD-WAN VPNs, and¶

• the properties of the underlay networks over which the client routes can be carried.¶

Some SD-WAN peers are connected by both trusted VPNs and untrusted public networks. Some SD-WAN peers are connected only by untrusted public networks. For the traffic over untrusted networks, IPsec Security Associations (IPsec SA) must be established and maintained. For the trusted VPNs, IPsec Security associations may not be set-up. If an edge node has network ports behind a NAT, the NAT properties need to be discovered by the authorized SD-WAN peers.¶

Like any VPN networks, the attached client routes belonging to specific SD-WAN VPNs can only be exchanged with the SD-WAN peer nodes authorized to communicate.¶

3.2. Comparing with Pure IPsec VPN

A pure IPsec VPN has IPsec tunnels connecting all edge nodes over public networks. Therefore, it requires stringent authentication and authorization (i.e., IKE Phase 1) before other properties of IPsec SA can be exchanged. The IPsec Security Association (SA) between two untrusted nodes typically requires the following configurations and message exchanges:¶

• IPsec IKEV2:

  messages sent to authenticate with each other.¶

  Establish IPsec SA

  : requires the following set-up¶

  • Local key configuration,¶

  • Setting the Remote Peer address,¶

  • Information from IKEv2 Proposal directly sent to peer (Encryption method, Integrity sha512, etc.), and¶

  • Transform set.¶

  Attached client prefixes discovery acchieved by:

  • Running routing protocol within each IPsec SA.¶

  • If multiple IPsec SAs between two peer nodes are established to achieve load sharing, each IPsec tunnel needs to run its own routing protocol to exchange client routes attached to the edges.¶

  Set-up Access List or Traffic Selector:

  such as Permit Local-IP1, Remote-IP2, and other parameters.¶

In a BGP-controlled SD-WAN network over hybrid MPLS VPN and public internet underlay networks, all edge nodes and RRs are already connected by private secure paths. The RRs have the policies to manage the authentication of all peer nodes. More importantly, when an edge node needs to establish multiple IPsec tunnels to many edge nodes, all the management information can be multiplexed into the secure management tunnel between RR and the edge node operating as a BGP peer. Therefore, the amount of authentication in a BGP-Controlled SD-WAN network can be significantly reduced.¶

Client VPNs are configured via VRFs, just like the configuration of the existing MPLS VPN. The IPsec equivalent traffic selectors for local and remote routes are achieved by importing/exporting VPN Route Targets. The binding of client routes to IPsec SA is dictated by policies. As a result, the IPsec configuration for a BGP controlled SD-WAN (with mixed MPLS VPN) can be simplified in the following manner:¶

• The SD-WAN controller has the authority to authenticate edges and peers so the Remote Peer association is controlled by the SD-WAN Controller (RR).¶

• The IKEv2 proposals (including the IPsec Transform set) can be sent directly to peers, or incorporated in a BGP UPDATE.¶

• The BGP UPDATE announces the client route reachability through the SDWAN hybrid tunnels. A SDWAN hybrid tunnel combines several other tunnels into a single logical tunnel. The SD-WAN Hybrid tunnel implementations insure that all tunnels within are either running over secure network links or secured by IPsec.¶

• Importing/exporting Route Targets under each client VPN (VRF) achieves the traffic selection (or permission) among clients' routes attached to multiple edge nodes.¶

Note: The web of SDWAN hybrid tunnels in a network is denoted in this document as an SD-WAN underlay. BGP passes information about the SDWAN hybrid tunnels between BGP peers by passing an SD-WAN Underlay NLRI paired with the tunnel encapsulation attribute (TEA) with an SDWAN tunnel type SD-WAN-Hybrid TLV.¶

Also, note that with this method there is no need to run multiple routing protocols in each IPsec tunnel.¶

3.3. Client Routes and SDWAN UPDATE

There are two different sequences of BGP UPDATE messages are used for SD-WAN Edge Discovery. The first associates Client routes BGP Updates with a SD-WAN Hybrid Tunnel. The second passes information regarding the SD-WAN web of Hybrid tunnels (denoted as the SD-WAN Underlay) between Egress routers and Ingress routers to aid set-up of SD-WAN Hybrid tunnels in a network./¶

# Update #1 for Port B2 on C-PE2
  SD-WAN NLRI (AFI=1/SAFI=SD-WAN)
    Port-id = Local port ID for WAN Port 192.0.0.1
    SD-WAN Color = Color indicates Group of underlay tunnels
    SD-WAN Node ID = 2.2.2.2 (C-PE2)
  Tunnel Encaps Attribute (TEA)
    Tunnel TLV: (type: Hybrid-SD-WAN)
      Tunnel Egress Endpoint SubTLV: 2.2.2.2
      Extended-Port Attribute SubTLV: Port B2 (192.0.0.1)
      IPsec SA Identifier SubTLV:  SA-1, SA-2

# Update #2 for Port B3 (IPsec) in Hybrid tunnel on PE2
  SD-WAN NLRI (AFI=1/SAFI=SD-WAN)
    Port-id = Local port ID for WAN Port 170.0.0.1
    SD-WAN Color = Color indicates Group of Underlay tunnels
    SD-WAN Node ID = 2.2.2.2 (C-PE2)
  Tunnel Encaps Attribute (TEA)
    Tunnel TLV: (type: SD-Wan-Hybrid)
      Tunnel Egress Endpoint SubTLV (SubTLV=6) = 2.2.2.2
          Extended Port Attribute SubTLV: Port B3 (170.0.0.1)
          Simplified IPSec SD-WAN SubTLV: SA-3, SA-4

See section 6.1 for SD-WAN NLRI definition.
See section 7.1 for Extended Port Attribute SubTLV definition.
See section 8.1 for IPsec SA Identifier SubTLV.
See section 8.5 for Simplified IPsec SD-WAN SubTlv.

# Ports B1 (MPLS), B2 (IPsec), B3 (IPsec)
# Update for C-PE2 (IPSec)
  SD-WAN NLRI (AFI=1 / SAFI = SD-WAN)
    SD-WAN Color = Match to Color-EC of Client routes
        Port ID = 0
        SD-WAN Node ID = 2.2.2.2
  Tunnel Encaps Attribute (TEA)
        Tunnel TLV (type: SD-Wan-Hybrid)
      Egress Endpoint = 2.2.2.2
      IPsec-SA-ID: SA-1, SA-2, SA-3, SA-4

3.4. Edge Node Discovery

The basic scheme of SD-WAN edge node discovery using BGP consists of the following:¶

• Secure connection to a SD-WAN controller (BGP RR in this context):¶

  • For an SD-WAN edge with both MPLS and IPsec paths, the edge node should already have a secure connection to its controller (RR in this context). For an SD-WAN edge that is only accessible via Internet, the SD-WAN edge upon power-up establishes a secure tunnel (such as TLS or SSL) with the SD-WAN central controller whose address is preconfigured on the edge node. The central controller informs the edge node of its local RR. The edge node then establishes a transport layer secure session with the RR (such as TLS or SSL).¶

• The BGP Peer Edge node will advertise the properties of its Hybrid SD-WAN Tunnel to its designated RR via the secure connection.¶

• The RR propagates the received information to the authorized BGP peers.¶

• The authorized BGP peers can establish the secure data channels via Hybrid SD-WAN tunnel and exchange more information among each other.¶

For an SD-WAN deployment with multiple RRs, it is assumed that there are secure connections among those RRs. How secure connections are established among those RRs is out of the scope of this document. The existing BGP UPDATE propagation mechanisms control the edge properties propagation among the RRs.¶

For some environments where the communication to RR is highly secured, [RFC9016] IKE-less can be deployed to simplify IPsec SA establishment among edge nodes.¶

4.1. SD-WAN Segmentation, Virtual Topology and Client VPN

In SD-WAN deployment, SD-WAN Segmentation is a frequently used term which refers to partitioning a network into multiple subnetworks, just like MPLS VPNs. SD-WAN Segmentation is achieved by creating SD-WAN virtual topologies and SD-WAN VPNs. An SD-WAN virtual topology consists of a set of edge nodes and the tunnels (a.k.a. underlay paths) interconnecting those edge nodes. These tunnels forming the underlay paths can be IPsec tunnels, or MPLS VPN tunnels, or other tunnels.¶

An SD-WAN VPN is configured in the same way as the VRFs of an MPLS VPN. One SD-WAN client VPN can be mapped to multiple SD-WAN virtual topologies. SD-WAN Controller governs the policies of mapping a client VPN to SD-WAN virtual topologies.¶

Each SD-WAN edge node may need to support multiple VPNs. Route Target is used to differentiate the SD-WAN VPNs. For example, in the picture below, the Payment-Flow on C-PE2 is only mapped to the virtual topology of C-PEs to/from Payment Gateway, whereas other flows can be mapped to a multipoint-to-multipoint virtual topology.¶

                                  +---+
                   +--------------|RR |----------+
                  /  Untrusted    +-+-+           \
                 /                                 \
                /                                   \
        +---------+  MPLS Path                      +-------+
11.1.1.x| C-PE1   A1-------------------------------B1 C-PE2 |10.1.1.x
        |         |                                 |       |
21.1.1.x|         A2(192.10.0.10)------( 192.0.0.1)B2       |20.1.1.x
        |         |                                 |       |
        | Addr    A3(160.0.0.1) --------(170.0.0.1)B3 Addr  |11.2.2.x
        | 1.1.1.1 |                              /  |2.2.2.2|
        +---------+                             /   +-------+
                   \                           /
                    \                         /PaymentFlow
                     \                       /
                      \                +----+----+
                       +---------------| payment |
                                       | Gateway |
                                       +---------+

4.2. Constrained Propagation of Edge Capability

BGP Route Reflectors [RFC4456] may be configured to constrain the distribution of BGP informtion to specific BGP clients.¶

       NLRI={SD-WAN#1, SD-WAN#2}       NLRI={SD-WAN#1, SD-WAN#3}
            ----->                 +---+      <-----------
              +--------------------|RR1|------------------+
              | Outbound Filter    +---+  Outbound Filter |
              | Permit SD-WAN#1,#2      Permit SD-WAN#1,#3|
              |   <-------                --------->      |
              |                                           |
        +-----+---+  MPLS Path                      +-----+-+
11.1.1.x| C-PE1   A1-------------------------------B1 C-PE2 |10.1.1.x
        |         |                                 |       |
21.1.1.x|         A2(192.10.0.10)------( 192.0.0.1)B2       |20.1.1.x
        |         |                                 |       |
        | Addr    A3(160.0.0.1) --------(170.0.0.1)B3 Addr  |
        | 1.1.1.1 |                                 |2.2.2.2|
        +---------+                                 +-------+
SD-WAN VPN #1                                          SD-WAN VPN #1
SD-WAN VPN #2                                          SD-WAN VPN #3

The RR is configured to speak to the BGP clients (CE-PE1 and CE-PE2) over secure virtual links (IPsec), and send only certain routes. The configuration on the RR and the BGP Peers sending SD-WAN routes forms a "walled garden" for the SD-WAN information.¶

It is out of the scope of this document on how RR is configured with the policies to filter out unauthorized nodes for specific SD-WAN VPNs.¶

5.1. Tunnel Form 1: Encapsulation Extended Community (Encaps-EC)

The SD-WAN Client Route UPDATE message uses the Encapsulation Extended Community (Encap-EC) to identify the Hybrid SD-WAN tunnel and the Tunnel Egress Endpoint. Per [RFC9012], the Encapsulation Extended Community uses the NextHop Field in the BGP UPDATE as the Tunnel Egress EndPoint. The validation for the Tunnel Egress Endpoint uses the validation in section 6, 8, and 13 applied to the NextHop.¶

A Color Extended Community (Color-EC) or local policy applied to the client route directs the traffic for the client route to across appropriate interface within the Hybrid SD-WAN Tunnel to the Tunnel Egress Endpoint.¶

5.2. Tunnel Form 2: Tunnel Encapsulation Path Attribute (TEA)

The Client route with the Tunnel Encapsulation Path Attribute (TEA) with the Hybrid SD-WAN route TLV must have the Tunnel Egress Endpoint (SubTLV=6) and any of the SubTLVs found in [RFC9012]. The validation for the Tunnel Egress Endpoint uses the validation in section 6, 8, and 13 from [RFC9012].¶

5.3. Multiple tunnels attached to One Route

A single SD-WAN client route may be attached to multiple SD-WAN Hybrid tunnels. An Update with an SD-WAN client route may express these tunnels as an Encap-EC or a TEA. Each of these tunnel descriptions is treated as a unique Hybrid SD-WAN tunnel with a unique Egress Endpoint. Local Policy on the BGP Peer determines which tunnel the client data traffic will use.¶

5.4. SD-WAN VPN ID in Client Route Update

An SD-WAN VPN ID is same as a client VPN in a BGP controlled SD-WAN network. The Route Target Extended Community should be included in a Client Route UPDATE message to differentiate the client routes from routes belonging to other VPNs. Route Target value is taken as the VPN ID (for 1/1 and 2/1). For 1/128 and 2/128, the RD from the NLRI identifies the VPN ID. For EVPN, picking up the VPN-ID from EVPN SAFI.¶

5.5. SD-WAN VPN ID in Data Plane

SD-WAN edge node which can be reached by either an MPLS path or an IPsec path within the hybrid SD-WAN tunnel. If client packets are sent via a secure MPLS network within the Hybrid SD-WAN tunnel, then the data packets will have MPLS headers with the MPLS Labels based on the scheme specified by [RFC8277]. It is assumed the secure MPLS network assures the security outer MPLS Label header.¶

If the packets are sent via a link with IPsec outer encryption across a public network, the payload is still encrypted with GRE or VXLAN encryption. For GRE Encapsulation within an IPsec tunnel, the GRE key field can be used to carry the SD-WAN VPN ID. For network virtual overlay (VxLAN, GENEVE, etc.) encapsulation within the IPsec tunnel, the Virtual Network Identifier (VNI) field is used to carry the SD-WAN VPN ID.¶

6.1. NLRI for SD-WAN Underlay Tunnel Update

A new NLRI SAFI (SD-WAN SAFI=74) is introduced within the MP_REACH_NLRI Path Attribute of [RFC4760] for advertising the detailed properties of the SD-WAN tunnels terminated at the edge node:¶

+------------------+
|    Route Type    | 2 octets
+------------------+
|     Length       | 2 Octets
+------------------+
|   Type Specific  |
~ Value (Variable) ~
|                  |
+------------------+

Where¶

- Route (NLRI) Type (16 bits):

specifies the encoding for the remaining portion of the SD-WAN NLRI.¶

- Length (16 bits):

specifies the length of the NLRI Value field in bits, as defined in [RFC4760].¶

This document defines the following SD-WAN Route type:¶

- NLRI Route-Type = 1:

For advertising the detailed properties of the SD-WAN tunnels terminated at the edge, where the transport network port can be uniquely identified by a tuple of three values (Port-Local-ID, SD-WAN-Color, SD-WAN-Node-ID). The SD-WAN NLRI Route-Type =1 has the following encoding:¶

     +------------------+
     |  Route Type = 1  | 2 octets
     +------------------+
     |     Length       | 2 Octets
     +------------------+
     |   Port-Local-ID  | 4 octets
     +------------------+
     |   SD-WAN-Color   | 4 octets
     +------------------+
     |  SD-WAN-Node-ID  | 4 or 16 octets
     +------------------+

- Port-local-ID:

SD-WAN edge node Port identifier, which is locally significant. If the SD-WAN NLRI applies to multiple WAN ports, this field is zero.¶

- SD-WAN-Color:

represents a group of tunnels corresponding to the Color-Extended-community included in the client route's UPDATE messages. When a client route is reachable through multiple SD-WAN edges co-located at a single site, the SD-WAN-Color can represent a group of tunnels terminated at all those edges. If an SD-WAN-Color represents all tunnels at a site, it effectively represents the entire site.¶

- SD-WAN Node ID:

The node's IPv4 or IPv6 address.¶

Route Type values outside of 1 are out of scope for this document.¶

6.2. SD-WAN-Hybrid Tunnel Encoding

A new BGP Tunnel-Type SD-WAN-Hybrid (code point 25) indicates hybrid underlay tunnels.¶

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Tunnel-Type=25(SD-WAN-Hybrid )| Length (2 Octets)             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           Sub-TLVs                            |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The valid SubTLVs for the Hybrid Tunnel type include subTLVs specified in [RFC9012] and the following SubTLVs specified in this document:¶

• Extended Port Attributes Sub-TLV¶

• IPsec SA-ID Sub-TLV¶

• IPSec SA Rekey Sub-TLV¶

• IPsec Public Key Sub-TLV¶

• IPsec SA Proposal Sub-TLV¶

• Simplified IPsec SA Sub-TLV¶

The Extended Port Attributes are described in section 7, and the IPsec related SubTLVs are described in section 8.¶

7.1. Extended Sub-Sub-TLV

One Extended Sub-Sub-TLVs is specified in this document: Underlay Network Property Sub-Sub-TLV¶

     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | UnderlayType  |      Length   |      Reserved                 |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |Connection Type|   Port Type   |        Port Speed             |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

8.1. IPsec-SA-ID Sub-TLV

IPsec-SA-ID Sub-TLV within the Hybrid Underlay Tunnel UPDATE indicates one or more pre-established IPsec SAs by using their identifiers, instead of listing all the detailed attributes of the IPsec SAs.¶

Using an IPsec-SA-ID Sub-TLV not only greatly reduces the size of BGP UPDATE messages, but also allows the pairwise IPsec rekeying process to be performed independently.¶

The following is the structure of the IPsec-SA-ID sub-TLV¶

 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|IPSec-SA-ID Sub|   Length  |  Reserved                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      IPsec SA Identifier #1                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      IPsec SA Identifier #2                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
~                                                               ~
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      IPsec SA Identifier #n                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

where:¶

• - IPSec-SA-ID Sub-Type (8 bits): Assigned the value of 64 by this document.¶

• - length (8 bits): Specifies the total length in octets of the value field (excluding the Type and Length fields). For the IPSec-SA-ID Sub-Type, the length should be the 2 + 4 *(number of IPsec SA IDs) .¶

• - Reserved: Reserved for future use. In this version of the document, the Reserved field MUST be set to zero and MUST be ignored upon receipt. Received values MUST be propagated without change.¶

• - The value field consists of a sequence of IPsec SA Identifiers, each 4 octets long. As shown in figure above, n IPsec SAs are attached in the IPsec-SA-ID Sub-TLV:¶

  • IPSec SA Identifier #1: A 4 octet identifier for a pre-established IP security association.¶

  • IPSec SA Identifier #2: A 4 octet identifier for a pre-established IP security association.¶

  • IPSec SA Identifier #n: A 4 octet identifier for a pre-established IP security association.¶

8.2. IPsec SA Rekey Counter Sub-TLV

The IPsec SA Rekey Counter Sub-TLV provides the rekey counter for a security association (identified by IPsec SA Identifier).¶

The IPsec SA Rekey Counter Sub-TLV Type is assigned value of 66 by this document.¶

The Value field of this Sub-TLV is as follows:¶

        0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |   ID Length   |       Nonce Length            |I| reserved    |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                             Rekey                             |
       |                            Counter                            |
       +---------------------------------------------------------------+
       |                IPsec SA Identifier                            |
       +---------------------------------------------------------------+
       |                                                               |
       ~                          Nonce Data                           ~
       |                                                               |
       +---------------------------------------------------------------+

where:¶

• - ID Length (8 bits): indicates the length in octets of SA-Identifer. This length should be 4 octets.¶

• - Nonce length (16 bits): indicates the length in octets of the Nonce Data.¶

• - I Flag: when set to 1, the I-flag indicates that the communication being established is new. when set to 0, it signals that the communication is a continuation of an existing session.¶

• - Reserved (7 bits): Reserved for future use. In this version of the document, the Reserved field MUST be set to zero and MUST be ignored upon receipt. Received values MUST be propagated without change.¶

• - Rekey Counter (64 bits): the number of key updates or rekeys that have occurred. Each time a key is rotated or replaced, the ReKey Counter is incremented.¶

• - IPsec SA Identifier (IPSec-SA-ID): a specific IPsec SAs terminated at the edge.¶

• - Nonce Data: a random or pseudo-random number for preventing replay attacks. Its length is a multiple of 32 bits[RFC7296].¶

Note:¶

• The IPsec-SA-ID may also refer to the values carried in the same TEA in the same Tunnel TLV (type SD-WAN Hybrid) as the IPsec SA Rekey SubTLV in either the IPsec Public Key SubTLV or the IPsec SA Proposal SubTLV. The IPsec SA Rekey Counter, IPsec Public Key, and IPsec SA Proposal SubTLVs work together to create security associations.¶

• The IPsec-SA-ID may refer to information in another Tunnel TLV in the same TEA associated wiht the same BGP UPDATE message as the IPsec SA Rekey Counter sub-TLV.¶

• The IPsec-SA-ID can be used in the IPsec-SA-ID subTLV of a different BGP UPDATE message.¶

8.3. IPsec Public Key Sub-TLV

The IPsec Public Key Sub-TLV Type is assigned value of 67 by this document.¶

The Value field of this Sub-TLV is as follows:¶

        0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |   Diffie-Hellman Group Num    |          Reserved             |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                                                               |
       ~                       Key Exchange Data                       ~
       |                                                               |
       +---------------------------------------------------------------+
       |                            Duration                           |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

where:¶

• Diffie-Hellman Group Num (16-bits): identifies the Diffie-Hellman group used to compute the Key Exchange Data. Details on Diffie-Hellman group numbers can be found in Appendix B of IKEv2 [RFC7296] and [RFC5114].¶

• The Key Exchange data: This refers to a copy of the sender's Diffie-Hellman public value. The length of the Diffie-Hellman public value is defined for MODP groups in [RFC 7296] and for ECP groups in [RFC 5903].¶

• Duration (32 bits): a 4-octet value specifying the life span of the Diffie-Hellman key in seconds.¶

8.4. IPsec SA Proposal Sub-TLV

The IPsec SA Proposal Sub-TLV is to indicate the number of Transform Sub-TLVs.¶

The IIPsec SA Proposal Sub-TLV Type is assigned value of 68 by this document.¶

The Value field of this Sub-TLV is as follows:¶

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|    Transform Attr Length      |Transform Type |    Reserved.  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|     Transform ID              |          Reserved             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
~                        Transform Attributes                   ~
|                                                               |
+---------------------------------------------------------------+

where:¶

• Transform Attr Length (16 bits): indicates the length of the Transform Attributes field.¶

• Transform Type (8 bits): The Transform Type values are defined in Section 3.3.2 of [RFC7296] and [IKEV2IANA]. Only ENCR, INTEG, and ESN values are permitted.¶

• Reserved (8 bits): reserved for future use. These bits are ignored upon receipt and set to zero when transmitted.¶

• Transform ID (16 bits): An identifer for the transform defined by the associated transform attributes.¶

• Transform Attributes: These Sub-SubTLV are defined in section 3.3.5 of [RFC7296].¶

8.5. Simplified IPsec SA sub-TLV

For a simple SD-WAN network with edge nodes supporting only a few pre-defined encryption algorithms, a simple IPsec sub-TLV can be used to encode the pre-defined algorithms, as below:¶

     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |IPsec-simType  |IPsecSA Length |             Reserved          |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | Transform     | Mode          | AH algorithms |ESP algorithms |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |         ReKey Counter (SPI)                                   |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | key1 length   |         Key 1                                 ~
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | key2 length   |         Key 2                                 ~
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | nonce length  |         Nonce                                 ~
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |        Duration                                               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Where:¶

- IPsec-SimType (8 bits):

assigned the value 69 by this document, indicating the type of the simplified IPsec SA Sub-TLV.¶

- IPsec-SA subTLV Length (8 bits):

specifies the length of this Sub-TLV in octets, excluding the Type and the Length fields. The length is 25 or longer.¶

- Reserved (16 bits):

Reserved for future use. It SHOULD be set to zero on transmission and MUST be ignored on receipt.¶

- Transform (8 bits):

• Transform = 1 means AH,¶

• Transform = 2 means ESP, or¶

• Transform = 3 means AH+ESP.¶

All other transform values are outside the scope of this document.¶

- IPsec Mode (8 bits):

• Mode = 1 indicates that the Tunnel Mode is used.¶

• Mode = 2 indicates that the Transport mode is used.¶

All mode values besides 1 and 2 are outside the scope of this document.¶

- AH algorithms (8 bits):

AH authentication algorithms supported. The values are specified by [IANA-AH]. Each SD-WAN edge node can support multiple authentication algorithms; send to its peers to negotiate the strongest one.¶

- ESP algorithms (8 bits):

the ESP algorithms supported. Its values are specified by [IANA-ESP]. One SD-WAN edge node can support multiple ESP algorithms and send them to its peers to negotiate the strongest one. The default algorithm is AES-256.¶

When a node supports multiple authentication algorithms, the initial UPDATE needs to include the "Transform Sub-TLV"¶

- Rekey Counter (Security Parameter Index) (4 octet):

indicates the count for rekeying.¶

- key1 length (8 bits):

indicates the IPsec public key 1 length¶

- Public Key 1:

IPsec public key 1¶

- key2 length (8 bits):

indicates the IPsec public key 2 length¶

- Public Key 2:

IPsec public key 2¶

- none-length (8 bits):

indicates the Nonce key length¶

- Nonce:

IPsec Nonce¶

- Duration (32 bits):

specifying the security association (SA) life span in seconds.¶

9.1. Error handling for the Tunnel Encapsulation Signaling

The error handling for the tunnel encapsulation signaling (Encaps-EC and TEA) adheres to the error handling and validation specified by [RFC9012].¶

The Tunnel encapsulation signaled with the client routes indicates the Egress endpoint via Next Hop in the Encaps-EC or the TEA SubTLV for Tunnel Egress Endpoints. As indicated in sections 5.1 and 5.2, the SD-WAN Hybrid tunnel follows the validation section 6, 8, and 13 from [RFC9012].¶

The SD-WAN client routes associate the same NLRIs that [RFC9012] associates with the Encaps-EC and the TEA using the validation specified in [RFC9012] in sections 6, 8, and 13. When the SD-WAN Hybrid Tunnel is associated with the SD-WAN NLRI, and all RFC9012 validation rules in section 6, 8, and 13 are extended to apply to the SD-WAN NLRI.¶

[RFC9012] contains the necessary detail to specify validation for the new SubTLVs present for the SD-WAN Tunnel type. However, to aid users of this document the following recap of validation of [RFC9012] is provided below. The validation from section 13 of [RFC9012] includes:¶

• Invalid tunnel type must be treat if the TLV was not present.¶

• A malformed subTLVs must be treated as an unrecognized subTLV except for Tunnel Egress Endpoint. If Tunnel Egress Endpoint is malformed, the entire TLV must be ignored.¶

• Multiple incidents of Tunnel Egress Endpoint, Encapsulation, DS, UDP Destination Port, Embedded Label Handling, MPLS Label Stack, Prefixes-SID cause the first incident of these subTLVs to be utilized. Subsequent TLVs after the first one per type are ignored (per RFC9012), but propogated.¶

• If a subTLV is meaningless for a tunnel type, the subTLV is ignored, but the subTLV is not considered malformed or removed from the Tunnel Attribute propagated with the NLRI.¶

For SD-WAN client routes with a TEA with a SD-WAN Hybrid Tunnel type, the Extended Port subTLV and the IPSec SubTLVs (IPsec SA-ID, IPSec nonce, IPSec Public Key, IPsec Proposal, and Simplified IPsec SA) are meaningful, but may be rarely sent.¶

For SD-WAN NLRI underlay routes, the the Extended Port subTLV and the IPSec SubTLVs (IPsec SA-ID, IPSec nonce, IPSec Public Key, IPsec Proposal, and Simplified IPsec SA) are valid and meaningful. Incorrect fields within any of these 5 TLVs or subSubTLVs within the TLVs should cause the subTLV to be treated as malformed SubTLV. Per [RFC9012], a malformed subTLV is treated as an unrecognized subTLV. Multiple copies of each SubTLV may be included in a single TLV.¶

9.2. Error Handling for NLRI

The SD-WAN NLRI [AFI 1/SAFI = 74] utilizes a route type field to describe the format of the NLRI. This specification only allows an NLRI with a type value of 1. An NLRI with a type of field of another value is ignored and not processed. The implementation MAY log an error upon a reception of an type value outside of Route Type 1. Error handling for the SD-WAN NLRI also adheres to the BGP Update error handling specified in [RFC7606].¶

Section 6.1 specifies that Route Type 1 has a tuple of (Port-Local-ID, SD-WAN-Color, SD-WAN-Node-ID). Port-Local-ID may be zero if the NLRI applies to multiple ports. The BGP Peer receiving the NLRI must have pre-configured inbound filters to set the preference for the SD-WAN NLRI tuple.¶

Since a Port-Local-ID value of zero indicates the NLRI applies to multiple ports, it is possible to have the following NLRI within a packet (or received in multiple packets):¶

• Port-Local-ID (0), SD-WAN-Color (10), SD-WAN-Node-ID (2.2.2.2),¶

• Port-Local-ID (0), SD-WAN-Color (20), SD-WAN-Node-ID (2.2.2.2), and¶

• Port-Local-ID (0), SD-WAN-Color (30), SD-WAN-Node-ID (2.2.2.2).¶

These NLRI may simply indicate that there are three groups of tunnels for SD-WAN-Node-ID (2.2.2.2) assigned three colors. For example, these tunnels could represent three types of gold, silver and bronze network service.¶

The local policy configuration in the BGP peer receiving this NLRI must determine the validity of the route based on policy. Local configuration and policy must be carefully constrain the SD-WAN-NLRI, tunnels, and IPsec security associations in to create a "walled garden".¶

In the future, other proposals for a SD-WAN NLRI may specify a different route type. Those proposals must specify the following:¶

• validation for new Route Type in the SD-WAN-NLRI, and¶

• how the new Route Type interacts with the Route Type 1.¶

9.3. SDWAN NLRI and Tunnel Encapsulation Attribute

The SD-WAN NLRI (AFI=1/SAFI=74) must be paired with Tunnel Encapsulation attribute with a tunnel TLV for tunnel type SD-WAN-Hybrid. If the SD-WAN NLRI exist in an BGP UPDATE without a Tunnel Encapsulation Attribute (TEA) with a tunnel TLV for tunnel type SD-WAN-Hybrid, the NLRI is considered malformed and Treat-as-withdraw approach (per RFC7606).¶

The SD-WAN NLRI should not be paired with Encapsulation Extended Community. If an SD-WAN NLRI is paried Encapsulation Extended Community rather than a Tunnel Encapsulation Attribute, the SD-WAN NLRI is considered malformed and Treat-as-withdraw approach (per [RFC7606]) should be used.¶

10.1. Detecting Misaligned Tunnels

It is critical that the Hybrid SD-WAN Tunnel have correctly forward traffic based on the local policy on the client routes, the tunnel egress and tunnel ingress, and the security association. The RR reflector and the BGP peer must check that the client routes, tunnel egress, tunnel ingress, and security associations align with expected values for a tunnel.¶

10.2. IPsec Attributes Mismatch

Each BGP peer (e.g. a C-PE) advertises a SD-WAN SAFI Underlay NLRI to the other BGP peers via a BGP Route Reflector to establish pairwise IPsec Security Associations (SA) between itself and other remote BGP Peers. During the SD-WAN SAFI NLRI advertisement, the BGP Peer originating may pass information about security asssociation in one of three forms:¶

• an identifier for a pre-configured and established IPsec Security Association,¶

• a simplified set of security parameters for setting up a IPsec Security association (Transform, IPsec Mode, AH and ESP Algorithms, rekey counter, 2 public keys, nonce, and duration of security association), or¶

• a flexible set of security parameters where Nonce, Public Key, and SA Proposal are uniquely specified.¶

For existing IPsec Security associations, the receiving BGP peer can simply utilize one of these existing security associations to pass data. If multiple IPsec associations are pre-configured, the local policy on the SD-WAN Edge Node may may help select which security association is chosen for the SD-WAN Hybrid Tunnel.¶

If the receiving and originating BGP peer engage in a set-up for the IPsec security associations for the link within the SD-WAN Hybrid tunnel, IPsec mechansims require that there are matching IPsec transforms. Without common IPsec transforms, the IPsec set-up process cannot operate.¶

12.1. Hybrid (SD-WAN) Overlay SAFI

IANA has assigned SAFI = 74 as the Hybrid (SD-WAN) SAFI.¶

12.2. Tunnel Encapsulation Attribute Type

IANA is requested to assign a type from the BGP Tunnel Encapsulation Attribute Tunnel Types as follows [RFC8126]:¶

 Value   Description    Reference
-----   ------------   ---------
 25       SD-WAN-Hybrid   (this document)

12.3. Tunnel Encapsulation Attribute Sub-TLV Types

IANA is requested to assign the following sub-Types in the BGP Tunnel Encapsulation Attribute Sub-TLVs registry:¶

   Value   Type Description            Reference     Section
   -----   -----------------------     ------------- -------
    64  IPSEC-SA-ID Sub-TLV            This document  8.1
    65  Extended Port Property Sub-TLV This document  7.0
    66  IPsec SA Rekey Counter Sub-TLV This document  8.2
    67  IPsec Public Key Sub-TLV       This document  8.3
    68  IPsec SA Proposal Sub-TLV      This document  8.4
    69  Simplified IPsec SA sub-TLV    This document  8.5

12.4. Sub-Sub-TLV Types within Extended Port Property Sub-TLV

IANA is requested to assign the following Sub-Sub-Types within the Extended Port Property Sub-TLV:¶

   Value   Type Description                      Reference     Section
   -----   -----------------------               ------------- -------
    1     Underlay Network Property Sub-Sub-TLV   This document  7.1

13.1. Normative References

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>.

[RFC4271]

Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A Border Gateway Protocol 4 (BGP-4)", RFC 4271, DOI 10.17487/RFC4271, January 2006, <https://www.rfc-editor.org/info/rfc4271>.

[RFC4360]

Sangli, S., Tappan, D., and Y. Rekhter, "BGP Extended Communities Attribute", RFC 4360, DOI 10.17487/RFC4360, February 2006, <https://www.rfc-editor.org/info/rfc4360>.

[RFC4364]

Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 2006, <https://www.rfc-editor.org/info/rfc4364>.

[RFC4456]

Bates, T., Chen, E., and R. Chandra, "BGP Route Reflection: An Alternative to Full Mesh Internal BGP (IBGP)", RFC 4456, DOI 10.17487/RFC4456, April 2006, <https://www.rfc-editor.org/info/rfc4456>.

[RFC4760]

Bates, T., Chandra, R., Katz, D., and Y. Rekhter, "Multiprotocol Extensions for BGP-4", RFC 4760, DOI 10.17487/RFC4760, January 2007, <https://www.rfc-editor.org/info/rfc4760>.

[RFC7296]

Kaufman, C., Hoffman, P., Nir, Y., Eronen, P., and T. Kivinen, "Internet Key Exchange Protocol Version 2 (IKEv2)", STD 79, RFC 7296, DOI 10.17487/RFC7296, October 2014, <https://www.rfc-editor.org/info/rfc7296>.

[RFC7606]

Chen, E., Ed., Scudder, J., Ed., Mohapatra, P., and K. Patel, "Revised Error Handling for BGP UPDATE Messages", RFC 7606, DOI 10.17487/RFC7606, August 2015, <https://www.rfc-editor.org/info/rfc7606>.

[RFC8126]

Cotton, M., Leiba, B., and T. Narten, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 8126, DOI 10.17487/RFC8126, June 2017, <https://www.rfc-editor.org/info/rfc8126>.

[RFC8174]

Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>.

[RFC8277]

Rosen, E., "Using BGP to Bind MPLS Labels to Address Prefixes", RFC 8277, DOI 10.17487/RFC8277, October 2017, <https://www.rfc-editor.org/info/rfc8277>.

[RFC8489]

Petit-Huguenin, M., Salgueiro, G., Rosenberg, J., Wing, D., Mahy, R., and P. Matthews, "Session Traversal Utilities for NAT (STUN)", RFC 8489, DOI 10.17487/RFC8489, February 2020, <https://www.rfc-editor.org/info/rfc8489>.

[RFC9012]

Patel, K., Van de Velde, G., Sangli, S., and J. Scudder, "The BGP Tunnel Encapsulation Attribute", RFC 9012, DOI 10.17487/RFC9012, April 2021, <https://www.rfc-editor.org/info/rfc9012>.

13.2. Informative References

[IANA-AH]

IANA, "IANA-AH", <https://www.iana.org/assignments/isakmp-registry/isakmp-registry.xhtml#isakmp-registry-9>.

[IANA-ESP]

IANA, "IANA-ESP", <https://www.iana.org/assignments/isakmp-registry/isakmp-registry.xhtml#isakmp-registry-9>.

[Net2Cloud]

L. Dunbar, A Malis, C. Jacquenet, M. Toy and K. Majumdar, "Dynamic Networks to Hybrid Cloud DCs: Problem Statement and Mitigation Practice", September 2023, <https://datatracker.ietf.org/doc/draft-ietf-rtgwg-net2cloud-problem-statement/>.

[RFC5114]

Lepinski, M. and S. Kent, "Additional Diffie-Hellman Groups for Use with IETF Standards", RFC 5114, DOI 10.17487/RFC5114, January 2008, <https://www.rfc-editor.org/info/rfc5114>.

[RFC5903]

Fu, D. and J. Solinas, "Elliptic Curve Groups modulo a Prime (ECP Groups) for IKE and IKEv2", RFC 5903, DOI 10.17487/RFC5903, June 2010, <https://www.rfc-editor.org/info/rfc5903>.

[RFC9016]

Varga, B., Farkas, J., Cummings, R., Jiang, Y., and D. Fedyk, "Flow and Service Information Model for Deterministic Networking (DetNet)", RFC 9016, DOI 10.17487/RFC9016, March 2021, <https://www.rfc-editor.org/info/rfc9016>.