Brkdcn-2035.pdf

#CLMEL

VXLAN BGP EVPN Based Multi-Site Lukas Krattiger – Principal Engineer BRKDCN-2035

#CLMEL

Cisco Webex Teams Questions?

Use Cisco Webex Teams (formerly Cisco Spark) to chat with the speaker after the session

How 1 Open the Cisco Events Mobile App 2 Find your desired session in the “Session Scheduler” 3 Click “Join the Discussion” 4 Install Webex Teams or go directly to the team space 5 Enter messages/questions in the team space cs.co/ciscolivebot#BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

3

Session Objectives At the end of the session, the participants should be able to: 

Articulate the value proposition of the new VXLAN MultiSite architecture and list several use cases for which it should be positioned



Understand the functionalities and specific design considerations associated to VXLAN Multi-Site

Initial assumption: 

The audience already has a good knowledge of the VXLAN EVPN technology and its use to deploy modern Data Centre Fabrics #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

4

Agenda

 Data Centre Interconnect (DCI) Evolution

 VXLAN Multi-Site Introduction • Functional Components and Use Cases • HW/SW Support and Scalability Values • Supported Topologies

 VXLAN Multi-Site Deep Dive • Border Gateway Deployment Considerations • Inter-Site BUM Traffic Handling • Control and Data Planes

• Failure Detection on BGWs • Connectivity to the External Layer 3 Domain • Network Services Integration • Legacy Site Integration • Configuration Specifics

 Conclusions #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

5

Data Centre Interconnect (DCI) Evolution

#CLMEL

Overlays Evolve and Spread DC Local Overlay

SS

End-to-End Overlay

S

L

L

S

S

S

L

L

....

SS

SS

SS

S

L

L

L

S

S

S

L

L

....

L

Single Logical Data Centre #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

8

Back Then

VXLAN for Interconnecting Networks

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

9

Changing the Paradigm with Overlays DC Local Overlay

SS

Multi-Site Overlay

S

L

L

S

S

S

L

L

....

SS

SS

SS

S

L

L

L

S

S

S

L

L

....

L

Multiple Logical Data Centre #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

10

VXLAN Evolves as the Control Plane Evolves! Before Yesterday Yet Another Encapsulation  

Flood and Learn (Multicast-based) Data-Plane only

Yesterday

VXLAN for the Data Centre – Intra-DC   

Control-Plane Active VTEP Discovery Multicast and Unicast

Today

VXLAN for DCI – Inter-DC     

#CLMEL

DCI Ready ARP/ND caching/suppress Multi-Homing Failure Domain Isolation Loop Protection

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

11

Inter-X Connectivity VXLAN Multi-Fabric

VXLAN Multi-Pod BGP EVPN

EVPN ControlFabric #1 Plane Domain 1

Overlay VTE P

VTE P

Bar em eta l

Bar em eta l

VTE P

EVPNFabric Control-Plane #1 Domain 1

EVPN ControlFabric #2 Plane Domain 2

Overlay VTE P

VTE P

VTE P

VTE P

VTE P

VTE P

Bar em eta l

Bar em eta l

Bar em etal

Bar em etal

Single Data-Plane – End-to-End

VTE P

EVPNFabric Control-Plane #1 Domain 1

EVPNFabric Control-Plane #2 Domain 2

Overlay

VTE P

VXLAN Multi-Site

Overlay

Overlay VTE P

Data-Plane Domain 1

VTE P

DCI Data-Plane

VTE P

EVPNFabric Control-Plane #2 Domain 2

BGP EVPN

VTE P

VTE P

VTE P

VTE P

Bar em etal

Bar em etal

Bar em etal

Bar em etal

Data-Plane Domain 2

VTE P

Data-Plane Domain 1

Overlay VTE P

VTE P

DCI Data-Plane

VTE P

 Multiple Fabrics – Normalised through Ethernet

 Multiple Fabrics with Integrated DCI (DCI2)

 Build Hierarchy in the Underlay – Flatten it in the Overlay

 Multiple Fabrics Interconnect using DCI (Layer 2 and Layer 3)

 Integrated DCI – Scaling within and between Fabrics

BRKDCN-2035

VTE P

Bar em etal

Bar em etal

Data-Plane Domain 2

 Single Fabric with End-toEnd Encapsulation

#CLMEL

VTE P

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

12

VXLAN Multi-Site Introduction

#CLMEL

Functional Components and Use Cases

#CLMEL

VXLAN Multi-Site Functional Components

https://tools.ietf.org/html/draft-sharma-multi-site-evpn Site-External DCI

(IP Routing and Increased MTU Support)

Border Gateways

(Key Functional Components of VXLAN Multi-Site Architecture)

Spine

VTEP

VTEP

VTEP

VTEP

VTEP

VTEP

VTEP

BGW

BGW

BGW

BGW

Spine

Spine

VTEP

Site 1

Spine

VTEP

Spine

VTEP

VTEP

VTEP

VTEP

Site-Internal Fabric (Common VXLAN and BGP-EVPN Functions) #CLMEL

BRKDCN-2035

VTEP

Spine

Spine

VTEP

Spine

VTEP

VTEP

VTEP

Site n © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

24

VXLAN Multi-Site Characteristics 

Multiple Overlay Domains – Interconnected and Controlled



Multiple Overlay Control-Plane Domains – Interconnected and Controlled



Multiple Underlay Domains - Isolated



Multiple Replication Domains for BUM – Interconnected and Controlled



Multiple VNI Administrative Domains – Phase 2

Underlay Isolation – Overlay Hierarchies #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

25

VXLAN Multi-Site Main Use Cases

Scale-Up Model to Build a Large Intra-DC Network

Data Centre Interconnect (DCI)

Integration with Legacy Networks (Coexistence and/or Migration) #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

26

VXLAN Multi-Site

Inter-Site Network Routing Table Border Site1: Border Site2: 10.1.1.101 10.2.2.101 10.1.1.102 10.2.2.102 10.1.1.111 10.2.2.222

Underlay Isolation

Multi-Site VIP 10.1.1.111 VTEP

VTEP

BGW

BGW

Border (PIP) 10.1.1.101 Spine

VTEP 10.1.1.1 VTEP

VTEP

VTEP

Border (PIP) 10.1.1.102

Spine

Spine

VTEP

Site 1

Multi-Site VIP 10.2.2.222

Site-External DCI

Spine

Site 1 Underlay Routing Table Border: Leaf: 10.1.1.101 10.1.1.1 10.1.1.102 10.1.1.2 10.1.1.111 10.1.1.3 10.1.1.4 10.1.1.5 10.1.1.6 10.1.1.7

VTEP

VTEP

BGW

BGW

Border (PIP) 10.2.2.101

Spine

VTEP

VTEP

VTEP

#CLMEL

VTEP

VTEP

VTEP

Border (PIP) 10.2.2.102

Spine

Spine

VTEP

Site n BRKDCN-2035

Spine

Site n Underlay Routing Table Border: Leaf: VTEP 10.2.2.101 10.2.2.1 10.2.2.7 10.2.2.102 10.2.2.2 10.2.2.222 10.2.2.3 10.2.2.4 10.2.2.5 10.2.2.6 10.2.2.7

VTEP

VTEP

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

VTEP

27

VXLAN Multi-Site

Introducing the Border Gateway Overlay Multi-Site Border Gateway (BGW) - Anycast Cluster -

Spine

Multi-Site VIP 10.1.1.111

Multi-Site VIP 10.2.2.222

VTEP

VTEP

VTEP

VTEP

BGW

BGW

BGW

BGW

Overlay Site 1 Spine

Spine

Spine

Spine

Overlay Site n Spine

Spine

Spine

Any VTEP VTEP

VTEP

VTEP

VTEP

VTEP

VTEP

VTEP

VTEP

VTEP

Site 1

VTEP

VTEP

VTEP

VTEP

VTEP

Site n #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

28

Multi-Site – VXLAN Tunnel Adjacencies BG102# show nve peers Interface Peer-IP VNI ---------- ----------- -----Overlay Multi-Site nve1 10.1.1.1 30000 nve1 10.1.1.4 30000 nve1 10.2.2.222 30000

Spine

VTEP 10.1.1.1 VTEP

VTEP

Multi-Site VIP 10.1.1.111

Multi-Site VIP 10.2.2.222

VTEP

VTEP

VTEP

VTEP

BGW

BGW

BGW

BGW

Overlay Site 1

VTEP

Up Time ---------00:12:16 03:18:06 00:12:23

Spine

Spine

VTEP 10.1.1.4 VTEP

Leaf1-1# show nve peers Interface Peer-IP ---------- ----------Site 1 nve1 10.1.1.4 nve1 10.1.1.111

Spine

VTEP

VNI -----30000 30000

Spine

VTEP

VTEP

VTEP

Up Time ---------03:18:06 00:12:23 #CLMEL

VTEP

Overlay Site n

VTEP

Spine

Spine

VTEP

VTEP

Leaf2-7# show nve peers Interface Peer-IP VNI Site n--------------- ----------nve1 10.2.2.222 30000 BRKDCN-2035

Spine

VTEP

VTEP 10.2.2.7 VTEP

Up Time ---------00:12:25

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

29

HW/SW Support and Scalability Values

#CLMEL

VXLAN Multi-Site HW/SW Support 

Minimum hardware and software requirements for Border Gateways Item



Requirement Cisco Nexus Cisco Nexus Cisco Nexus Cisco Nexus Cisco Nexus Cisco Nexus

9300 EX platform 9300 FX platform 9300 FX2 platform 9364C platform 9500 platform with X9700-EX line card 9500 platform with X9700-FX line card

Cisco Nexus Hardware

• • • • • •

Cisco Nexus Software

Cisco NX-OS Software Release 7.0(3)I7(1) or later

The hardware and software requirements for the Site-Internal nodes of a VXLAN BGP EVPN site remain the same as those without the EVPN Multi-Site BGW #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

31

VXLAN Multi-Site

Scalability Values as of 9.2(2) Release Multi-Site Scale Number of Sites

10

Number of BGWs per Site

4 (Anycast), 2 (vPC)

VTEPs per Site

256

Border Gateway (BGW) Scale

EX/FX/FX2

N9364C

Number of Layer-2 VNI (VLAN)

2,000

Number of Layer-3 VNI (VRF)

1,000

MAC per BGW

90,000

64,000

IPv4 Host Routes per BGW*

~530,000

~60,000

IPv4 Network Routes per BGW*

~530,000

~8,000

~24,000

~7,000

~260,000

~2,000

IPv6 Host Routes per BGW* IPv6 Network Routes per BGW*

*The values provided in these tables focus on the scalability of one particular route scale at a time #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

32

Supported Topologies

#CLMEL

BGW-to-Cloud Layer-3 Network

Site 1

Leaf

Leaf

BGW

BGW

Spine

Spine

Leaf

Leaf

Site 2

Leaf

Leaf

Leaf

BGW

Leaf

BGW

BGW

Spine

Spine

Leaf

Leaf

#CLMEL

BGW

Leaf

Site n

Leaf

BRKDCN-2035

Leaf

BGW

Leaf

BGW

BGW

Spine

Spine

Leaf

Leaf

BGW

Leaf

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

Leaf

34

BGWs Between Spine and Super-Spine Super-Spine

Site 1

Leaf

Leaf

BGW

BGW

Spine

Spine

Leaf

Leaf

Site 2

Leaf

Leaf

Leaf

BGW

Leaf

Super-Spine

BGW

BGW

Spine

Spine

Leaf

Leaf

#CLMEL

BGW

Leaf

Site n

Leaf

BRKDCN-2035

Leaf

BGW

Leaf

BGW

BGW

Spine

Spine

Leaf

Leaf

BGW

Leaf

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

Leaf

36

BGWs on Spine Super-Spine

Site 1

Leaf

Leaf

BGW Spine

BGW Spine

Leaf

Leaf

Site 2

Leaf

Leaf

Leaf

Super-Spine

BGW Spine

BGW Spine

BGW Spine

BGW Spine

Leaf

Leaf

Leaf

Leaf

#CLMEL

Site n

Leaf

BRKDCN-2035

Leaf

BGW Spine

BGW Spine

BGW Spine

BGW Spine

Leaf

Leaf

Leaf

Leaf

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

Leaf

38

BGWs Back-to-Back

Site 1

Leaf

Leaf

BGW

BGW

Spine

Spine

Leaf

Leaf

Site 2

Leaf

Leaf

Leaf

#CLMEL

BRKDCN-2035

Leaf

BGW

BGW

Spine

Spine

Leaf

Leaf

Leaf

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

Leaf

40

VXLAN Multi-Site Deep Dive

#CLMEL

Border Gateway Deployment Considerations

#CLMEL

VXLAN Multi-Site

Border Gateways Deployment Considerations 



Border Gateways used for two main functions: •

Interconnecting each site to the Inter-Site network (for East-West traffic flows)

•

Connecting each site to the external Layer 3 domain (for North-South traffic flows)

•

May also be used to connect endpoints and/or network service nodes (FWs, ADCs)

Possible deployment models: • •



Anycast Border Gateways BGW

BGW

BGW

BGW

VTEP

VTEP

VTEP

VTEP

Site 1

VPC Border Gateways

Anycast Border Gateways VPC Border Gateways (from ACI release 9.2(1))

BGW

BGW

VTEP

VTEP

BGW function enablement in the VXLAN EVPN fabric: •

BGWs as leaf nodes

•

BGWs as spine nodes (Border-Spines) Site 1 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

44

Anycast Border Gateways

#CLMEL

VXLAN Multi-Site

Anycast Border Gateway (1) Anycast Border Gateway

BGW

BGW

BGW

BGW

VTEP

VTEP

VTEP

VTEP



Up to 4 Border Gateways



Border Gateway •

Deploying at Leaf – 7.0(3)I7(1)

•

Deploying at Spine – 7.0(3)I7(2)

Site 1 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

46

VXLAN Multi-Site

Anycast Border Gateway (2) Anycast Border Gateway 

Multi-Site VIP 10.1.1.111 BGW

BGW

BGW

BGW

VTEP

PIP-BGW2 10.1.1.102

VTEP

PIP-BGW3 10.1.1.103

VTEP

PIP-BGW4 10.1.1.104

PIP-BGW1 10.1.1.101

Common Multi-Site Virtual IP (Multi-Site VIP) across BGWs •

Multi-Site VIP for communication between the Border Gateways in different Sites

•

Multi-Site VIP for communication between Border Gateways and Leaf nodes within a Site

VTEP

Multi-Site VIP 10.1.1.111



Site 1 #CLMEL

Individual Primary IP (PIP) per BGW •

Used for Broadcast, Unknown Unicast and Multicast (BUM) replication

•

PIP for communication with Single-Homed endpoints (routed only), intra- and interSite BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

47

VXLAN Multi-Site

Anycast Border Gateway (3) Anycast Border Gateway Type: 00 System MAC: 00:00:00:00:00:01 Ethernet Segment: 00:00:07

4

DF 30010

IP: 10.1.1.101 VNI: 30010

BGW

BGW

BGW

BGW

VTEP

VTEP

VTEP

VTEP

DF 30011

DF 30012

DF 30099





BGP EVPN

RR

RR

Spine

Spine

Per-VNI Designated Forwarder (DF) election •

Each BGW can serve as DF for a single or a set of Layer-2 VNIs

•

DF election and assignment is automatic

Using BGP EVPN Route Type 4 for DF election •

Operator Managed Assignment (Type: 00)

•

Six Octet Site Identifier (System MAC: 00:00:00:00:00:01)

•

Multi-Site Discriminator (Ethernet-Segment: 00:00:07)

•

Originators IP Address (PIP): 10.1.1.101

•

Layer-2 VNI: 30010

Site 1 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

48

VXLAN Multi-Site

Anycast Border Gateway (4) External Connectivity

Point-to-Point L3 Links (Physical/Sub-Interfaces)



BGW

BGW

BGW

BGW

VTEP

PIP-BGW2 10.1.1.102

VTEP

PIP-BGW3 10.1.1.103

VTEP

PIP-BGW4 10.1.1.104

PIP-BGW1 10.1.1.101 .1

Anycast Border Gateway

.1

VTEP



Point-to-Point L3 Links (Physical/Sub-Interfaces)

Point-to-Point L3 Links ADC

ADC

ADC 0000.3010.1102 192.168.10.102

ADC 0000.3010.1101 192.168.10.101

Single-Homed End-Points only connected with L3 links •

Services Appliance (i.e. Firewall, ADC etc.)

•

External routers

•

No SVI support on BGW nodes

Advertised and Reachable through Individual Primary IP Address (PIP) •

Intra-Site: Leaf nodes use PIP to reach the device connected to Border Gateways

•

Inter-Site: Remote Border Gateways use PIP to reach the device connected to Border Gateways

VTEP

Site 1 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

49

VPC Border Gateways NXOS Release 9.2(1)

#CLMEL

Anycast BGW vs. VPC Border Gateway Anycast Border Gateway

NXOS Release 9.2(1)

VPC Border Gateway

• Up to 4 BGW

• 2 BGW with physical VPC Peer-Link

• Shared Nothing

• Small Deployments

• Simple Failure Scenarios

• End-Point or Network Services Connectivity on BGW

• Any Deployments

• Migration Use-Cases (Brownfield)

• No End-Point or Network Services Connectivity on BGW

• Pseudo-BGW to BGW

• Greenfield Deployments

• Classic Ethernet/FabricPath to VXLAN EVPN

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

51

NXOS Release 9.2(1)

Multi-Site Border Gateway – Anycast vs. vPC •

Both Anycast and vPC Border Gateway needs to be configured with a common Multi-Site VIP address and an individual Primary IP (PIP) address

•

vPC Border Gateways share a secondary IP address to be used as vPC virtual IP (vPC VIP)

vPC BGW

Anycast BGW

vPC VIP 11.11.11.11 Multi-Site VIP 100.100.100.100

Multi-Site VIP 100.100.100.100 PIP1 10.1.1.1

….

VTEP

Fabric

BGW1

VTEP BGW4

Spine

VTEP

PIP1 10.1.1.1

PIP1 10.1.2.1 Fabric

Spine

VTEP

VTEP

PIP1 10.1.2.1

VTEP

VTEP

BGW1

BGW2

Spine

VTEP

VTEP

#CLMEL

BRKDCN-2035

Spine

VTEP

VTEP

VTEP

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

52

NXOS Release 9.2(1)

VXLAN Multi-Site

VPC Border Gateway and Transit Traffic VPC Border Gateway  Multi-Site VIP 10.1.1.111 BGW

•

BGW

VTEP

PIP-BGW1 10.1.1.101

Common Multi-Site Virtual IP (Multi-Site VIP) across BGWs

VTEP

VPC VIP 10.1.1.121

PIP-BGW2 10.1.1.102

•

Multi-Site VIP 10.1.1.111



Multi-Site VIP for Inter-Site transit communication (transit)

Common VPC Virtual IP (VPC VIP) across BGWs •

Used by default for communication with external networks

•

Used for Broadcast, Unknown Unicast and Multicast (BUM) replication

Individual Primary IP (PIP) per BGW •

Used for communication with external networks when “advertised-pip” is configured

Site 1 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

53

NXOS Release 9.2(1)

VXLAN Multi-Site

VPC Border Gateway and Locally Attached End-Points VPC Border Gateway  Multi-Site VIP 10.1.1.111 BGW

BGW

VTEP

VTEP

Anycast Gateway

Anycast Gateway

VPC VIP 10.1.1.121

 Multi-Site VIP 10.1.1.111 ADC

Baremetal

ADC 0000.3010.1102 192.168.10.102

EP 0000.3010.1101 192.168.10.101

Single- or Dual-Homed End-Points •

Services Appliance (i.e. Firewall, ADC etc.)

•

Physical or Virtual Servers

•

Anycast Gateway function offered to the endpoints

Advertised and Reachable through VPC Virtual IP Address (VPC VIP) •

Intra-Site: Leaf nodes use VPC VIP to reach EndPoints connected to Border Gateways

•

Inter-Site: Remote Border Gateways use VPC VIP to reach End-Points connected to Border Gateways

•

Traffic potentially traverses VPC Peer-Link

Site 1 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

54

NXOS Release 9.2(1)

VXLAN Multi-Site

VPC Border Gateway and Designated BUM Forwarder VPC Border Gateway

BGW

BGW

VTEP

VTEP

DF



VPC-based Designated Forwarder Election



Per-Site Designated Forwarder (DF) election •

Using same approach as in VPC

•

Best Path to Rendezvous-Point or VPC Primary Node

•

Same VPC node is elected DF for all the Layer-2 VNIs

VPC VIP 10.1.1.121

Site 1 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

55

NXOS Release 9.2(1)

VPC Border Gateways

Traffic between Locally Connected Endpoints across Sites Src vPC VIP1

Dst vPC VIP2

VXLAN Header

Original Packet

Inter-Site Network vPC VIP2 22.22.22.22

vPC VIP1 11.11.11.11 VTEP

VTEP

VTEP

VTEP

BGW1

BGW2

BGW1

BGW2

Site 1

Site 2

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

56

NXOS Release 9.2(1)

VPC Border Gateways

Traffic between Locally Connected Endpoints and Remote L3Out Src vPC VIP1

Dst vPC VIP2

VXLAN Header

Original Packet

Inter-Site Network vPC VIP2 22.22.22.22

vPC VIP1 11.11.11.11 VTEP

VTEP

VTEP

VTEP

BGW1

BGW2

BGW1

BGW2

L3

Site 1

Site 2

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

57

NXOS Release 9.2(1)

VPC Border Gateways

Traffic between Site Connected Endpoints across Sites Src Multi-Site VIP1

Dst Multi-Site VIP2

VXLAN Header

Original Packet

Inter-Site Network Multi-Site VIP2 200.200.200.200

Multi-Site VIP1 100.100.100.100 VTEP

VTEP

VTEP

VTEP

BGW1

BGW2

BGW1

BGW2

VTEP

VTEP

Site 2

Site 1

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

58

NXOS Release 9.2(1)

VPC Border Gateways BUM Traffic across Sites Src vPC VIP1

Dst vPC VIP2

VXLAN Header

Original Packet

Inter-Site Network

DF

vPC VIP1 11.11.11.11

DF

vPC VIP2 22.22.22.22

VTEP

VTEP

VTEP

VTEP

BGW1

BGW2

BGW1

BGW2

BUM Traffic redirected via vPC peer-link toward the DF

VTEP

VTEP

Site 2

Site 1

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

59

Inter-Site BUM Traffic Handling

#CLMEL

VXLAN Multi-Site BUM Traffic Forwarding

Overlay Multi-Site

Spine

VTEP

VTEP

VTEP

VTEP

VTEP

VTEP

BGW

BGW

BGW

BGW

Overlay Site 1

VTEP

Spine

Spine

VTEP

VTEP

Spine

Spine

VTEP

VTEP

VTEP

VTEP

Overlay Site n

VTEP

Spine

Spine

VTEP

VTEP

Spine

VTEP

VTEP

BUM Site 1

Site n Baremetal

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

61

VXLAN Multi-Site

BUM Replication Modes (Multicast Intra-Site) Overlay Multi-Site Ingress Replication

Spine

VTEP

VTEP

VTEP

VTEP

BGW

BGW

BGW

BGW

Overlay Site 1 Spine

Spine

Spine

Spine

Overlay Site n

Multicast VTEP

VTEP

VTEP

VTEP

Spine

Spine

Spine

Multicast VTEP

VTEP

VTEP

VTEP

VTEP

Site 1

VTEP

VTEP

VTEP

VTEP

VTEP

Site n #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

62

VXLAN Multi-Site

BUM Replication Modes (Ingress Replication Only) Overlay Multi-Site Ingress Replication

Spine

VTEP

VTEP

VTEP

VTEP

BGW

BGW

BGW

BGW

Overlay Site 1 Spine

Spine

Spine

Spine

Ingress Replication VTEP

VTEP

VTEP

VTEP

VTEP

Overlay Site n Spine

Spine

Spine

Ingress Replication VTEP

VTEP

VTEP

VTEP

Site 1

VTEP

VTEP

VTEP

VTEP

VTEP

Site n #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

63

VXLAN Multi-Site

BUM Replication Modes (Mixed Mode Intra-Site) Overlay Multi-Site Ingress Replication

Spine

VTEP

VTEP

VTEP

VTEP

BGW

BGW

BGW

BGW

Overlay Site 1 Spine

Spine

Spine

Spine

Overlay Site n

Ingress Replication VTEP

VTEP

VTEP

VTEP

VTEP

Spine

Spine

Spine

Multicast VTEP

VTEP

VTEP

VTEP

Site 1

VTEP

VTEP

VTEP

VTEP

VTEP

Site n #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

64

VXLAN Multi-Site BUM Traffic Policing

Overlay Multi-Site

Spine

VTEP

VTEP

VTEP

VTEP

BGW

BGW

Storm Control Broadcast 0-100% Unknown Unicast 0-100% Multicast 0-100%

Overlay Site 1

VTEP

Spine

Spine

VTEP

VTEP

Spine

Spine

VTEP

VTEP

VTEP

VTEP

VTEP

VTEP

BGW

BGW

Overlay Site n

VTEP

Spine

Spine

VTEP

VTEP

Spine

VTEP

VTEP

BUM Site 1

Site n Baremetal

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

65

Control and Data Planes

#CLMEL

VXLAN Multi-Site Control Plane

#CLMEL

VXLAN Multi-Site

Control Plane Deployment Considerations 

MP-eBGP EVPN only inter-Sites •



Next-hop behavior (VXLAN tunnel termination and reorigination) and loop protection (as-path attribute)

Two main options for underlay and overlay control plane deployment I-E-I (Recommended)

1. • •

E-E-E*

2. •



Intra-Site: IGP (OSPF, IS-IS) as underlay CP, iBGP as overlay CP Inter-Sites: eBGP for both underlay and overlay CPs Intra-Site and Inter-Sites: eBGP for both underlay and overlay CPs

Full mesh of MP-eBGP EVPN adjacencies across sites •

Recommended to deploy a couple of Route-Servers with 3 or more sites

•

RS in a separate AS only perform control plane functions (“eBGP Route-Reflectors”, IETF RFC 7947)

•

RS functions: EVPN routes reflection, next-hop-unchanged, route-target rewrite

*For more information on why eBGP for both underlay and overlay CP is not a good idea: https://learningnetwork.cisco.com/blogs/community_cafe/2017/10/17/the-magic-of-super-spines-and-rfc7938-with-overlays-guest-post #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

68

VXLAN Multi-Site Underlay Control Plane

DC Core

(Layer-3 Unicast)

DCI

….

VTEP

BGW

Fabric

Spine

VXLAN EVPN

VTEP

VTEP

BGW

BGW

Spine

Spine

….

VTEP

BGW

VXLAN EVPN

Site1 VTEP

VTEP

Spine

Site2 VTEP

VTEP

VTEP

#CLMEL

BRKDCN-2035

VTEP

VTEP

VTEP

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

69

VXLAN Multi-Site

Overlay Control Plane (L3 Core) RS

Route Server (eBGP ”Route Reflector”)

DC Core

(Layer-3 Unicast)

DCI

….

VTEP

BGW

Fabric

Spine

iBGP-EVPN VXLAN EVPN

RR

VTEP

VTEP

BGW

BGW

Spine

Spine

….

VTEP

BGW

iBGP-EVPN VXLAN EVPN

RR

Site1 VTEP

VTEP

Spine

Site2 VTEP

VTEP

VTEP

#CLMEL

BRKDCN-2035

VTEP

VTEP

VTEP

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

70

VXLAN Multi-Site

Overlay Control Plane (L3 Core, no RS) eBGP-EVPN

DC Core

(Layer-3 Unicast)

DCI

….

VTEP

BGW

Fabric

Spine

iBGP-EVPN VXLAN EVPN

RR

VTEP

VTEP

BGW

BGW

Spine

Spine

….

VTEP

BGW

iBGP-EVPN VXLAN EVPN

RR

Site1 VTEP

VTEP

Spine

Site2 VTEP

VTEP

VTEP

#CLMEL

BRKDCN-2035

VTEP

VTEP

VTEP

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

71

VXLAN Multi-Site Overlay Control Plane

RS L3VNI: 50001 Route-Target: 65501:50001

L3VNI: 50001 Route-Target: 65502:50001

DC Core

(Layer-3 Unicast)

VRF Tenant1

DCI

VRF Tenant1

….

VIP1 10.1.1.111

VTEP

BGW

Fabric

Spine

iBGP-EVPN VXLAN EVPN

RR

VTEP

BGW

BGW

Spine

Spine

Site1

L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) VTEP

VTEP

VTEP

….

VIP2 10.2.2.222

VTEP

VTEP

Host1 0000.3010.1101 192.168.10.101

Host2 0000.3020.2101 192.168.20.101 #CLMEL

BRKDCN-2035

BGW

iBGP-EVPN VXLAN EVPN

RR

Site2

L2VNI: 30020 (VLAN 20) L3VNI: 50001 (Tenant1) VTEP

VTEP

VTEP

Spine

L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) VTEP

VTEP

Host3 0000.3010.1102 192.168.10.102 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

72

VXLAN Multi-Site

Overlay Control Plane (Site 1) RS L3VNI: 50001 Route-Target: 65501:50001 Type

VRF Tenant1

DCI

….

VIP1 10.1.1.111

VTEP

BGW

Fabric

Spine

(Layer-3 Unicast)

MAC / Length

L2VNI / RT

IP / Length

L3VNI / RT

2

0000.3010.1101/48

30010, 65501:30010

192.168.10.101/32

50001, 65501:50001

2

0000.3020.2101/48

30020, 65501:30020

192.168.20.101/32

2

0000.3010.1102/48

30010, 65501:30010

192.168.10.102/32 50001, 65501:50001 VIP2 VTEP VTEP 10.2.2.222

VTEP

BGW

VXLAN EVPN RR

Spine

Spine

VTEP

RR

Site2

VTEP

VTEP

VTEP

Host1 0000.3010.1101 192.168.10.101

Host2 0000.3020.2101 192.168.20.101 #CLMEL

BRKDCN-2035

VTEP

10.1.1.1

10.1.1.111

BGW

VXLAN EVPN

L2VNI: 30020 (VLAN 20) L3VNI: 50001 (Tenant1)

Next-Hop

VRF 50001, 65501:50001 Tenant1 10.1.1.111

….

BGW

Site1

L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) VTEP

L3VNI: 50001 Route-Target: 65502:50001

DC Core

Spine

L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) VTEP

VTEP

Host3 0000.3010.1102 192.168.10.102 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

73

Seq.

VXLAN Multi-Site

Overlay Control Plane (Site 2) RS L3VNI: 50001 Route-Target: 65502:50001 Type

MAC / Length

(Layer-3 Unicast) L2VNI / RT

2

VRF 30010, 65502:30010 0000.3020.2101/48 Tenant1 30020, 65502:30020

2

0000.3010.1102/48

2

0000.3010.1101/48

DCI

VTEP

BGW

Fabric

30010, 65502:30010 VIP1 10.1.1.111

Spine

….

IP / Length

L3VNI / RT

Next-Hop

192.168.10.101/32

50001, 65502:50001

10.2.2.222

192.168.20.101/32

50001, 65502:50001

10.2.2.1

192.168.10.102/32

50001, 65502:50001

10.2.2.3

VTEP

RR

VRF Tenant1

….

VIP2 10.2.2.222

VTEP

BGW

Spine

Spine

Site1 VTEP

Seq.

BGW

VXLAN EVPN

L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) VTEP

L3VNI: 50001 Route-Target: 65502:50001

DC Core

VTEP

VTEP

Host1 0000.3010.1101 192.168.10.101

Host2 0000.3020.2101 192.168.20.101 #CLMEL

BRKDCN-2035

BGW

VXLAN EVPN RR

Site2

L2VNI: 30020 (VLAN 20) L3VNI: 50001 (Tenant1) VTEP

VTEP

VTEP

Spine

L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) VTEP

VTEP

Host3 0000.3010.1102 192.168.10.102 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

74

VXLAN Multi-Site

Overlay Control Plane (DCI) RS L3VNI: 50001 Route-Target: 65501:50001

L3VNI: 50001 Route-Target: 65502:50001

DC Core

(Layer-3 Unicast)

VRF Tenant1

DCI

VTEP

VRF Tenant1 Type

BGW 2

Fabric

2

Spine

VTEP

L2VNI / RT

L3VNI / RT

0000.3010.1101/48BGW30010, 65599:30010

192.168.10.101/32

BGW 50001, 65599:50001

0000.3020.2101/48

192.168.20.101/32

50001, 65599:50001

30020, 65599:30020

RR

Site1 VTEP

VTEP

IP / Length

VXLAN EVPN Spine30010, 65599:30010 2 0000.3010.1102/48

L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) VTEP

….

VIP1 MAC / Length 10.1.1.111

192.168.10.102/32

VTEP

Host2 0000.3020.2101 192.168.20.101 #CLMEL

10.2.2.222

Site2

BRKDCN-2035

VTEP

VTEP

BGW

10.1.1.111

RR

VTEP

Host1 0000.3010.1101 192.168.10.101

Seq.

VXLAN EVPN 50001, 65599:50001 10.2.2.222 Spine

L2VNI: 30020 (VLAN 20) L3VNI: 50001 (Tenant1) VTEP

….

VIP2 Next-Hop 10.2.2.222

Spine

L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) VTEP

VTEP

Host3 0000.3010.1102 192.168.10.102 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

75

VXLAN Multi-Site Data Plane

#CLMEL

VXLAN Multi-Site Overlay Data Plane

Inter-site VXLAN Data Plane

DC Core

De-capsulation and Re-encapsulation on BGW (L2 or L3 lookup)

(Layer-3 Unicast)

De-capsulation and Re-encapsulation on BGW (L2 or L3 lookup)

DCI

….

Multi-Site VIP1 10.1.1.111

VTEP

BGW

Fabric

Spine

VXLAN EVPN

VTEP

VTEP

BGW

BGW

Spine

Spine

Intra-site VXLAN Data Plane

….

Multi-Site VIP2 10.2.2.222

VTEP

VTEP

BGW

VXLAN EVPN

Site1 VTEP

VTEP

Spine

Site2 VTEP

VTEP

Host1 0000.3010.1101 192.168.10.101

Host2 0000.3020.2101 192.168.20.101 #CLMEL

BRKDCN-2035

VTEP

VTEP

VTEP

Host3 0000.3010.1102 192.168.10.102 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

77

Multi-Site Packet Walk (BUM)

#CLMEL

VXLAN Multi-Site Packet Walk Layer 2 (BUM) – Site 1

Bridge

SIP

DIP

VXLAN

SMAC

DMAC

SIP

DIP

L10

DGROUP

30010

H1-MAC

ALL-F

H1-IP

ALL-255

Traffic is replicated intra-Site

VTEP

Payload

DF 30010

VTEP

BGW11

BGW21

2 VTEP Leaf10

1

Host 1 sends a L2 BUM frame

VXLAN EVPN Site1

VXLAN EVPN Site2

VXLAN EVPN DCI DF 30010

VTEP BGW12

VTEP Leaf20

VTEP BGW22

Baremetal

Baremetal

Host 1 0000.3010.1101 192.168.10.101

Host 2 0000.3010.1102 192.168.10.102 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

79

VXLAN Multi-Site Packet Walk Layer 2 (DF and Split Horizon) – Site 1 Bridge

SIP

DIP

VXLAN

SMAC

DMAC

SIP

DIP

L10

DGROUP

30010

H1-MAC

ALL-F

H1-IP

ALL-255

VTEP

Payload

DF 30010

VTEP

BGW11

VTEP Leaf10

VXLAN EVPN Site1

BGW21

BUM Forward

VXLAN EVPN Site2

VXLAN EVPN DCI DF 30010

VTEP BGW12

VTEP Leaf20

VTEP BGW22

Baremetal

Host 1 0000.3010.1101 192.168.10.101

Baremetal

Drop due to Designated Forwarder (DF) rule

Host 2 0000.3010.1102 192.168.10.102

Drop due to Split-Horizon rule #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

80

VXLAN Multi-Site Packet Walk Layer 2 (BUM) – DCI

Bridge SIP

DIP

VXLAN

SMAC

DMAC

SIP

DIP

BGW11PIP

BGW21

30010

H1-MAC

ALL-F

H1-IP

ALL-255

BGW11PIP

BGW22

30010

H1-MAC

ALL-F

H1-IP

ALL-255

VTEP

DF 30010

BGW11

VTEP Leaf10

VXLAN EVPN Site1

BUM Forward

BGW11 replicates traffic inter-Sites toward BGW nodes

Payload

VTEP BGW21

3

VXLAN EVPN Site2

VXLAN EVPN DCI DF 30010

VTEP BGW12

VTEP Leaf20

VTEP BGW22

Baremetal

Baremetal

Host 1 0000.3010.1101 192.168.10.101

Host 2 0000.3010.1102 192.168.10.102 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

81

VXLAN Multi-Site Packet Walk Layer 2 (DF and Split Horizon) – DCI

Bridge SIP

DIP

VXLAN

SMAC

DMAC

SIP

DIP

BGW11PIP

BGW21

30010

H1-MAC

ALL-F

H1-IP

ALL-255

BGW11PIP

BGW22

30010

H1-MAC

ALL-F

H1-IP

ALL-255

VTEP

DF 30010

VTEP

BGW11

VTEP Leaf10

VXLAN EVPN Site1

Payload

BGW21

BUM Forward

VXLAN EVPN Site2

VXLAN EVPN DCI DF 30010

VTEP BGW12

VTEP Leaf20

VTEP BGW22

BUM Forward Baremetal

Host 1 0000.3010.1101 192.168.10.101

Baremetal

Drop due to Designated Forwarder (DF) rule

Host 2 0000.3010.1102 192.168.10.102

Drop due to Split-Horizon rule #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

82

VXLAN Multi-Site Packet Walk Layer 2 (BUM) – Site 2

Bridge

VTEP

SIP

DIP

VXLAN

SMAC

DMAC

SIP

DIP

BGW22-PIP

DGROUP

30010

H1-MAC

ALL-F

H1-IP

ALL-255

DF 30010

VTEP

BGW11

VTEP Leaf10

VXLAN EVPN Site1

BGW21

4

DF 30010

VTEP BGW12

Traffic is replicated intra-Site VXLAN EVPN Site2

VXLAN EVPN DCI

Payload

VTEP Leaf20

VTEP BGW22

BUM Forward Baremetal

Baremetal

Host 1 0000.3010.1101 192.168.10.101

Host 2 0000.3010.1102 192.168.10.102 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

83

VXLAN Multi-Site Packet Walk Layer 2 (DF and Split Horizon) – Site 2 Bridge

VTEP

SIP

DIP

VXLAN

SMAC

DMAC

SIP

DIP

BGW22-PIP

DGROUP

30010

H1-MAC

ALL-F

H1-IP

ALL-255

DF 30010

VTEP

BGW11

VTEP Leaf10

VXLAN EVPN Site1

Payload

BGW21

VXLAN EVPN Site2

VXLAN EVPN DCI DF 30010

VTEP BGW12

VTEP Leaf20

VTEP BGW22

BUM Forward Baremetal

Host 1 0000.3010.1101 192.168.10.101

Baremetal

Drop due to Designated Forwarder (DF) rule

Host 2 0000.3010.1102 192.168.10.102

Drop due to Split-Horizon rule #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

84

VXLAN Multi-Site Packet Walk Layer 2 (BUM) – Site 2

Bridge

VTEP

DF 30010

VTEP

BGW11

VTEP Leaf10

VXLAN EVPN Site1

BGW21

DF 30010

VTEP BGW12

VTEP

VXLAN EVPN Site2

VXLAN EVPN DCI

Leaf20

VTEP BGW22

5 Leaf20 sends traffic to local Host 2

Baremetal

Host 1 0000.3010.1101 192.168.10.101

Baremetal

Host 2 0000.3010.1102 192.168.10.102 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

85

Multi-Site Packet Walk (Bridging)

#CLMEL

VXLAN Multi-Site Packet Walk Layer 2 (Host 1 to Host 2) – Site 1

Bridge SIP

DIP

VXLAN

SMAC

DMAC

SIP

DIP

L10

BGW-VIP1

30010

H1-MAC

H2-MAC

H1-IP

H2-IP

Leaf10 performs L2 lookup and encapsulates toward local BGW VIP1 address

Payload

VTEP

VTEP

BGW11

BGW21

2 VTEP Leaf10

1

VXLAN EVPN Site1

Host 1 sends traffic destined to remote Host 2

VIP1

VXLAN EVPN DCI

VIP2

VTEP

VTEP

BGW12

BGW22

VXLAN EVPN Site2

VTEP Leaf20

Baremetal

Baremetal

Host 1 0000.3010.1101 192.168.10.101

Host 2 0000.3010.1102 192.168.10.102 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

87

VXLAN Multi-Site Packet Walk Layer 2 (Host 1 to Host 2) – DCI

Bridge SIP

DIP

VXLAN

SMAC

DMAC

SIP

DIP

BGW-VIP1

BGW-VIP2

30010

H1-MAC

H2-MAC

H1-IP

H2-IP

VTEP BGW11

VTEP Leaf10

BGW11 performs L2 lookup and encapsulates toward remote BGW VIP2 address

Payload

VTEP BGW21

3 VXLAN EVPN Site1

VIP1

VXLAN EVPN DCI

VIP2

VTEP

VTEP

BGW12

BGW22

VXLAN EVPN Site2

VTEP Leaf20

Baremetal

Baremetal

Host 1 0000.3010.1101 192.168.10.101

Host 2 0000.3010.1102 192.168.10.102 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

88

VXLAN Multi-Site Packet Walk Layer 2 (Host 1 to Host 2) – Site 2

Bridge

VTEP Leaf10

VXLAN EVPN Site1

SIP

DIP

VXLAN

SMAC

DMAC

SIP

DIP

BGW-VIP2

L20

30010

H1-MAC

H2-MAC

H1-IP

H2-IP

VTEP

VTEP

BGW11

BGW21

VIP1

VXLAN EVPN DCI

VIP2

VTEP

VTEP

BGW12

BGW22

Payload

BGW22 performs L2 lookup and encapsulates toward destination L20 node

4

VTEP

VXLAN EVPN Site2

Leaf20

5 Leaf20 bridges traffic to local Host 2

Baremetal

Baremetal

Host 1 0000.3010.1101 192.168.10.101

Host 2 0000.3010.1102 192.168.10.102 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

89

VXLAN Multi-Site Packet Walk Layer 2 (Host 2 to Host 1) – Site 2

Bridge

VTEP Leaf10

VXLAN EVPN Site1

SIP

DIP

VXLAN

SMAC

DMAC

SIP

DIP

L20

BGW-VIP2

30010

H2-MAC

H1-MAC

H2-IP

H1-IP

VTEP

VTEP

BGW11

BGW21

VIP1

VXLAN EVPN DCI

VIP2

VTEP

VTEP

BGW12

BGW22

Payload

Leaf20 performs L2 lookup and encapsulates toward local BGW VIP2 address

7

VTEP

VXLAN EVPN Site2

Leaf20

6 Host 2 replies to remote Host 1

Baremetal

Baremetal

Host 1 0000.3010.1101 192.168.10.101

Host 2 0000.3010.1102 192.168.10.102 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

90

VXLAN Multi-Site Packet Walk Layer 2 (Host 2 to Host 1) – DCI

Bridge SIP

DIP

VXLAN

SMAC

DMAC

SIP

DIP

BGW-VIP2

BGW-VIP1

30010

H2-MAC

H1-MAC

H2-IP

H1-IP

VTEP BGW11

VTEP Leaf10

BGW21 performs L2 lookup and encapsulates toward remote BGW VIP1 address

Payload

VTEP BGW21

8

VXLAN EVPN Site1

VIP1

VXLAN EVPN DCI

VIP2

VTEP

VTEP

BGW12

BGW22

VXLAN EVPN Site2

VTEP Leaf20

Baremetal

Baremetal

Host 1 0000.3010.1101 192.168.10.101

Host 2 0000.3010.1102 192.168.10.102 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

91

VXLAN Multi-Site Packet Walk Layer 2 (Host 2 to Host 1) – Site 1

Bridge SIP

DIP

VXLAN

SMAC

DMAC

SIP

DIP

BGW-VIP1

L10

30010

H2-MAC

H1-MAC

H2-IP

H1-IP

Payload

BGW12 performs L2 lookup and encapsulates toward destination L10 node VTEP Leaf10

10

VTEP

VTEP

BGW11

BGW21

9 VXLAN EVPN Site1

Leaf10 bridges traffic toward Host 1

VIP1

VXLAN EVPN DCI

VIP2

VTEP

VTEP

BGW12

BGW22

VXLAN EVPN Site2

VTEP Leaf20

Baremetal

Baremetal

Host 1 0000.3010.1101 192.168.10.101

Host 2 0000.3010.1102 192.168.10.102 #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

92

Failure Detection on BGWs

#CLMEL

Anycast BGWs

#CLMEL

VXLAN Multi-Site

Failure Detection on BGWs – Fabric Isolation

Site-Internal

Site-External

 Multi-Site VIP 10.111.111.1 BGW

BGW

BGW

BGW

VTEP

VTEP

VTEP

VTEP

PIP-BGW2 10.200.200.22

PIP-BGW3 10.200.200.23

PIP-BGW4 10.200.200.24

Spine

The Site-Internal interfaces on BGW nodes are constantly tracked to determine their status (‘evpn multisite fabric-tracking’ command)

Spine

Site 1

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

95

VXLAN Multi-Site

Site-Internal

Site-External

Failure Detection on BGWs – Fabric Isolation 

The Site-Internal interfaces on BGW nodes are constantly tracked to determine their status (‘evpn multisite fabric-tracking’ command)



If all the Site-Internal interfaces are detected as down:

Multi-Site VIP 10.111.111.1 BGW

BGW

BGW

BGW

VTEP

VTEP

VTEP

VTEP

PIP-BGW2 10.200.200.22

PIP-BGW3 10.200.200.23

PIP-BGW4 10.200.200.24

Spine

1. 2.

The isolated BGW stops advertising PIP/VIP addresses toward the Site-External network The remaining BGWs perform new DF elections for the L2VNIs owned by the isolated BGW

Spine

Site 1

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

96

VXLAN Multi-Site

Site-Internal

Site-External

Failure Detection on BGWs – Fabric Isolation 

The Site-Internal interfaces on BGW nodes are constantly tracked to determine their status (‘evpn multisite fabric-tracking’ command)



If all the Site-Internal interfaces are detected as down:

Multi-Site VIP 10.111.111.1 BGW

BGW

BGW

BGW

VTEP

VTEP

VTEP

VTEP

PIP-BGW2 10.200.200.22

PIP-BGW3 10.200.200.23

PIP-BGW4 10.200.200.24

Spine

Spine

1. 2.

The isolated BGW stops advertising PIP/VIP addresses toward the Site-External network The remaining BGWs perform new DF elections for the L2VNIs owned by the isolated BGW



As a result, the BGW becomes isolated from both the Site-Internal and Site-External networks



Seamless BGW node reinsertion using a “delayrestore” timer for the VIP address

Site 1

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

97

VXLAN Multi-Site

Failure Detection on BGWs – DCI Isolation

Site-Internal

Site-External

DC Core

(Layer-3 Unicast)

BGW

BGW

BGW

BGW

VTEP

VTEP

VTEP

VTEP

PIP-BGW1 10.200.200.21

PIP-BGW2 10.200.200.22

PIP-BGW3 10.200.200.23

PIP-BGW4 10.200.200.24



The Site-External interfaces on BGW nodes are also tracked to determine their status (‘evpn multisite dci-tracking’ command)



If all the Site-External interfaces are detected as down, the isolated BGW node: 1. 2.

Multi-Site VIP 10.111.111.1

3.

Stops advertising VIP VTEP address toward the Site-Internal network Withdraws BGP EVPN Type-4 advertisements (triggering a new DF election between other BGWs) Starts functioning as a regular VTEP (PIP still up)



As a result, the BGW continues to operate as a Site-Internal VTEP



Seamless BGW node reinsertion using a “delayrestore” timer for the VIP address

Site 1

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

98

vPC BGWs

#CLMEL

NXOS Release 9.2(1)

VXLAN Multi-Site

Failure Detection on vPC BGWs – Fabric Isolation

Site-External

 Multi-Site VIP 10.111.111.1 VPC VIP 10.1.1.121

BGW VTEP

BGW VTEP

PIP-BGW1 10.1.1.101

Site-Internal

The Site-Internal interfaces on BGW nodes are constantly tracked to determine their status (‘evpn multisite fabric-tracking’ command)

PIP-BGW2 10.1.1.102

Spine

Spine

Site 1

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

100

NXOS Release 9.2(1)

VXLAN Multi-Site

Site-External

Failure Detection on vPC BGWs – Fabric Isolation The Site-Internal interfaces on BGW nodes are constantly tracked to determine their status (‘evpn multisite fabric-tracking’ command)



If all the Site-Internal interfaces are detected as down:

Multi-Site VIP 10.111.111.1 VPC VIP 10.1.1.121

BGW VTEP

BGW VTEP

PIP-BGW1 10.1.1.101

Site-Internal



•

PIP-BGW2 10.1.1.102

Spine

The isolated BGW keeps advertising PIP/VIP addresses toward the Site-Internal and SiteExternal network

Spine

Site 1

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

101

NXOS Release 9.2(1)

VXLAN Multi-Site

Site-External

Failure Detection on vPC BGWs – Fabric Isolation The Site-Internal interfaces on BGW nodes are constantly tracked to determine their status (‘evpn multisite fabric-tracking’ command)



If all the Site-Internal interfaces are detected as down:

Multi-Site VIP 10.111.111.1 VPC VIP 10.1.1.121

BGW VTEP

BGW VTEP

PIP-BGW1 10.1.1.101

Site-Internal



•

PIP-BGW2 10.1.1.102

 Spine

Spine

The isolated BGW keeps advertising PIP/VIP addresses toward the Site-Internal and SiteExternal network

As a result, the VPC Peer-Link will be used for Site-External to Site-Internal communication

Site 1

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

102

NXOS Release 9.2(1)

VXLAN Multi-Site

Site-External

Failure Detection on vPC BGWs – Fabric Isolation The Site-Internal interfaces on BGW nodes are constantly tracked to determine their status (‘evpn multisite fabric-tracking’ command)



If all the Site-Internal interfaces are detected as down:

Multi-Site VIP 10.111.111.1 VPC VIP 10.1.1.121

BGW VTEP

ADC

ADC 0000.3010.1102 192.168.10.102

BGW VTEP

PIP-BGW1 10.1.1.101

Site-Internal



•

PIP-BGW2 10.1.1.102

Spine



As a result, the VPC Peer-Link will be used for Site-External to Site-Internal communication



Locally attached Single or Dual-Connected EndPoints stay reachable from Site-External and from Site-Internal via VPC Peer-Link

Spine Baremetal

Site 1

EP 0000.3010.1101 192.168.10.101

The isolated BGW keeps advertising PIP/Multi-Site VIP/VPC VIP addresses toward the Site-Internal and Site-External network (via VPC Peer-Link)

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

103

NXOS Release 9.2(1)

VXLAN Multi-Site

Failure Detection on vPC BGWs – DCI Isolation

Site-External

DC Core

(Layer-3 Unicast)

BGW

VPC VIP 10.1.1.121

VTEP



The Site-External interfaces on BGW nodes are also tracked to determine their status (‘evpn multisite dci-tracking’ command)



If all the Site-External interfaces are detected as down:

BGW VTEP

•

PIP-BGW2 10.1.1.102

Site-Internal

PIP-BGW1 10.1.1.101

Multi-Site VIP 10.111.111.1

•

Site 1



#CLMEL

The isolated BGW keeps advertising PIP/VPC VIP addresses toward the Site-External and SiteInternal network (for External Connectivity and Local Hosts) The isolated BGW stops advertising the Multi-Site VIP address toward the Site-Internal network and Site-External (via VPC Peer-Link)

Seamless BGW node reinsertion by readvertising via VPC Peer-Link (ensure eBGP multi-hop peering) BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

104

NXOS Release 9.2(1)

VXLAN Multi-Site

Failure Detection on vPC BGWs – DCI Isolation DC Core

Site-External

(Layer-3 Unicast)

BGW

VPC VIP 10.1.1.121

VTEP

The Site-External interfaces on BGW nodes are also tracked to determine their status (‘evpn multisite dci-tracking’ command)



If all the Site-External interfaces are detected as down:

BGW VTEP

PIP-BGW1 10.1.1.101

Site-Internal



•

PIP-BGW2 10.1.1.102

Multi-Site VIP 10.111.111.1

• ADC

ADC 0000.3010.1102 192.168.10.102

Baremetal

Site 1

EP 0000.3010.1101 192.168.10.101



#CLMEL

The isolated BGW keeps advertising PIP/VPC VIP addresses toward the Site-External and SiteInternal network (for External Connectivity and Local Hosts) The isolated BGW stops advertising the Multi-Site VIP address toward the Site-Internal network and Site-External (via VPC Peer-Link)

Single or Dual-Connected End-Points stay reachable from Site-External and from SiteInternal via VPC Peer-Link BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

105

NXOS Release 9.2(1)

VXLAN Multi-Site

Failure Detection on vPC BGWs – ZigZag Isolation

Site-External

DC Core

(Layer-3 Unicast)

VPC VIP 10.1.1.121

BGW

BGW VTEP

PIP-BGW1 10.1.1.101

•

PIP-BGW2 10.1.1.102

Multi-Site VIP 10.111.111.1 Spine

If all the Site-External interfaces are detected as down on BGW1 (Leaf-Mode): •

Multi-Site VIP 10.111.111.1

VTEP

Site-Internal





Spine

If all the Site-Internal interfaces are detected as down on BGW2: •

Site 1

#CLMEL

The isolated BGW keeps advertising PIP/VPC VIP addresses toward the Site-External and SiteInternal network (for External Connectivity and Local Hosts) The isolated BGW stops advertising the Multi-Site VIP address toward the Site-Internal network and Site-External (via VPC Peer-Link)

The isolated BGW keeps advertising PIP/VPC VIP addresses toward the Site-External and SiteInternal network (via VPC Peer-Link)

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

106

NXOS Release 9.2(1)

VXLAN Multi-Site

Failure Detection on vPC BGWs – ZigZag Isolation DC Core

Site-External

VPC VIP 10.1.1.121

BGW

BGW VTEP

PIP-BGW1 10.1.1.101

•

PIP-BGW2 10.1.1.102

Multi-Site VIP 10.111.111.1 ADC

ADC 0000.3010.1102 192.168.10.102

Spine

Site 1

If all the Site-External interfaces are detected as down on BGW1 (Leaf-Mode): •

Multi-Site VIP 10.111.111.1

VTEP

Site-Internal



(Layer-3 Unicast)

Spine

 Baremetal

If all the Site-Internal interfaces are detected as down on BGW2: •

EP 0000.3010.1101 192.168.10.101



#CLMEL

The isolated BGW keeps advertising PIP/VPC VIP addresses toward the Site-External and SiteInternal network (for External Connectivity and Local Hosts) The isolated BGW stops advertising the Multi-Site VIP address toward the Site-Internal network and Site-External (via VPC Peer-Link)

The isolated BGW keeps advertising PIP/VPC VIP addresses toward the Site-External and SiteInternal network (via VPC Peer-Link)

Single or Dual-Connected End-Points stay reachable from Site-External and from SiteInternal via VPC Peer-Link BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

107

Connectivity to the External Layer 3 Domain

#CLMEL

VXLAN Multi-Site

Connectivity to the External Layer 3 Domain



The BGW nodes can also be used to provide Layer-3 external connectivity to each site



Different connectivity models are supported • • •



VRF-Lite peering with external WAN Edge routers MP-BGP EVPN peering with external WAN Edge routers (Shared Border deployment model, aka GOLF) Dedicated or shared pair of WAN Edge routers across sites

External Layer-3 network may be different from the DCI network used for inter-site communication

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

109

VXLAN Multi-Site

Border Gateways and VRF-Lite to External Routers

VRF-B VRF-C External Connectivity

VRF-A

Separate IPv4/IPv6 routing peering for each VRF (IGP or eBGP)

Site-Internal

Site-External

Dedicated physical interfaces / subinterfaces for each VRF



Separate IPv4/IPv6 routing peering for each VRF established with the external routers on dedicated physical interfaces/sub-interfaces



Must use separate interfaces for inter-site communication

Multi-Site Overlay

BGW

BGW

BGW

BGW

VTEP

VTEP

VTEP

VTEP



No support for VXLAN encapsulated traffic on sub-interfaces

Site 1

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

110

VXLAN Multi-Site

Border Gateway and Shared Border (aka ‘GOLF’) External router operates like a traditional VXLAN EVPN VTEP (Layer 3 only) VRF-B VRF-C External Connectivity



Single MP-BGP EVPN peering established with the external routers to exchange routes for all the VRFs



VXLAN Data-Plane between the BGWs and the external routers



Same spine uplinks used for all VXLAN encapsulated traffic (NorthSouth and East-West)

VRF-A

Single MP-BGP EVPN routing instance to exchange routes for all VRFs

Routed interface extending ‘underlay’ connectivity to the external routers

Site-Internal

Site-External

Multi-Site Overlay

BGW

BGW

BGW

BGW

VTEP

VTEP

VTEP

VTEP



VXLAN Data Plane between BGW and WAN Edge Router

 Site 1

#CLMEL

Required because of the use of DCI link tracking

Various northbound hand-off options depending on specific HW support: VRF-Lite, MPLS-VPN, LISP BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

111

Legacy Site Integration

#CLMEL

VXLAN Multi-Site Legacy Site Integration

Spine

VTEP

VTEP

VTEP

VTEP

VTEP

VTEP

BGW

BGW

BGW

Spine

VTEP

Baremetal

Spine

VTEP

VTEP

Legacy Site

Coexistence and/or migration use cases •



ADC

VTEP

Greenfield Site



Pair of vPC BGWs (EX/FX Switches)

BGW

Spine

VTEP

IR for BUM + aggregated BUM containment

Extend Layer-2 and Layer-3 multi-tenant connectivity across sites

Deploy a pair of vPC BGWs in the legacy site •

Simplified configuration required on vPC BGW nodes

•

Still offering native Multi-Site functions (Ingress Replication for BUM, BUM containment, etc.) #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

124

Multi-Site and Legacy Site Integration Default Gateway Deployment – Recommended Greenfield VXLAN EVPN Fabric offers L2 and L3 services for the stretched IP subnets Spine

VTEP

VTEP

BGW

BGW

Spine

L3 L2

Spine

VTEP

VTEP

BGW

BGW

Default Gateway migrated to the Border Gateways (VXLAN EVPN Anycast Gateway)

Spine

Legacy infrastructure offers only L2 services L3

VTEP

VTEP

VTEP

VTEP

VTEP

VTEP

VTEP

L2 Distributed Anycast Gateway function

Legacy Site

Greenfield Site



Recommended approach is to migrate the default gateway from the legacy aggregation devices to the Border Gateways (VXLAN EVPN Anycast Gateway)



Optimise routing between endpoints deployed across sites #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

125

Multi-Site and Legacy Site Integration Layer-2 Control Plane Exchange across Sites eBGP-EVPN MAC

NH

0000.3010.1101

Leaf1

0000.3010.1102

VIP2

MAC

VIP1 10.1.1.111 VTEP

VTEP

BGW

BGW

vPC Anycast VTEP

VIP2 10.2.2.222 VTEP

VTEP

BGW

BGW

Po1 Spine

VTEP

VTEP

Spine

VTEP

Spine

VTEP

Spine

VTEP

VTEP

0000.3010.1101

VIP1

0000.3010.1102

Po1

All End-Points in the legacy site are learned as directly connected to the BGW

VTEP

Baremetal

Host 1 0000.3010.1101 192.168.10.101

NH

Baremetal

Greenfield Site

Legacy Site

#CLMEL

BRKDCN-2035

Host 2 0000.3010.1102 192.168.10.102

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

126

Multi-Site and Legacy Site Integration Layer-3 Control Plane Exchange across Sites eBGP-EVPN IP

NH

192.168.10.101

Leaf1

192.168.20.101

VIP2

IP

VIP2 10.2.2.222

VIP1 10.1.1.111 VTEP

VTEP

L3

VTEP

VTEP

BGW

BGW

L2

BGW

BGW

Po1 Spine

VTEP

VTEP

Spine

VTEP

Spine

VTEP

Spine

VTEP

VTEP

192.168.10.101

VIP1

192.168.20.101

Po1

All End-Points in the legacy site are learned as directly connected to the BGWs

VTEP

Baremetal

Host 1 0000.3010.1101 192.168.10.101

L3VNI

Baremetal

Greenfield Site

Legacy Site

#CLMEL

BRKDCN-2035

Host 3 0000.3010.1102 192.168.20.101

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

127

VXLAN Multi-Site and Legacy Site Integration Starting from Legacy Networks Only (1)

Pair of vPC BGWs (EX/FX Switches)

Pair of vPC BGWs (EX/FX Switches)

BGW VTEP

VTEP

Legacy Site 1



VTEP

BGW

Legacy Site 2

A pair of vPC BGWs inserted in each legacy site to extend Layer-2 and Layer-3 connectivity between sites •



VTEP

BGW

Replacement of traditional DCI technologies (EoMPLS, VPLS, OTV, …)

Slowly phase out the legacy networks and replace them with VXLAN EVPN fabrics #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

128

VXLAN Multi-Site and Legacy Site Integration Starting from Legacy Networks Only (2) Convert the nodes to full BGWs functions VTEP

VTEP

VTEP

VTEP

BGW

BGW

BGW

BGW

Spine

VTEP

Spine

VTEP

Spine

VTEP

VTEP

‘Mixed’ Site 1

 

Spine

VTEP

VTEP

‘Mixed’ Site 2

Introduce VXLAN EVPN spines and additional VTEPs in each site Migrate endpoints between the legacy network and the new VXLAN EVPN fabric

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

129

VXLAN Multi-Site and Legacy Site Integration Starting from Legacy Networks Only (3)

VTEP

VTEP

VTEP

VTEP

BGW

BGW

BGW

BGW

Spine

VTEP

VTEP

VTEP

Spine

VTEP

Spine

VTEP

VTEP

VTEP

VTEP

Greenfield Site 1



VTEP

VTEP

Spine

VTEP

VTEP

VTEP

VTEP

Greenfield Site 2

Decommission the legacy networks and leave only the VXLAN EVPN fabrics in place

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

130

VXLAN Multi-Site and Legacy Site Integration Starting from Legacy Networks Only (4)

VTEP

VTEP

VTEP

VTEP

BGW

BGW

BGW

BGW

Spine

VTEP

VTEP

VTEP

Spine

VTEP

Spine

VTEP

VTEP

VTEP

VTEP

Greenfield Site 1



VTEP

VTEP

Spine

VTEP

VTEP

VTEP

VTEP

Greenfield Site 2

Move endpoints directly connected to the vPC BGW nodes (if any) to regular leaf nodes and migrate to the Anycast BGW model 

Anycast BGW is the recommended deployment options



The migration can be done in a non disruptive way, one node at the time #CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

131

Conclusions

#CLMEL

VXLAN EVPN – Multi-Site Multi-Site Core • Border Gateway (BGW) to Border Gateway (BGW) reachability required • Reachability Back-to-Back (full-mesh) or via Layer-3 transport network • Any Routing Protocol for BG reachability No Underlay Extension • IPv4 Unicast Transport (Ingress Replication) VTEP VTEP VTEP VTEP • BGP full-mesh or Route-Server (eBGP ”Route Reflector”) Multi-Site Border Gateway (BGW): for Overlay Control-Plane • Seamless insertion into existing VXLAN EVPN Fabrics (Border Gateways require Nexus 9x00-EX/-FX) • Layer-2 and Layer-3 extension to other Sites • BGP- or VPC-based Border Gateway (BGW) Cluster (up to 4 nodes when using BGP) • All Border Gateways (BGW) are representing a common Anycast VTEP • Failure containment through Broadcast, Unknown Unicast and Layer-2 Multicast limiter (off or rate-based) • Co-Existence with VRF-Lite for External Connectivity • Core and Fabric link tracking Site 1 Site n Spine

VTEP

VTEP

Spine

VTEP

Spine

VTEP

Spine

VTEP

Spine

VTEP

VTEP

VTEP

#CLMEL

VTEP

BRKDCN-2035

Spine

VTEP

Spine

VTEP

Spine

VTEP

VTEP

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

VTEP

156

Multi-Site Advantages – ”The Multiple”  Multiple Overlay Domains – Interconnected and Controlled

•

Scaling and Segregating VXLAN EVPN Networks

 Multiple Overlay Control-Plane Domains – Interconnected and Controlled

•

Limited Overlay Control-Plane Update Propagation

 Multiple Underlay Domains - Isolated

•

Isolated Underlay Domains – No need for Extension

 Multiple Replication Domains for BUM – Interconnected and Controlled

•

Individual BUM flooding domain with Traffic control

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

157

Inter-X Connectivity Multi-Pod

Multi-Fabric

Multi-Site

Underlay Control Plane

Unified Underlay Domain

Separated Underlay Domains

Separated Underlay Domains

Overlay Control Plane

Separated Overlay Control-Plane Domains

Overlay Data Plane

Single Data-Plane

Separated Data-Planes

Separated Data-Planes

BUM Replication in DCI

Unified Underlay Domain (All Multicast or All Ingress Replication)

Dependency on DCI

Choice (Unicast/Multicast)

ARP Flood Suppression (DCI)

yes

yes

yes

Unknown Unicast Flood Suppression (DCI)

no

yes

yes

Broadcast Suppression/Limit (DCI)

no

yes

yes

Layer-2 Loop Prevention

Loop mitigation (Edge Protection)

VPC at Border

Loop mitigation (At DCI)

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

158

Resources •

VXLAN EVPN Multi-Site Design and Deployment White Paper

•

Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide - Configuring VXLAN EVPN Multi-Site

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11739942.html

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7x/vxlan/configuration/guide/b_Cisco_Nexus_9000_Series_NXOS_VXLAN_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NXOS_VXLAN_Configuration_Guide_7x_chapter_01100.html

•

Cisco Live Online - VXLAN BGP EVPN based Multi-POD, Multi-Fabric and Multi-Site - BRKDCN-2035 https://www.ciscolive.com/global/on-demand-library/?search=BRKDCN-2035&showMyInterest=false#/

•

Cisco DCNM 11.1(1) - Multi-Site Domain for VXLAN BGP EVPN Fabrics https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/11_1_1/config_guide/lanfabric/b_dcnm_fabric_lan/cont rol.html#concept_nhz_lfc_yfb

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

159

In Summary…

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

160

Q&A

#CLMEL

#CLMEL

BRKDCN-2035

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

162

Complete Your Online Session Evaluation •

Give us your feedback and receive a complimentary Cisco Live 2019 Power Bank after completing the overall event evaluation and 5 session evaluations.

•

All evaluations can be completed via the Cisco Live Melbourne Mobile App.

•

Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at: https://ciscolive.cisco.com/on-demand-library/

#CLMEL

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

Thank you

#CLMEL

#CLMEL