Having highly available layer-2 uplinks in a service provider has been around for a long time and there have been various ways to do it over the years, multi-chassis lag, stacking, active passive pseudowires, and more. If you use traditional VPLS and dual home a customer edge router you have to account for loop prevention.
We’re going to take a look at another way to handle this. EVPN has been used for multihoming in various environments for years. You might be familiar with EVPN/VxLAN in datacenters and dual attaching servers to an MLAG pair. Well there is another option – EVPN Multihoming – which uses ethernet segment identifiers to multi-home CEs. Think of this like MLAG but in BGP instead and since we’re looking at service providers we’ll use a MPLS dataplane instead of VxLAN.
Some of the advantages:
- Easy Multihoming
- Active/Active forwarding
- High Availability
- Resilient layer-2 overlay
- one control plane – BGP
This design is going to utilize a netElastic vBNG and IP Infusion‘s OcNOS on a mix of edgecore and ufispace platforms to run ISIS-SR w/EVPN on top for layer-2 services.
The service provider core and access network utilizes ISIS-SR for MPLS services. If you need a refresher on Segment Routing there are several other articles on this blog or at stubarea51.net.
IPI-1 and IPI-2 are aggregation routers that aggregate last mile circuits which could be fiber, fixed wireless, or various other technologies. Then EVPN VPLS is overlayed on top of the MPLS network to deliver a layer-2 service from the vBNG to the subscriber equipment, in our case a mikrotik rb4011.
The vBNG then handles subscriber management. It takes the DHCP request from the rb4011 and asks the radius server if this subscriber is authorized. If the subscriber is authorized it hands out an IP address to the client.
The ufispace-q2c, a 9600-72xc, is utilized as an EVPN route reflector. In production IPA often recommends an out of path router reflector running on FRR. Lets verify that our control plane setup.
9600-72xc-1#show bgp l2vpn evpn summary
BGP router identifier 100.127.0.0, local AS number 1016
BGP table version is 10
1 BGP AS-PATH entries
0 BGP community entries
Neighbor V AS MsgRcv MsgSen TblVer InQ OutQ Up/Down State/PfxRcd AD MACIP MCAST ESI PREFIX-ROUTE
100.127.0.2 4 1016 38460 38496 10 0 0 01w4d09h 3 0 2 1 0 0
100.127.0.3 4 1016 38471 38490 10 0 0 01w4d09h 2 0 1 1 0 0
100.127.0.7 4 1016 312 314 10 0 0 02:11:38 2 0 1 1 0 0
Total number of neighbors 3
Total number of Established sessions 39600-72xc-1#show run router bgp
router bgp 1016
neighbor 100.127.0.2 remote-as 1016
neighbor 100.127.0.3 remote-as 1016
neighbor 100.127.0.7 remote-as 1016
neighbor 100.127.0.2 update-source lo
neighbor 100.127.0.3 update-source lo
neighbor 100.127.0.7 update-source lo
!
address-family l2vpn evpn
neighbor 100.127.0.2 activate
neighbor 100.127.0.2 route-reflector-client
neighbor 100.127.0.3 activate
neighbor 100.127.0.3 route-reflector-client
neighbor 100.127.0.7 activate
neighbor 100.127.0.7 route-reflector-client
exit-address-family
!The route-reflector clients are all configured the same way.
ipi-1.lab.jan1.us.ipa.net#show bgp l2vpn evpn summary
BGP router identifier 100.127.0.2, local AS number 1016
BGP table version is 7
1 BGP AS-PATH entries
0 BGP community entries
Neighbor V AS MsgRcv MsgSen TblVer InQ OutQ Up/Down State/PfxRcd AD MACIP MCAST ESI PREFIX-ROUTE
100.127.0.0 4 1016 38501 38466 7 0 0 01w4d10h 4 0 2 2 0 0
Total number of neighbors 1
Total number of Established sessions 1
ipi-2.lab.jan1.us.ipa.net#show bgp l2vpn evpn summary
BGP router identifier 100.127.0.3, local AS number 1016
BGP table version is 9
1 BGP AS-PATH entries
0 BGP community entries
Neighbor V AS MsgRcv MsgSen TblVer InQ OutQ Up/Down State/PfxRcd AD MACIP MCAST ESI PREFIX-ROUTE
100.127.0.0 4 1016 38494 38476 9 0 0 01w4d10h 5 0 3 2 0 0
Total number of neighbors 1
Total number of Established sessions 15916-54XKS#show run router bgp
!
router bgp 1016
neighbor 100.127.0.0 remote-as 1016
neighbor 100.127.0.0 update-source lo
!
address-family l2vpn evpn
neighbor 100.127.0.0 activate
exit-address-family
!Now that the BGP signaling is up and verified we will build the EVPN VPLS and EVPN-Multihoming (EVPN-MH) interface.
5916-54XKS#
!
hardware-profile filter evpn-mpls-mh enable
hardware-profile statistics ingress-acl enable
!
evpn mpls enable
!
evpn mpls multihoming enable
!
mac vrf ORANGE
rd 100.127.0.7:1
route-target both evpn-auto-rt
!
evpn mpls vtep-ip-global 100.127.0.7
!
evpn mpls id 1
host-reachability-protocol evpn-bgp ORANGE
!
interface xe44.10 switchport
encapsulation dot1q 10-50
access-if-evpn
arp-cache disable
nd-cache disable
map vpn-id 1
!Some of the general setup on the OcNOS devices is the same. You first need to enable EVPN to utilize the MPLS dataplane. IMPORTANT this requires a reboot! Make sure you set the VTEP IP as the loopback and then build the mac vrf for the layer2 vpn service and tie it to an EVPN id. Finally to utilize the service you attach it to a switchport as seen on xe44.10 switchport. In this case everything with a vlan id of 10-50 is placed into EVPN vpn-id 1.
5916-54XKS#show evpn mpls
EVPN-MPLS Information
=================
Codes: NW - Network Port
AC - Access Port
(u) - Untagged
VPN-ID EVI-Name EVI-Type Type Interface ESI VLAN DF-Status Src-Addr Dst-Addr
_______________________________________________________________________________________________________________________________
1 ---- L2 NW ---- ---- ---- ---- 100.127.0.7 100.127.0.3
1 ---- L2 NW ---- ---- ---- ---- 100.127.0.7 100.127.0.2
1 ---- -- AC xe44.10 --- Single Homed Port --- ---- ---- ---- ----
5916-54XKS#show evpn mpls mac-table
=========================================================================================================================================
EVPN MPLS MAC Entries
=========================================================================================================================================
VNID Interface VlanId In-VlanId Mac-Addr VTEP-Ip/ESI Type Status MAC move AccessPortDesc
_________________________________________________________________________________________________________________________________________
1 xe44.10 ---- ---- 2cc8.1b80.cebb 100.127.0.7 Dynamic Local ------- 0 -------
1 ---- ---- ---- 2cc8.1b80.cebc 100.127.0.3 Dynamic Remote ------- 0 -------
1 ---- ---- ---- 5254.0006.eb8f 00:00:00:5e:00:00:01:00:00:00 Dynamic Remote ------- 0 -------
1 ---- ---- ---- c4ad.34f4.b81d 100.127.0.2 Dynamic Remote ------- 0 -------
Total number of entries are : 4
5916-54XKS#show bgp l2vpn evpn
BGP table version is 8, local router ID is 100.127.0.7
Status codes: s suppressed, d damped, h history, a add-path, * valid, > best, i - internal,
l - labeled, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
[EVPN route type]:[ESI]:[VNID]:[relevent route informantion]
1 - Ethernet Auto-discovery Route
2 - MAC/IP Route
3 - Inclusive Multicast Route
4 - Ethernet Segment Route
5 - Prefix Route
Network Next Hop Metric LocPrf Weight Path Peer Encap
RD[100.127.0.1:2]
*>i [2]:[00:00:00:5e:00:00:01:00:00:00]:[1]:[48,5254:0006:eb8f]:[0]:[640]
100.127.0.2 0 100 0 i 100.127.0.0 MPLS
*>i [2]:[0]:[1]:[48,c4ad:34f4:b81d]:[0]:[640]
100.127.0.2 0 100 0 i 100.127.0.0 MPLS
*>i [3]:[1]:[32,100.127.0.2]
100.127.0.2 0 100 0 i 100.127.0.0 MPLS
RD[100.127.0.3:1]
*>i [2]:[0]:[1]:[48,2cc8:1b80:cebc]:[0]:[640]
100.127.0.3 0 100 0 i 100.127.0.0 MPLS
*>i [3]:[1]:[32,100.127.0.3]
100.127.0.3 0 100 0 i 100.127.0.0 MPLS
RD[100.127.0.7:1] VRF[ORANGE]:
*> [2]:[0]:[1]:[48,2cc8:1b80:cebb]:[0]:[640]
100.127.0.7 0 100 32768 i ---------- MPLS
* i [2]:[0]:[1]:[48,2cc8:1b80:cebc]:[0]:[640]
100.127.0.3 0 100 0 i 100.127.0.0 MPLS
* i [2]:[00:00:00:5e:00:00:01:00:00:00]:[1]:[48,5254:0006:eb8f]:[0]:[640]
100.127.0.2 0 100 0 i 100.127.0.0 MPLS
* i [2]:[0]:[1]:[48,c4ad:34f4:b81d]:[0]:[640]
100.127.0.2 0 100 0 i 100.127.0.0 MPLS
* i [3]:[1]:[32,100.127.0.2]
100.127.0.2 0 100 0 i 100.127.0.0 MPLS
* i [3]:[1]:[32,100.127.0.3]
100.127.0.3 0 100 0 i 100.127.0.0 MPLS
*> [3]:[1]:[32,100.127.0.7]
100.127.0.7 0 100 32768 i ---------- MPLS
The multihoming side takes a little additional configuration for the access interface.
ipi-1.lab.jan1.us.ipa.net#show run int xe10
!
interface xe10
channel-group 1 mode active
!
ipi-1.lab.jan1.us.ipa.net#show run int po1
!
interface po1
evpn multi-homed system-mac 0000.5e00.0001
!
ipi-1.lab.jan1.us.ipa.net#show run int po1.532
!
interface po1.532 switchport
encapsulation dot1q 532 inner-dot1q 10-50
rewrite pop
access-if-evpn
arp-cache disable
nd-cache disable
map vpn-id 1
!
ipi-2.lab.jan1.us.ipa.net#show run int xe10
!
interface xe10
channel-group 1 mode active
!
ipi-2.lab.jan1.us.ipa.net#show run int po1
!
interface po1
evpn multi-homed system-mac 0000.5e00.0001
!
ipi-2.lab.jan1.us.ipa.net#show run int po1.532
!
interface po1.532 switchport
encapsulation dot1q 532 inner-dot1q 10-50
rewrite pop
access-if-evpn
arp-cache disable
nd-cache disable
map vpn-id 1
!On each PE we need to build the portchannel. This port-channel needs a system-mac defined for the multihoming. Then the criteria to place traffic into the EVPN-MH VPN needs to be defined. In this case it matches doubled tagged traffic. It’s not uncommon on SP networks to utilize double tagged traffic. So we add the outer tag and remove it at the attachment point. This is done with rewrite pop which removes or adds the outer tag.
netelastic-1# show running-config interface
interface 10gei-1/1/0
exit
interface 10gei-1/1/1
exit
interface 10gei-1/1/2
exit
interface 10gei-1/1/2.100
ipv4 address 100.124.0.10 29
dot1q 100
exit
interface 10gei-1/1/3
mtu 9216
exit
interface 10gei-1/1/3.100
ipv4 address 100.124.0.2 29
dot1q 100
exit
interface eth-trunk1
trunk-mode lacp-static
trunk-port 10gei-1/1/0
lacp port-priority 32768
lacp timeout long
exit
trunk-port 10gei-1/1/1
lacp port-priority 32768
lacp timeout long
exit
exit
interface eth-trunk1.532
qinq internal 11 external 532
exitThe netElastic box has a stand LACP channel built and then matches the double tag on the interface defined. This interface will be mapped to BRAS for authentication prior to assigning an IP address.
First we need to build radius profiles so we can utilize them later.
netelastic-1# show running-config radius
radius vendor-id 54268
radius accounting-on enable
radius attribute-usermac-as mac
radius framed-route-delimiter ";"
radius username-override disable
radius authentication group RADIUS_AUTH
server-type ipv4-server
timeout 3
retry-times 3
nas-ip-address 100.127.0.6
master-server 1
algorithm master
dead-time 5
dead-count 10
class-as-car disable
filter-id-type user-acl
server 1 ipv4-address 192.168.0.2 port 1812 key netelastic
exit
radius accounting group RADIUS_ACCT
server-type ipv4-server
timeout 3
retry-times 3
nas-ip-address 100.127.0.6
algorithm master
dead-time 5
dead-count 10
flow-unit byte
align-authenreply disable
server 1 ipv4-address 192.168.0.2 port 1813 key netelastic
exit
radius dmcoa group
server-type ipv4-server
server 1 ipv4-address 192.168.0.2 key netelasticThen we need to build a DHCP server and IP Pool on the netElastic box.
netelastic-1# show running-config ippool
ippool group TEST
gateway-ip 192.168.11.1 gateway-mask 255.255.255.0
lease-time 3600
dns-primary 8.8.8.8
ippool-status unlock
warning-threshold 80
warning-exhaust disable
frame-ip lease manage disable
section start-ip 192.168.11.2 end-ip 192.168.11.254
exit
exit
netelastic-1# show running-config dhcp
dhcp
dhcp enable
relay max-user 128000
relay option82 policy keep
relay option82 format china-tel
relay option82 user-configuration-policy interface
interface eth-trunk1.532
mode server
user-quota 32000
exit
exitFinally, we tie it all together with the BRAS configuration.
netelastic-1# show run bras
bras
l2tp alive-check tcp-port 0
access username-check disable
access password-check disable
domain-name-delimiter @
domain-location after-delimiter
domainname-parse-direction left-to-right
authentication RADIUS_AUTH
authentication-type radius
radius-authentication-group RADIUS_AUTH
user-name-format strip-domain
nas-port-format class1
called-station-id-format class1
nas-port-id-format class1
calling-station-id-format class1
mac-format mac-format-type lower mac-delimiter :
invalid-vlan-tag 0
exit
accounting RADIUS_ACCT
accounting-type radius
accounting-update 600
first-radius-accounting-group RADIUS_ACCT
accounting-start-fail online
accounting-update-fail online
accounting-update-immediately disable
l2tp-accounting vpdn-model
user-name-format strip-domain
nas-port-format class1
called-station-id-format class1
nas-port-id-format class1
calling-station-id-format class1
mac-format mac-format-type lower mac-delimiter :
invalid-vlan-tag 0
exit
authorization TEST_AUTH
authorization-type mix-radius
bind nat-domain-name cgnat_rules
nat-type inside
radius-nat-switch disable
exit
domain IPA_DOMAIN
bind authentication-template RADIUS_AUTH
bind accounting-template RADIUS_ACCT
bind authorization-template TEST_AUTH
vgi vgi1
domain-status unlock
user-routing-distribute disable
user-ipv6-routing-distribute disable
tunnel-domain disable
flow-statistic enable
radius-attribute qos-acl-profile no-exist-policy offline
radius-attribute qos-profile no-exist-avp online
quota-out offline
bind-pool 1 TEST
exit
ipoe template IPOE_TEST
authentication-type ipv4 dhcpv4 option
authentication-type ipv6 dhcpv6 option
dhcp-v4 auth-on-up password-type config config-password 123456
dhcp-v4 auth-on-up username-type mac
dhcp-v4 auth-on-up domain-type pre-domain
dhcp-v6 auth-on-up password-type config config-password 123456
dhcp-v6 auth-on-up username-type mac
dhcp-v6 auth-on-up domain-type option
exit
subscriber-manage
user-group-identify circuit
offline-speed 1000
session abnormal-offline-record 32000
session normal-offline-record 32000
session online-fail-record 32000
exit
vgi-configuration
interface vgi1
exit
exit
vci-configuration
interface eth-trunk1.532
ipoe template IPOE_TEST
max-ipox-session 32000
max-pppox-session 32000
encapsulation multi
pre-domain IPA_DOMAIN
ip-access-type ipv4
exit
exit
exit
This is a very simple setup where the CPE, the rb4011, is authorized by it’s mac address. There are more options. We also are not going to assign speed plans, etc…
Prior to this working end to end we also need to build the freeradius server. This setup uses daloRADIUS. I won’t go into details on the freeradius initial build here.
The rb4011 is setup to request a DHCP address on vlan 11.
When this request hits the netElastic server over the EVPN L2VPN, netElastic will ask the radius server if the user is valid.
14:43:55.817264 IP 100.127.0.6.64001 > 192.168.0.2.radius: RADIUS, Access-Request (1), id: 0x8b length: 202
14:43:55.825564 IP 192.168.0.2.radius > 100.127.0.6.64001: RADIUS, Access-Accept (2), id: 0x8b length: 20
14:43:56.077261 IP 100.127.0.6.64011 > 192.168.0.2.radius-acct: RADIUS, Accounting-Request (4), id: 0x02 length: 119
14:43:56.077270 IP 100.127.0.6.64011 > 192.168.0.2.radius-acct: RADIUS, Accounting-Request (4), id: 0x03 length: 245
14:43:56.083223 IP 192.168.0.2.radius-acct > 100.127.0.6.64011: RADIUS, Accounting-Response (5), id: 0x02 length: 20
14:43:59.437271 IP 100.127.0.6.64011 > 192.168.0.2.radius-acct: RADIUS, Accounting-Request (4), id: 0x03 length: 245If the RADIUS server authenticates the user, which we defined with the mac address earlier, then the rb4011 will get an IP address. You can see the approval in the messages above and vlan11 has an IP address on the previous screen shot.
This lab does not have internet access but we’ll add a static route on the rb4011 to verify reachability through netElastic to the rest of the lab loopbacks.
The 3 lost packet seen in this screenshot were during the failure of IPI-1. Shutting po1 on IPI-1 forced traffic through IPI-2.
Conclusion
EVPN-MH is a great technology to uplink layer-2 services to a BNG for subscriber management and achieve high availability. This concept can continue to be built on by adding a second BNG for BNG resiliency and EVPN-MH on the remote side for high profile sites. We did not cover various other authentication methods and assigning speed packages to the subscriber but these all build off the radius server and tie back to the vBNG operation. netElastic also offers an optional CGNAT engine which can be added to this setup as well as DCHPv6.
