Having highly available layer-2 uplinks in a service provider has been around for a long time and there have been various ways to do it over the years, multi-chassis lag, stacking, active passive pseudowires, and more. If you use traditional VPLS and dual home a customer edge router you have to account for loop prevention.

We’re going to take a look at another way to handle this. EVPN has been used for multihoming in various environments for years. You might be familiar with EVPN/VxLAN in datacenters and dual attaching servers to an MLAG pair. Well there is another option – EVPN Multihoming – which uses ethernet segment identifiers to multi-home CEs. Think of this like MLAG but in BGP instead and since we’re looking at service providers we’ll use a MPLS dataplane instead of VxLAN.

Some of the advantages:

• Easy Multihoming
• Active/Active forwarding
• High Availability
• Resilient layer-2 overlay
• one control plane – BGP

This design is going to utilize a netElastic vBNG and IP Infusion‘s OcNOS on a mix of edgecore and ufispace platforms to run ISIS-SR w/EVPN on top for layer-2 services.

The service provider core and access network utilizes ISIS-SR for MPLS services. If you need a refresher on Segment Routing there are several other articles on this blog or at stubarea51.net.

IPI-1 and IPI-2 are aggregation routers that aggregate last mile circuits which could be fiber, fixed wireless, or various other technologies. Then EVPN VPLS is overlayed on top of the MPLS network to deliver a layer-2 service from the vBNG to the subscriber equipment, in our case a mikrotik rb4011.

The vBNG then handles subscriber management. It takes the DHCP request from the rb4011 and asks the radius server if this subscriber is authorized. If the subscriber is authorized it hands out an IP address to the client.

The ufispace-q2c, a 9600-72xc, is utilized as an EVPN route reflector. In production IPA often recommends an out of path router reflector running on FRR. Lets verify that our control plane setup.

9600-72xc-1#show bgp l2vpn evpn summary
BGP router identifier 100.127.0.0, local AS number 1016
BGP table version is 10
1 BGP AS-PATH entries
0 BGP community entries

Neighbor                 V   AS   MsgRcv    MsgSen TblVer   InQ   OutQ    Up/Down   State/PfxRcd     AD  MACIP  MCAST    ESI  PREFIX-ROUTE
100.127.0.2              4  1016 38460      38496      10      0      0  01w4d09h               3      0      2      1      0      0
100.127.0.3              4  1016 38471      38490      10      0      0  01w4d09h               2      0      1      1      0      0
100.127.0.7              4  1016  312        314      10      0      0  02:11:38               2      0      1      1      0      0

Total number of neighbors 3

Total number of Established sessions 3

9600-72xc-1#show run router bgp
router bgp 1016
 neighbor 100.127.0.2 remote-as 1016
 neighbor 100.127.0.3 remote-as 1016
 neighbor 100.127.0.7 remote-as 1016
 neighbor 100.127.0.2 update-source lo
 neighbor 100.127.0.3 update-source lo
 neighbor 100.127.0.7 update-source lo
 !
 address-family l2vpn evpn
 neighbor 100.127.0.2 activate
 neighbor 100.127.0.2 route-reflector-client
 neighbor 100.127.0.3 activate
 neighbor 100.127.0.3 route-reflector-client
 neighbor 100.127.0.7 activate
 neighbor 100.127.0.7 route-reflector-client
 exit-address-family
 !

The route-reflector clients are all configured the same way.

ipi-1.lab.jan1.us.ipa.net#show bgp l2vpn evpn summary 
BGP router identifier 100.127.0.2, local AS number 1016
BGP table version is 7
1 BGP AS-PATH entries
0 BGP community entries

Neighbor                 V   AS   MsgRcv    MsgSen TblVer   InQ   OutQ    Up/Down   State/PfxRcd     AD  MACIP  MCAST    ESI  PREFIX-ROUTE
100.127.0.0              4  1016 38501      38466       7      0      0  01w4d10h               4      0      2      2      0      0

Total number of neighbors 1

Total number of Established sessions 1

ipi-2.lab.jan1.us.ipa.net#show bgp l2vpn evpn summary 
BGP router identifier 100.127.0.3, local AS number 1016
BGP table version is 9
1 BGP AS-PATH entries
0 BGP community entries

Neighbor                 V   AS   MsgRcv    MsgSen TblVer   InQ   OutQ    Up/Down   State/PfxRcd     AD  MACIP  MCAST    ESI  PREFIX-ROUTE
100.127.0.0              4  1016 38494      38476       9      0      0  01w4d10h               5      0      3      2      0      0

Total number of neighbors 1

Total number of Established sessions 1

5916-54XKS#show run router bgp
!
router bgp 1016
 neighbor 100.127.0.0 remote-as 1016
 neighbor 100.127.0.0 update-source lo
 !
 address-family l2vpn evpn
 neighbor 100.127.0.0 activate
 exit-address-family
!

Now that the BGP signaling is up and verified we will build the EVPN VPLS and EVPN-Multihoming (EVPN-MH) interface.

5916-54XKS#
!
hardware-profile filter evpn-mpls-mh enable
hardware-profile statistics ingress-acl enable
!
evpn mpls enable
!
evpn mpls multihoming enable
!
mac vrf ORANGE
 rd 100.127.0.7:1
 route-target both evpn-auto-rt
!
evpn mpls vtep-ip-global 100.127.0.7
!
evpn mpls id 1
 host-reachability-protocol evpn-bgp ORANGE
!
interface xe44.10 switchport
 encapsulation dot1q 10-50
 access-if-evpn
  arp-cache disable
  nd-cache disable
  map vpn-id 1
!

Some of the general setup on the OcNOS devices is the same. You first need to enable EVPN to utilize the MPLS dataplane. IMPORTANT this requires a reboot! Make sure you set the VTEP IP as the loopback and then build the mac vrf for the layer2 vpn service and tie it to an EVPN id. Finally to utilize the service you attach it to a switchport as seen on xe44.10 switchport. In this case everything with a vlan id of 10-50 is placed into EVPN vpn-id 1.

5916-54XKS#show evpn mpls 
EVPN-MPLS Information 
================= 
   Codes: NW - Network Port 
          AC - Access Port 
         (u) - Untagged 

VPN-ID   EVI-Name     EVI-Type Type Interface ESI                           VLAN      DF-Status Src-Addr         Dst-Addr        
_______________________________________________________________________________________________________________________________
1        ----         L2       NW   ----      ----                          ----      ----      100.127.0.7      100.127.0.3     
1        ----         L2       NW   ----      ----                          ----      ----      100.127.0.7      100.127.0.2     
1        ----         --       AC   xe44.10   --- Single Homed Port ---     ----      ----      ----             ----            

5916-54XKS#show evpn mpls mac-table 
=========================================================================================================================================
                                                     EVPN MPLS MAC Entries 
=========================================================================================================================================
VNID       Interface VlanId    In-VlanId Mac-Addr       VTEP-Ip/ESI                    Type            Status     MAC move AccessPortDesc
_________________________________________________________________________________________________________________________________________

1          xe44.10   ----      ----      2cc8.1b80.cebb 100.127.0.7                    Dynamic Local   -------    0        -------   
1          ----      ----      ----      2cc8.1b80.cebc 100.127.0.3                    Dynamic Remote  -------    0        -------   
1          ----      ----      ----      5254.0006.eb8f 00:00:00:5e:00:00:01:00:00:00  Dynamic Remote  -------    0        -------   
1          ----      ----      ----      c4ad.34f4.b81d 100.127.0.2                    Dynamic Remote  -------    0        -------   

Total number of entries are : 4
5916-54XKS#show bgp l2vpn evpn 
BGP table version is 8, local router ID is 100.127.0.7
Status codes: s suppressed, d damped, h history, a add-path, * valid, > best, i - internal,
              l - labeled, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

[EVPN route type]:[ESI]:[VNID]:[relevent route informantion]
1 - Ethernet Auto-discovery Route
2 - MAC/IP Route
3 - Inclusive Multicast Route
4 - Ethernet Segment Route
5 - Prefix Route

    Network          Next Hop            Metric    LocPrf       Weight     Path  Peer          Encap

RD[100.127.0.1:2]
*>i  [2]:[00:00:00:5e:00:00:01:00:00:00]:[1]:[48,5254:0006:eb8f]:[0]:[640]
                       100.127.0.2          0        100       0    i  100.127.0.0     MPLS      
*>i  [2]:[0]:[1]:[48,c4ad:34f4:b81d]:[0]:[640]
                       100.127.0.2          0        100       0    i  100.127.0.0     MPLS      
*>i  [3]:[1]:[32,100.127.0.2]
                       100.127.0.2          0        100       0    i  100.127.0.0     MPLS      

RD[100.127.0.3:1]
*>i  [2]:[0]:[1]:[48,2cc8:1b80:cebc]:[0]:[640]
                       100.127.0.3          0        100       0    i  100.127.0.0     MPLS      
*>i  [3]:[1]:[32,100.127.0.3]
                       100.127.0.3          0        100       0    i  100.127.0.0     MPLS      

RD[100.127.0.7:1] VRF[ORANGE]:
*>   [2]:[0]:[1]:[48,2cc8:1b80:cebb]:[0]:[640]
                       100.127.0.7          0        100       32768  i  ----------      MPLS      
* i  [2]:[0]:[1]:[48,2cc8:1b80:cebc]:[0]:[640]
                       100.127.0.3          0        100       0    i  100.127.0.0     MPLS      
* i  [2]:[00:00:00:5e:00:00:01:00:00:00]:[1]:[48,5254:0006:eb8f]:[0]:[640]
                       100.127.0.2          0        100       0    i  100.127.0.0     MPLS      
* i  [2]:[0]:[1]:[48,c4ad:34f4:b81d]:[0]:[640]
                       100.127.0.2          0        100       0    i  100.127.0.0     MPLS      
* i  [3]:[1]:[32,100.127.0.2]
                       100.127.0.2          0        100       0    i  100.127.0.0     MPLS      
* i  [3]:[1]:[32,100.127.0.3]
                       100.127.0.3          0        100       0    i  100.127.0.0     MPLS      
*>   [3]:[1]:[32,100.127.0.7]
                       100.127.0.7          0        100       32768  i  ----------      MPLS      

The multihoming side takes a little additional configuration for the access interface.

ipi-1.lab.jan1.us.ipa.net#show run int xe10
!
interface xe10
 channel-group 1 mode active
!
ipi-1.lab.jan1.us.ipa.net#show run int po1
!
interface po1
 evpn multi-homed system-mac 0000.5e00.0001
!
ipi-1.lab.jan1.us.ipa.net#show run int po1.532
!
interface po1.532 switchport
 encapsulation dot1q 532 inner-dot1q 10-50
 rewrite pop
 access-if-evpn
  arp-cache disable
  nd-cache disable
  map vpn-id 1
!
ipi-2.lab.jan1.us.ipa.net#show run int xe10
!
interface xe10
 channel-group 1 mode active
!
ipi-2.lab.jan1.us.ipa.net#show run int po1
!
interface po1
 evpn multi-homed system-mac 0000.5e00.0001
!
ipi-2.lab.jan1.us.ipa.net#show run int po1.532
!
interface po1.532 switchport
 encapsulation dot1q 532 inner-dot1q 10-50
 rewrite pop
 access-if-evpn
  arp-cache disable
  nd-cache disable
  map vpn-id 1
!

On each PE we need to build the portchannel. This port-channel needs a system-mac defined for the multihoming. Then the criteria to place traffic into the EVPN-MH VPN needs to be defined. In this case it matches doubled tagged traffic. It’s not uncommon on SP networks to utilize double tagged traffic. So we add the outer tag and remove it at the attachment point. This is done with rewrite pop which removes or adds the outer tag.

netelastic-1# show running-config interface 
interface 10gei-1/1/0
exit
interface 10gei-1/1/1
exit
interface 10gei-1/1/2
exit
interface 10gei-1/1/2.100
 ipv4 address 100.124.0.10 29
 dot1q 100
exit
interface 10gei-1/1/3
 mtu 9216
exit
interface 10gei-1/1/3.100
 ipv4 address 100.124.0.2 29
 dot1q 100
exit
interface eth-trunk1
 trunk-mode lacp-static
 trunk-port 10gei-1/1/0
  lacp port-priority 32768
  lacp timeout  long
 exit
 trunk-port 10gei-1/1/1
  lacp port-priority 32768
  lacp timeout  long
 exit
exit
interface eth-trunk1.532
 qinq internal 11 external 532
exit

The netElastic box has a stand LACP channel built and then matches the double tag on the interface defined. This interface will be mapped to BRAS for authentication prior to assigning an IP address.

First we need to build radius profiles so we can utilize them later.

netelastic-1# show running-config radius 
radius vendor-id       54268
radius accounting-on   enable
radius attribute-usermac-as mac
radius framed-route-delimiter ";"
radius username-override disable
radius authentication group RADIUS_AUTH
 server-type    ipv4-server
 timeout        3
 retry-times    3
 nas-ip-address 100.127.0.6
 master-server  1
 algorithm      master
 dead-time      5
 dead-count     10
 class-as-car   disable
 filter-id-type user-acl
 server 1 ipv4-address 192.168.0.2 port 1812 key netelastic
exit
radius accounting group RADIUS_ACCT
 server-type       ipv4-server
 timeout           3
 retry-times       3
 nas-ip-address    100.127.0.6
 algorithm         master
 dead-time         5
 dead-count        10
 flow-unit         byte
 align-authenreply disable
 server 1 ipv4-address 192.168.0.2 port 1813 key netelastic
exit
radius dmcoa group
 server-type ipv4-server
 server 1 ipv4-address 192.168.0.2 key netelastic

Then we need to build a DHCP server and IP Pool on the netElastic box.

netelastic-1# show running-config ippool
ippool group TEST
 gateway-ip 192.168.11.1 gateway-mask 255.255.255.0
 lease-time        3600
 dns-primary 8.8.8.8
 ippool-status     unlock
 warning-threshold 80
 warning-exhaust   disable
 frame-ip lease manage disable
 section start-ip 192.168.11.2 end-ip 192.168.11.254
 exit
exit
netelastic-1# show running-config dhcp  
dhcp
 dhcp enable
 relay max-user 128000
 relay option82 policy     keep
 relay option82 format     china-tel
 relay option82 user-configuration-policy interface
 interface eth-trunk1.532
  mode       server
  user-quota 32000
 exit
exit

Finally, we tie it all together with the BRAS configuration.

netelastic-1# show run bras
bras
 l2tp alive-check tcp-port 0
 access username-check disable
 access password-check disable
 domain-name-delimiter      @
 domain-location            after-delimiter
 domainname-parse-direction left-to-right
 authentication RADIUS_AUTH
  authentication-type         radius
  radius-authentication-group RADIUS_AUTH
  user-name-format            strip-domain
  nas-port-format             class1
  called-station-id-format    class1
  nas-port-id-format class1
  calling-station-id-format class1
  mac-format mac-format-type lower mac-delimiter :
  invalid-vlan-tag            0
 exit
 accounting RADIUS_ACCT
  accounting-type               radius
  accounting-update             600
  first-radius-accounting-group RADIUS_ACCT
  accounting-start-fail         online
  accounting-update-fail online
  accounting-update-immediately disable
  l2tp-accounting               vpdn-model
  user-name-format              strip-domain
  nas-port-format               class1
  called-station-id-format      class1
  nas-port-id-format class1
  calling-station-id-format class1
  mac-format mac-format-type lower mac-delimiter :
  invalid-vlan-tag              0
 exit
 authorization TEST_AUTH
  authorization-type mix-radius
  bind nat-domain-name cgnat_rules
  nat-type           inside
  radius-nat-switch  disable
 exit
 domain IPA_DOMAIN
  bind authentication-template RADIUS_AUTH
  bind accounting-template RADIUS_ACCT
  bind authorization-template TEST_AUTH
  vgi                          vgi1
  domain-status                unlock
  user-routing-distribute      disable
  user-ipv6-routing-distribute disable
  tunnel-domain                disable
  flow-statistic               enable
  radius-attribute qos-acl-profile no-exist-policy offline
  radius-attribute qos-profile no-exist-avp online
  quota-out offline
  bind-pool 1 TEST
 exit
 ipoe template IPOE_TEST
  authentication-type ipv4 dhcpv4 option
  authentication-type ipv6 dhcpv6 option
  dhcp-v4 auth-on-up password-type config config-password 123456
  dhcp-v4 auth-on-up username-type mac
  dhcp-v4 auth-on-up domain-type pre-domain
  dhcp-v6 auth-on-up password-type config config-password 123456
  dhcp-v6 auth-on-up username-type mac
  dhcp-v6 auth-on-up domain-type option
 exit
 subscriber-manage
  user-group-identify circuit
  offline-speed 1000
  session abnormal-offline-record 32000
  session normal-offline-record 32000
  session online-fail-record 32000
 exit
 vgi-configuration
  interface vgi1
  exit
 exit
 vci-configuration
  interface eth-trunk1.532
   ipoe template IPOE_TEST
   max-ipox-session  32000
   max-pppox-session 32000
   encapsulation     multi
   pre-domain        IPA_DOMAIN
   ip-access-type    ipv4
  exit
 exit
exit

This is a very simple setup where the CPE, the rb4011, is authorized by it’s mac address. There are more options. We also are not going to assign speed plans, etc…

Prior to this working end to end we also need to build the freeradius server. This setup uses daloRADIUS. I won’t go into details on the freeradius initial build here.

The rb4011 is setup to request a DHCP address on vlan 11.

When this request hits the netElastic server over the EVPN L2VPN, netElastic will ask the radius server if the user is valid.

14:43:55.817264 IP 100.127.0.6.64001 > 192.168.0.2.radius: RADIUS, Access-Request (1), id: 0x8b length: 202
14:43:55.825564 IP 192.168.0.2.radius > 100.127.0.6.64001: RADIUS, Access-Accept (2), id: 0x8b length: 20
14:43:56.077261 IP 100.127.0.6.64011 > 192.168.0.2.radius-acct: RADIUS, Accounting-Request (4), id: 0x02 length: 119
14:43:56.077270 IP 100.127.0.6.64011 > 192.168.0.2.radius-acct: RADIUS, Accounting-Request (4), id: 0x03 length: 245
14:43:56.083223 IP 192.168.0.2.radius-acct > 100.127.0.6.64011: RADIUS, Accounting-Response (5), id: 0x02 length: 20
14:43:59.437271 IP 100.127.0.6.64011 > 192.168.0.2.radius-acct: RADIUS, Accounting-Request (4), id: 0x03 length: 245

If the RADIUS server authenticates the user, which we defined with the mac address earlier, then the rb4011 will get an IP address. You can see the approval in the messages above and vlan11 has an IP address on the previous screen shot.

This lab does not have internet access but we’ll add a static route on the rb4011 to verify reachability through netElastic to the rest of the lab loopbacks.

The 3 lost packet seen in this screenshot were during the failure of IPI-1. Shutting po1 on IPI-1 forced traffic through IPI-2.

Conclusion

EVPN-MH is a great technology to uplink layer-2 services to a BNG for subscriber management and achieve high availability. This concept can continue to be built on by adding a second BNG for BNG resiliency and EVPN-MH on the remote side for high profile sites. We did not cover various other authentication methods and assigning speed packages to the subscriber but these all build off the radius server and tie back to the vBNG operation. netElastic also offers an optional CGNAT engine which can be added to this setup as well as DCHPv6.