Building 100Gig Core with EVPN and Segment routing

Building 100Gig Core with EVPN and Segment routing

by | Mar 20, 2024

Over the past few years, a number of internet Service Providers (ISPs) have upgraded to 100 Gigabit core networks. Many have found success using Mikrotik on the path to 100Gb core using Layer 2 VPN technology like VPLS. With the proper hardware and design, this solution can scale – to a certain degree.

Here, we explore alternative architectures for ISPs with a need to migrate to a 100 Gigabit core infrastructure without inherent limitations of VPLS. In our example, we focus on BGP EVPN technology with segment routing for transport with free-range routing (FRR) as the route reflector. EVPN supports current and future requirements of large-scale layer 2 VPNs without requiring any major change or upgrade and is somewhat futureproof. EVPN supports various L2VPN Ethernet over MPLS topologies, including Ethernet multipoint (E-LAN), Ethernet P2P (E-Line), and Ethernet rooted-multipoint (E-Tree), and solves problems introduced when migrating to 100G architectures. This protocol was approved by IETF and published in February 2015 which means that it’s been adopted as a mature technology. Please see below for reference

RFC 7432: BGP MPLS-Based Ethernet VPN

It is worth noting that EVPN came as a direct response to the limitations of VPLS in Datacenters and Service provider environments. Below are some of the limitations of VPLS directly addressed by BGP EVPN:

  • multihoming
  • redundancy
  • multicast optimization
  • provisioning simplicity
  • flow-based load balancing
  • multipathing
  • scalability

One of the elements is the subject this blog post; segment routing because it collapsed the need to require rsvp-te,ldp and igp and simplify it to only segment routing and igp. This simplified the deployment and segment routing labels are more predictable and much easier to troubleshoot on day to day operations.

Solution description

Below is the list of hardware and software which we used. All UFI Space devices are running IP Infusion’s Ocnos 6.4.2

  1. PE-01 – Ufi Space S9510-28DC-B
  2. PE-02 – Ufi Space S9600-32X-R
  3. P-01 – Ufi Space S9600-32X-R
  4. P-02 – Ufi Space S9600-32X-R
  5. Calix E9-2
  6. FRR – VM
  7. Test Computer

Below is the diagram for this discussion. We made use of FRR to deploy out of path route reflectors, Calix bng as the gateway for subscribers, UFI space as the P,PE routers running OcNOS-SP-6.4.2 and test client. One of the main advantages of using disaggregated networks is their flexibility when integrating and working with other vendors, in this case we have 3 main vendors efficiently delivering services and bringing full protocol stack of bgp evpn with segment routing. This means that Service providers can achieve operational efficiency with a minimal license and hardware costs where you can add functionality as you grow. This also provides you with many hardware options to achieve better results in your design.

In this solution all 4 routers running OcNOS will peer with FRR route reflectors 01 and 02 on address family l2vpn evpn to bring up the control plane.

BGP EVPN Control Plane
Below are the configs for PE-02 and FRR-01, to bring up the control plane. Please note that ISIS plays a critical role as an igp. BGP is used to propagate segment routing labels however we will only focus on the bgp configuration portion.

Once PE-02 and FRR-01 bgp l2vpn evpn is up you can use the same approach on all routers with FRR. We have assumed that at this point you have already completed creating mac vrfs, global vtep, access-if and global evpn configs.

Refer to other IPA blogs for details on the pre-requisites, or start a conversation by contacting us at [email protected]


FRR has the most efficient way of peering many vendors I have worked with before as I come from a cisco background, especially bgp listen range. This also can be done using OcNOS software command neighbor IPV4_IBGP_PEER peer- group range where is your internal loopback range for bgp peering.

Once above configs are set for all 4 routers and RR we will verify the control plane. First you want to check if peering at evpn address family level is up

Next you want to check if neighbors are accepting prefixes and announcing prefixes.

Caution; if some bgp parameters/capabilities are not negotiated properly routes might be received on FRR and not advertised.

This a is good quick view and we can see about 6 prefixes accepted on the route reflector

Before we go into details on confirming routes you might need a refresh on route types and their roles below so that you can understand the prefixes you are receiving and advertising. Route Types [1] Ethernet Auto-Discovery (AD) Route [2] MAC/IP Advertisement Route [3] Inclusive Multicast Route [4] Ethernet Segment Route [5] IP Prefix Advertisement Route New SAFI [70] Routes serve control plane purposes, including: MAC address reachability MAC mass withdrawal Split-Horizon label adv. Aliasing Multicast endpoint discovery Redundancy group discovery Designated forwarder election IP address reachability L2/L3 Integration For FRR the command that I like most is to check advertised routes using JSON data presentation as below. It gives you a very structured way to view the routes and their attributes. Although this is not a scalable view of the data, especially when working in large scale service provider environments you might need to filter to the specifics of what you are actually looking for.
You can also check routes coming from a specific neighbor as shown below on FRR
OcNOS access nodes verification PE-01, I want to check if im receiving remote routes and are they best in the bgp routing table.

BGP EVPN is a very scalable and stable solution – somewhat obvious as it derives from the very mature and well implemented BGP protocol stack.

BGP EVPN uses BGP as the control plane and is broadly implemented in Service Provider and Enterprise environments.

Data plane

Here we use Segment Routing (SR) as the Data Plane transport protocol.

We reserved labels 16000-23999 for global use and we are using ISIS to propagate the labels.

Since SR has been discussed a bit on the IPA website, we will not go into the configuration details and invite you to visit blogs specific to this topic, or reach out to us to start a conversation at [email protected].

We will examine route verification in ISIS and some useful troubleshooting tools to add to your arsenal.

After SR configuration, verify the MPLS Forwarding Table.

Refer to Figure 1 (above) and visualize formation of MPLS between PE-01 ( and PE-02 ( and verify that the appropriate labels are attached to the Forwarding Table:

To confirm end to end data path using mpls label is up I suggest using mpls pings as below
Assuming all of the above is working as expected, your next likely action will be to verify that all MPLS tunnels are UP and allocated the proper lables. The below commands are useful for this. In the lab, we used xconnect and general EVPN tunnel specific to xconnect tunnel label
specific to xconnect tunnel label
and at this point check the status of xconnect
For the final diagnostic step, verify that the routers are learning remote MAC addresses on the Data Plane. We already saw BGP carrying MAC addresses across the sites, thus we can confirm that we are able to see remote MAC address on data plane:

Troubleshooting Tips:
As discussed above, we highly recommend that you always follow the approach of breaking down this service into Data Plane and Control Plane specific configuration and verification.

This helps when troubleshooting BGP EVPN with Segment Routing service.

All of the commands used above indicate which portion of the network has a problem; and to some degree, they help to isolate the problem as well.

If after checking the environment using the above commands you still have your circuits down, move on to debugging commands.

OcNOS provides critical details of how the tunnels are being formed and at which stage the circuit is failing.

Below is one of the captures we did in the lab until it was successfully installed.

below are the last stages as the tunnel transition to installed


BGP L2VPN / EVPN is a highly scalable solution for Service Provider environments. With the capabilities of BGP as its foundation, a properly designed and implemented solution provides both scalability and stability to your network.
Adding the flexibility and cost savings inherent to Disaggregated “White Box” Networking will bring great value to both your operation and your customers.

The key to this White Box solution is the cost vs features that become available to the service provider at a significant reduction in TCO. If you are concerned about “the unknowns” of White Box networking, we encourage you to do a bit of investigating in the top tier vendors of both White Box hardware (EdgeCore, ufiSpace, etc.), Network OS’s (IP Infusion, Pica8, Celestica), and bundled packages offered by a number of Whate Box centric distributors such as EPSGlobal and RocNet. . As a matter of fact, IP Infusion has implemented these technologies for a decades and it has proven to be very stable and reliable.