Over the past few years, a number of internet Service Providers (ISPs) have upgraded to 100 Gigabit core networks. Many have found success using Mikrotik on the path to 100Gb core using Layer 2 VPN technology like VPLS. With the proper hardware and design, this solution can scale – to a certain degree.

Here, we explore alternative architectures for ISPs with a need to migrate to a 100 Gigabit core infrastructure without inherent limitations of VPLS. In our example, we focus on BGP EVPN technology with segment routing for transport with free-range routing (FRR) as the route reflector. EVPN supports current and future requirements of large-scale layer 2 VPNs without requiring any major change or upgrade and is somewhat futureproof. EVPN supports various L2VPN Ethernet over MPLS topologies, including Ethernet multipoint (E-LAN), Ethernet P2P (E-Line), and Ethernet rooted-multipoint (E-Tree), and solves problems introduced when migrating to 100G architectures. This protocol was approved by IETF and published in February 2015 which means that it’s been adopted as a mature technology. Please see below for reference

It is worth noting that EVPN came as a direct response to the limitations of VPLS in Datacenters and Service provider environments. Below are some of the limitations of VPLS directly addressed by BGP EVPN:

• multihoming
• redundancy
• multicast optimization
• provisioning simplicity
• flow-based load balancing
• multipathing
• scalability

One of the elements is the subject this blog post; segment routing because it collapsed the need to require rsvp-te,ldp and igp and simplify it to only segment routing and igp. This simplified the deployment and segment routing labels are more predictable and much easier to troubleshoot on day to day operations.

Solution description

Below is the list of hardware and software which we used. All UFI Space devices are running IP Infusion’s Ocnos 6.4.2

1. PE-01 – Ufi Space S9510-28DC-B
2. PE-02 – Ufi Space S9600-32X-R
3. P-01 – Ufi Space S9600-32X-R
4. P-02 – Ufi Space S9600-32X-R
5. Calix E9-2
6. FRR – VM
7. Test Computer

Below is the diagram for this discussion. We made use of FRR to deploy out of path route reflectors, Calix bng as the gateway for subscribers, UFI space as the P,PE routers running OcNOS-SP-6.4.2 and test client. One of the main advantages of using disaggregated networks is their flexibility when integrating and working with other vendors, in this case we have 3 main vendors efficiently delivering services and bringing full protocol stack of bgp evpn with segment routing. This means that Service providers can achieve operational efficiency with a minimal license and hardware costs where you can add functionality as you grow. This also provides you with many hardware options to achieve better results in your design.

In this solution all 4 routers running OcNOS will peer with FRR route reflectors 01 and 02 on address family l2vpn evpn to bring up the control plane.

BGP EVPN Control Plane
Below are the configs for PE-02 and FRR-01, to bring up the control plane. Please note that ISIS plays a critical role as an igp. BGP is used to propagate segment routing labels however we will only focus on the bgp configuration portion.

Once PE-02 and FRR-01 bgp l2vpn evpn is up you can use the same approach on all routers with FRR. We have assumed that at this point you have already completed creating mac vrfs, global vtep, access-if and global evpn configs.

Refer to other IPA blogs for details on the pre-requisites, or start a conversation by contacting us at [email protected]

PE-01

FRR-01
FRR has the most efficient way of peering many vendors I have worked with before as I come from a cisco background, especially bgp listen range. This also can be done using OcNOS software command neighbor IPV4_IBGP_PEER peer- group range 11.11.0.0/16 where 11.11.0.0/16 is your internal loopback range for bgp peering.

Next you want to check if neighbors are accepting prefixes and announcing prefixes.

Caution; if some bgp parameters/capabilities are not negotiated properly routes might be received on FRR and not advertised.

This a is good quick view and we can see about 6 prefixes accepted on the route reflector

BGP EVPN is a very scalable and stable solution – somewhat obvious as it derives from the very mature and well implemented BGP protocol stack.

BGP EVPN uses BGP as the control plane and is broadly implemented in Service Provider and Enterprise environments.

Data plane

Here we use Segment Routing (SR) as the Data Plane transport protocol.

We reserved labels 16000-23999 for global use and we are using ISIS to propagate the labels.

Since SR has been discussed a bit on the IPA website, we will not go into the configuration details and invite you to visit blogs specific to this topic, or reach out to us to start a conversation at [email protected].

We will examine route verification in ISIS and some useful troubleshooting tools to add to your arsenal.

After SR configuration, verify the MPLS Forwarding Table.

Refer to Figure 1 (above) and visualize formation of MPLS between PE-01 (100.127.190.11) and PE-02 (100.127.190.10) and verify that the appropriate labels are attached to the Forwarding Table:

Troubleshooting Tips:
As discussed above, we highly recommend that you always follow the approach of breaking down this service into Data Plane and Control Plane specific configuration and verification.

This helps when troubleshooting BGP EVPN with Segment Routing service.

All of the commands used above indicate which portion of the network has a problem; and to some degree, they help to isolate the problem as well.

If after checking the environment using the above commands you still have your circuits down, move on to debugging commands.

OcNOS provides critical details of how the tunnels are being formed and at which stage the circuit is failing.

Below is one of the captures we did in the lab until it was successfully installed.

Conclusion

BGP L2VPN / EVPN is a highly scalable solution for Service Provider environments. With the capabilities of BGP as its foundation, a properly designed and implemented solution provides both scalability and stability to your network.
Adding the flexibility and cost savings inherent to Disaggregated “White Box” Networking will bring great value to both your operation and your customers.

The key to this White Box solution is the cost vs features that become available to the service provider at a significant reduction in TCO. If you are concerned about “the unknowns” of White Box networking, we encourage you to do a bit of investigating in the top tier vendors of both White Box hardware (EdgeCore, ufiSpace, etc.), Network OS’s (IP Infusion, Pica8, Celestica), and bundled packages offered by a number of Whate Box centric distributors such as EPSGlobal and RocNet. . As a matter of fact, IP Infusion has implemented these technologies for a decades and it has proven to be very stable and reliable.