We knew it was coming when MikroTik announced the end of production for Tilera chips back in the fall of 2022, but recently (looks like it was last week), MikroTik moved almost all of the CCR1K series (CCR1009, CCR1016, CCR1036) to discontinued officially on the website.
The only CCR1K series that are still listed for sale are:
CCR1009-7G-1C-1S+ CCR1016-12S-1S+ CCR1072-1G-8S+
This likely indicates low or no stock to replenish distributors for the discontinued models, so the CCR1K market is going to get even tighter. Expect the three models still listed to go discontinued in the next 60 to 90 days if not sooner.
It’s a bit of a double-edged sword as operators that haven’t made the jump to ROSv7 will quickly get forced into CCR2K models that only support ROSv7 and while MikroTik has done an awesome job of closing the gap, there is still some work to achieve feature parity with ROSv6.
That said, the upside is that the massive rush to CCR2K that has been happening will help to refine ROSv7 with more bug fixes and improved stability due to the larger user base running it and reporting bugs.
CCR2K series
A worthy line of successors
Luckily we’ve seen rapid development on ROSv7 and the CCR2k line which is arm64 based with several models offering L3 hardware offload using Marvell Prestera chips. arm64
MikroTik has introduced several arm64 chips with the introduction of the CCR2k line. most notably, chips from annapurna labs which is owned by Amazon.
The CCR2116/CCR2216 models are outfitted with a 16 core Annapurna CPU that can move up to 200 Gbps of traffic.
Marvell Prestera
The Marvell series of ASICs has been incredibly successful for MikroTik and other vendors at providing a rich feature set for L2/L3 while maintaining a relatively low price point.
ROSv7
ROSv7 has been on fire this year (in a good way) as far as development is concerned and releases are coming in a timely manner with significant work put into each one fixing bugs. By the end of the year we’ll likely see a long-term release and ROSv6 start to wane in popularity.
New hardware
Tilera created some issues in what MikroTik was able to accomplish because it never became more than a niche CPU architecture. So form factors were limited to what Tilera was able to handle.
Now that arm64 & marvell form the base of CPU/ASIC architecture for MikroTik, the possibilities are broader and there are more hardware platforms on the way.
2023 has been a big year for MikroTik RouterOS v7 development. The OS has matured quite a bit and the number of operators that are using ROSv7 in prod has gone up significantly.
Most of the work this year has been centered around feature parity with ROSv6 and performance improvements.
Although there have been a few new additions like the Back to Home VPN (Which is now scheduled to release with 7.12) and IS-IS (Also slated for 7.12).
This is important because most of the development time by MikroTik has been consumed by stabilization, feature parity and perf improvements. However, as that gap closes, we should see long awaited features like SR-MPLS and EVPN make it into ROS v7 faster as development time frees up.
7.11 contains a long list of fixes and improvements and will likely be adopted quickly by the MikroTik community.
*) bfd – fixed “actual-tx-interval” value and added “remote-min-tx” (CLI only); *) bfd – improved system stability;
Bi-directional forwarding detection or BFD has been a feature gap in ROSv7 for a long time and was a showstopper for some ASNs that wanted to use the new CCR2K hardware as peering/border routers. Now that BFD has been implemented for BGP and OSPF with some stabilization fixes, it’s much easier to deploy a CCR2116 or CCR2216 in that role.
IPv6 support for containers
*) container – added IPv6 support for VETH interface;
Being able to use IPv6 for containers is a great addition given the rapid pace of IPv6 adoption in 2023. It also makes it easier to expose containers directly to the Internet without NAT if desired.
Continued improvement of l3hw offload
*) l3hw – changed minimal supported values for “neigh-discovery-interval” and “neigh-keepalive-interval” properties; *) l3hw – fixed /32 and /128 route offloading after nexthop change; *) l3hw – fixed incorrect source MAC usage for offloaded bonding interface; *) l3hw – improved system responsiveness during partial offloading; *) l3hw – improved system stability during IPv6 route offloading; *) l3hw – improved system stability;
There were a number of l3hw fixes in 7.11 which allows the platforms with Marvell Prestera chips to be put into more roles for L3 switching and lower the cost of wirespeed performance for network operators.
Wifiwave2 development continues
*) wifiwave2 – added “steering” parameters and menu to set up and monitor AP neighbor groups (CLI only); *) wifiwave2 – added more information on roaming candidates to BSS transition management requests (802.11v) and neighbor report responses (802.11k); *) wifiwave2 – added option to filter frames captured by the sniffer command (CLI only); *) wifiwave2 – automatically add wifi interfaces to appropriate bridge VLAN when wireless clients with new VLAN IDs connect; *) wifiwave2 – changed default behavior for handling duplicate client MAC addresses, added settings for changing it (CLI only); *) wifiwave2 – enabled PMK caching with EAP authentication types; *) wifiwave2 – fixed “reg-info” information for several countries; *) wifiwave2 – fixed “security.sae-max-failure” rate not limiting authentications correctly in some cases; *) wifiwave2 – fixed clearing CAPsMAN Common Name when disabling “lock-to-caps-man”; *) wifiwave2 – fixed interface hangs on IPQ6010-based boards (introduced in v7.9); *) wifiwave2 – improved stability when changing interface settings; *) wifiwave2 – improved stability when receiving malformed WPA3-PSK authentication frames; *) wifiwave2 – make info log less verbose during client roaming (some info moved to wireless,debug log); *) wifiwave2 – rename “reg-info” country argument from “Macedonia” to “North Macedonia”; *) wifiwave2 – use correct status code when rejecting WPA3-PSK re-association;
The Wifiwave2 package has gotten a lot of love from the developers since the release of AX hardware and has stabilized quite a bit. The new wireless chips + active development might put MikroTik back in contention for wireless against WISP and WiFi vendors after a long period of limited innovation prior to the 2020s.
MikroTik already has a wealth of VPN options and is introducing another with the Back to Home or BTH VPN service. The framework underneath is based on wireguard which MikroTik began supporting in the early 7.x series of RouterOS.
What makes this notable is the addition of relay servers that MikroTik hosts to facilitate ease of VPN access when behind NAT.
This VPN service appears to be targeted at home users, gamers and others that need a “push button” VPN solution without complex routing or advanced configuration options.
It’s also important to point out that the solution is dual stacked and uses IPv4 and IPv6 by default so it will scale well globally.
BTH tab under the cloud menu
App based remote connectivity.
MikroTik is developing Android & Apple apps to use as clients of the BTH VPN service. While not released yet, there are preview screen shots in the online docs.
Manual Configuration
MikroTik also allows manual configuration of a remote endpoint using either ROS config for wireguard to be pasted in or via a QR code.
Having highly available layer-2 uplinks in a service provider has been around for a long time and there have been various ways to do it over the years, multi-chassis lag, stacking, active passive pseudowires, and more. If you use traditional VPLS and dual home a customer edge router you have to account for loop prevention.
We’re going to take a look at another way to handle this. EVPN has been used for multihoming in various environments for years. You might be familiar with EVPN/VxLAN in datacenters and dual attaching servers to an MLAG pair. Well there is another option – EVPN Multihoming – which uses ethernet segment identifiers to multi-home CEs. Think of this like MLAG but in BGP instead and since we’re looking at service providers we’ll use a MPLS dataplane instead of VxLAN.
Some of the advantages:
Easy Multihoming
Active/Active forwarding
High Availability
Resilient layer-2 overlay
one control plane – BGP
This design is going to utilize a netElastic vBNG and IP Infusion‘s OcNOS on a mix of edgecore and ufispace platforms to run ISIS-SR w/EVPN on top for layer-2 services.
The service provider core and access network utilizes ISIS-SR for MPLS services. If you need a refresher on Segment Routing there are several other articles on this blog or at stubarea51.net.
IPI-1 and IPI-2 are aggregation routers that aggregate last mile circuits which could be fiber, fixed wireless, or various other technologies. Then EVPN VPLS is overlayed on top of the MPLS network to deliver a layer-2 service from the vBNG to the subscriber equipment, in our case a mikrotik rb4011.
The vBNG then handles subscriber management. It takes the DHCP request from the rb4011 and asks the radius server if this subscriber is authorized. If the subscriber is authorized it hands out an IP address to the client.
The ufispace-q2c, a 9600-72xc, is utilized as an EVPN route reflector. In production IPA often recommends an out of path router reflector running on FRR. Lets verify that our control plane setup.
9600-72xc-1#show bgp l2vpn evpn summary
BGP router identifier 100.127.0.0, local AS number 1016
BGP table version is 10
1 BGP AS-PATH entries
0 BGP community entries
Neighbor V AS MsgRcv MsgSen TblVer InQ OutQ Up/Down State/PfxRcd AD MACIP MCAST ESI PREFIX-ROUTE
100.127.0.2 4 1016 38460 38496 10 0 0 01w4d09h 3 0 2 1 0 0
100.127.0.3 4 1016 38471 38490 10 0 0 01w4d09h 2 0 1 1 0 0
100.127.0.7 4 1016 312 314 10 0 0 02:11:38 2 0 1 1 0 0
Total number of neighbors 3
Total number of Established sessions 3
The route-reflector clients are all configured the same way.
ipi-1.lab.jan1.us.ipa.net#show bgp l2vpn evpn summary
BGP router identifier 100.127.0.2, local AS number 1016
BGP table version is 7
1 BGP AS-PATH entries
0 BGP community entries
Neighbor V AS MsgRcv MsgSen TblVer InQ OutQ Up/Down State/PfxRcd AD MACIP MCAST ESI PREFIX-ROUTE
100.127.0.0 4 1016 38501 38466 7 0 0 01w4d10h 4 0 2 2 0 0
Total number of neighbors 1
Total number of Established sessions 1
ipi-2.lab.jan1.us.ipa.net#show bgp l2vpn evpn summary
BGP router identifier 100.127.0.3, local AS number 1016
BGP table version is 9
1 BGP AS-PATH entries
0 BGP community entries
Neighbor V AS MsgRcv MsgSen TblVer InQ OutQ Up/Down State/PfxRcd AD MACIP MCAST ESI PREFIX-ROUTE
100.127.0.0 4 1016 38494 38476 9 0 0 01w4d10h 5 0 3 2 0 0
Total number of neighbors 1
Total number of Established sessions 1
Some of the general setup on the OcNOS devices is the same. You first need to enable EVPN to utilize the MPLS dataplane. IMPORTANT this requires a reboot! Make sure you set the VTEP IP as the loopback and then build the mac vrf for the layer2 vpn service and tie it to an EVPN id. Finally to utilize the service you attach it to a switchport as seen on xe44.10 switchport. In this case everything with a vlan id of 10-50 is placed into EVPN vpn-id 1.
5916-54XKS#show evpn mpls
EVPN-MPLS Information
=================
Codes: NW - Network Port
AC - Access Port
(u) - Untagged
VPN-ID EVI-Name EVI-Type Type Interface ESI VLAN DF-Status Src-Addr Dst-Addr
_______________________________________________________________________________________________________________________________
1 ---- L2 NW ---- ---- ---- ---- 100.127.0.7 100.127.0.3
1 ---- L2 NW ---- ---- ---- ---- 100.127.0.7 100.127.0.2
1 ---- -- AC xe44.10 --- Single Homed Port --- ---- ---- ---- ----
5916-54XKS#show evpn mpls mac-table
=========================================================================================================================================
EVPN MPLS MAC Entries
=========================================================================================================================================
VNID Interface VlanId In-VlanId Mac-Addr VTEP-Ip/ESI Type Status MAC move AccessPortDesc
_________________________________________________________________________________________________________________________________________
1 xe44.10 ---- ---- 2cc8.1b80.cebb 100.127.0.7 Dynamic Local ------- 0 -------
1 ---- ---- ---- 2cc8.1b80.cebc 100.127.0.3 Dynamic Remote ------- 0 -------
1 ---- ---- ---- 5254.0006.eb8f 00:00:00:5e:00:00:01:00:00:00 Dynamic Remote ------- 0 -------
1 ---- ---- ---- c4ad.34f4.b81d 100.127.0.2 Dynamic Remote ------- 0 -------
Total number of entries are : 4
5916-54XKS#show bgp l2vpn evpn
BGP table version is 8, local router ID is 100.127.0.7
Status codes: s suppressed, d damped, h history, a add-path, * valid, > best, i - internal,
l - labeled, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
[EVPN route type]:[ESI]:[VNID]:[relevent route informantion]
1 - Ethernet Auto-discovery Route
2 - MAC/IP Route
3 - Inclusive Multicast Route
4 - Ethernet Segment Route
5 - Prefix Route
Network Next Hop Metric LocPrf Weight Path Peer Encap
RD[100.127.0.1:2]
*>i [2]:[00:00:00:5e:00:00:01:00:00:00]:[1]:[48,5254:0006:eb8f]:[0]:[640]
100.127.0.2 0 100 0 i 100.127.0.0 MPLS
*>i [2]:[0]:[1]:[48,c4ad:34f4:b81d]:[0]:[640]
100.127.0.2 0 100 0 i 100.127.0.0 MPLS
*>i [3]:[1]:[32,100.127.0.2]
100.127.0.2 0 100 0 i 100.127.0.0 MPLS
RD[100.127.0.3:1]
*>i [2]:[0]:[1]:[48,2cc8:1b80:cebc]:[0]:[640]
100.127.0.3 0 100 0 i 100.127.0.0 MPLS
*>i [3]:[1]:[32,100.127.0.3]
100.127.0.3 0 100 0 i 100.127.0.0 MPLS
RD[100.127.0.7:1] VRF[ORANGE]:
*> [2]:[0]:[1]:[48,2cc8:1b80:cebb]:[0]:[640]
100.127.0.7 0 100 32768 i ---------- MPLS
* i [2]:[0]:[1]:[48,2cc8:1b80:cebc]:[0]:[640]
100.127.0.3 0 100 0 i 100.127.0.0 MPLS
* i [2]:[00:00:00:5e:00:00:01:00:00:00]:[1]:[48,5254:0006:eb8f]:[0]:[640]
100.127.0.2 0 100 0 i 100.127.0.0 MPLS
* i [2]:[0]:[1]:[48,c4ad:34f4:b81d]:[0]:[640]
100.127.0.2 0 100 0 i 100.127.0.0 MPLS
* i [3]:[1]:[32,100.127.0.2]
100.127.0.2 0 100 0 i 100.127.0.0 MPLS
* i [3]:[1]:[32,100.127.0.3]
100.127.0.3 0 100 0 i 100.127.0.0 MPLS
*> [3]:[1]:[32,100.127.0.7]
100.127.0.7 0 100 32768 i ---------- MPLS
The multihoming side takes a little additional configuration for the access interface.
ipi-1.lab.jan1.us.ipa.net#show run int xe10
!
interface xe10
channel-group 1 mode active
!
ipi-1.lab.jan1.us.ipa.net#show run int po1
!
interface po1
evpn multi-homed system-mac 0000.5e00.0001
!
ipi-1.lab.jan1.us.ipa.net#show run int po1.532
!
interface po1.532 switchport
encapsulation dot1q 532 inner-dot1q 10-50
rewrite pop
access-if-evpn
arp-cache disable
nd-cache disable
map vpn-id 1
!
ipi-2.lab.jan1.us.ipa.net#show run int xe10
!
interface xe10
channel-group 1 mode active
!
ipi-2.lab.jan1.us.ipa.net#show run int po1
!
interface po1
evpn multi-homed system-mac 0000.5e00.0001
!
ipi-2.lab.jan1.us.ipa.net#show run int po1.532
!
interface po1.532 switchport
encapsulation dot1q 532 inner-dot1q 10-50
rewrite pop
access-if-evpn
arp-cache disable
nd-cache disable
map vpn-id 1
!
On each PE we need to build the portchannel. This port-channel needs a system-mac defined for the multihoming. Then the criteria to place traffic into the EVPN-MH VPN needs to be defined. In this case it matches doubled tagged traffic. It’s not uncommon on SP networks to utilize double tagged traffic. So we add the outer tag and remove it at the attachment point. This is done with rewrite pop which removes or adds the outer tag.
The netElastic box has a stand LACP channel built and then matches the double tag on the interface defined. This interface will be mapped to BRAS for authentication prior to assigning an IP address.
First we need to build radius profiles so we can utilize them later.
netelastic-1# show running-config radius
radius vendor-id 54268
radius accounting-on enable
radius attribute-usermac-as mac
radius framed-route-delimiter ";"
radius username-override disable
radius authentication group RADIUS_AUTH
server-type ipv4-server
timeout 3
retry-times 3
nas-ip-address 100.127.0.6
master-server 1
algorithm master
dead-time 5
dead-count 10
class-as-car disable
filter-id-type user-acl
server 1 ipv4-address 192.168.0.2 port 1812 key netelastic
exit
radius accounting group RADIUS_ACCT
server-type ipv4-server
timeout 3
retry-times 3
nas-ip-address 100.127.0.6
algorithm master
dead-time 5
dead-count 10
flow-unit byte
align-authenreply disable
server 1 ipv4-address 192.168.0.2 port 1813 key netelastic
exit
radius dmcoa group
server-type ipv4-server
server 1 ipv4-address 192.168.0.2 key netelastic
Then we need to build a DHCP server and IP Pool on the netElastic box.
This is a very simple setup where the CPE, the rb4011, is authorized by it’s mac address. There are more options. We also are not going to assign speed plans, etc…
Prior to this working end to end we also need to build the freeradius server. This setup uses daloRADIUS. I won’t go into details on the freeradius initial build here.
The rb4011 is setup to request a DHCP address on vlan 11.
When this request hits the netElastic server over the EVPN L2VPN, netElastic will ask the radius server if the user is valid.
If the RADIUS server authenticates the user, which we defined with the mac address earlier, then the rb4011 will get an IP address. You can see the approval in the messages above and vlan11 has an IP address on the previous screen shot.
This lab does not have internet access but we’ll add a static route on the rb4011 to verify reachability through netElastic to the rest of the lab loopbacks.
The 3 lost packet seen in this screenshot were during the failure of IPI-1. Shutting po1 on IPI-1 forced traffic through IPI-2.
Conclusion
EVPN-MH is a great technology to uplink layer-2 services to a BNG for subscriber management and achieve high availability. This concept can continue to be built on by adding a second BNG for BNG resiliency and EVPN-MH on the remote side for high profile sites. We did not cover various other authentication methods and assigning speed packages to the subscriber but these all build off the radius server and tie back to the vBNG operation. netElastic also offers an optional CGNAT engine which can be added to this setup as well as DCHPv6.
Multicast used to be the defacto standard for delivering IPTV and a lot of legacy IPTV systems are still fully based on multicast or require sending local channels via multicast to a large provider to get the stream back for unicast delivery to clients. This has only been sped up by home users cancelling traditional cable and moving to things like roku, hulu live, apple tv, and then insert your streaming service of choice here.
In many networks, it is no longer advantageous or necessary to use complex mLDP and MVPN to deliver multicast traffic given the larger capacity of the links and reduced multicast traffic.
We’ve put in a lot of IPTV solutions and almost always outside of the content providers themselves, a centralized headend for delivery works well. This means building traditional VPLS tunnels back to the video source. It is simple, works well, and easy to maintain in comparison to mLDP, P2MP, and MVPN. Granted it does not scale as well as mLDP/MVPN but most tier 2 and tier 3 providers are not operating at a scale where the limits are met.
Additionally, to get these protocols it often means expensive hardware to deliver a complex solution.
Lets explore a solution put together with whitebox hardware; edgecore and ufispace running IP Infusions OcNOS SP 6.0.2.
IGP and MPLS Topology
This example utilizes LDP signaled VPLS running on top of ISIS-SR.
VPLS Topology
We’re running LDP signaled VPLS for service delivery. The latest version of OcNOS has also fixed the dependency of having to run LDP on an interface to operate LDP VPLS over SR-MPLS. This allows for easier overall configuration with less overhead in the cli.
We are going to statically join the group 239.0.0.1 on the multicast receivers; QFX-2, Mikrotik-1, and Mikrotik-2. Then we will send from the source 192.168.0.2 to group 239.0.0.1 utilizing iperf and verify receipt of traffic.
First lets look at the VPLS tunnels and verify they are up. We aren’t going to verify on every router.
9510-28DC-2#show mpls vpls detail
Virtual Private LAN Service Instance: LDP, ID: 1
SIG-Protocol: LDP
Attachment-Circuit :UP
Learning: Enabled
Control-Word: Disabled
Flow Label Status: Disabled, Direction: None, Static: No
Group ID: 0, VPLS Type: Ethernet, Configured MTU: 1500
Description: none
service-tpid: dot1.q
Operating mode: Raw
Configured interfaces:
Interface: xe27.100
Subinterface Match Criteria(s) :
dot1q 100
Interface: ce2.100
Subinterface Match Criteria(s) :
dot1q 100
Mesh Peers:
100.127.0.3 (Up)
100.127.0.4 (Up)
9510-28DC-2#show mpls ldp session
Peer IP Address IF Name My Role State KeepAlive UpTime
100.127.0.4 ce2 Active OPERATIONAL 30 3d23h31m
100.127.0.3 ce3 Active OPERATIONAL 30 3d23h32m
9510-28DC-2#show mpls vpls mac-address name LDP
MAC address Learned from Vlan-Id Peer address Time-out
000c.42b2.a63c ce2 - 100.127.0.4 300
4c5e.0c23.df4e xe27.100
84c1.c132.5032 ce3 - 100.127.0.3 300
So now we can see that targeted sessions up, attachment circuit up, and local/remote mac addresses learned across the circuit. Lets move on to multicast verification.
First we’ll verify that routing over OSPF is up and reachability then PIM.
Here is the RP and the PIM DR for the routers connecting over the VPLS tunnels.
and for people that like the CLI.
[email protected]# run show pim neighbors
B = Bidirectional Capable, G = Generation Identifier
H = Hello Option Holdtime, L = Hello Option LAN Prune Delay,
P = Hello Option DR Priority, T = Tracking Bit,
A = Hello Option Join Attribute
Instance: PIM.master
Interface IP V Mode Option Uptime Neighbor addr
xe-0/0/47.100 4 2 HPLGT 05:04:56 172.16.0.1
xe-0/0/47.100 4 2 HPLGT 05:03:43 172.16.0.2
{master:0}[edit]
[email protected]# run show ospf neighbor
Address Interface State ID Pri Dead
172.16.0.2 xe-0/0/47.100 Full 172.16.0.2 128 35
172.16.0.1 xe-0/0/47.100 Full 192.168.0.1 128 35
Lets verify that we got the s,g entry and the multicast traffic on the receivers. We will look at the torch output on mikrotik-2 to see the inbound traffic.
[email protected]# run show multicast route
Instance: master Family: INET
Group: 239.0.0.1
Source: 192.168.0.2/32
Upstream interface: xe-0/0/47.100
As seen above we have the appropriate entries for the group and inbound traffic.
Conclusion
It is possible and often better to deploy a simple solution using MPLS and VPLS overlayed on top for multicast delivery. This is simple to deploy, troubleshoot, and maintain. This same principle can be extended with EVPN for a layer 2 overlay with either MPLS or VxLAN data plane. This extensibility makes the solution future proof and easy to migrate as the underlying service or transport changes.
We’ve worked with ISPs on IPTV deployments using L2VPN into the tens of thousands of receivers with the correct planning considerations.
Please look for future blog posts for scaling multicast in large networks or working with multicast in a bandwidth-constrained environment.
We recently did a webinar on alternatives to Juniper Network’s MX204 platform. While the MX204 was a fully capable router it was End Of Life’d and then recently the EOL was revoked after approximately a year. During the period before the end of life revocation operators searched for various alternatives.
IP Infusion’s OcNOS rose as one of the prime contenders as a replacement after they announced support for newer silicon, Qumran2 from broadcom, allowing them to enter the market as a capable border and peering router.
We went through some of the supported platforms, support systems, and whitebox ecosystem in this webinar.
Cribl presented at Security Field Day 8 and while they look to have an awesome product for observability we’re going to look at something else, a refreshing way to provide training, resources, and product information.
It became immediately apparent during their presentation that their core values actually meant something:
customers first always
irreverent, but serious
curious
transparent
together
What does Cribl do?
Without getting into the product to much, I’ll do this after I review their free resource (more on this later), this picture gives a great overview.
The data is separated and enriched to allow for the right data to go to the right place. That is to say, the marshmallows, things people care about, are separated and put into the analytics tool. While everything is stored to meet compliance requirements.
Resources
The thing that really got me excited about Cribl was their approach to resources, training, and demos.
There are still some vendors that give free training and discounted certification, Juniper’s program comes to mind, but not to the extent here.
One of the presenters made a comment about how easy it was to get started and you didn’t need accounts/wouldn’t be pestered by sales the instant you signed up. This sounded too good to be true so I went over to their website to see for myself. I was able to get setup on their sandbox just by putting my email in for a verification code to make sure I wasn’t a bot. It’s been several days and I still haven’t gotten any sales spam.
There is also a free training and certification program. That was just as easy to get started with.
If that wasn’t enough you can download the full product suite with 1TB/day of data. How’s that for a test drive? It also was extremely easy to follow their licensing model and determine what you get.
Conclusion
It is refreshing to see a company take such a customer first approach. It was easy to get started, learn, and evaluate the software. All without having to go through a drawn out sales process we’ve become all too accustom too in technology. I didn’t have to go to a fancy dinner or jump through hoops to get a demo. I just went to the website and got all the resources I needed to start. More to come as I work through the training materials.
One of the biggest misconceptions I had before moving into the service provider space was that all layer 2 operations had been replaced with layer 3. I quickly found out that even if you are routing everything there are still a lot of layer 2 overlays in place, carrier ethernet, or even spanning-tree (STP). Running these technologies, hopefully not STP, is especially important rural broadband, electric co-ops, and telcos to enable things such as IP conservation and subscriber management on a broadband network gateway.
The side effect of layer 2 transport technologies being so heavily used in last mile internet service providers is having to understand vlan tagging operations. The are occasions where one side of the circuit will be double tagged and the other side will be single tagged or everything will be double tagged/single tagged/untagged. Let’s take a quickly look at some simple tag operations on IP Infusions OcNOS SP 6.0.
IGP and MPLS setup
We setup a simple topology of isis and ldp to create VPWS circuits. We are going to build two circuits in this example.
We are going to attach the circuits to the same physical ports and utilize tags to place the traffic into the correct VPWS circuit. Mikrotik-1 is double tagged and Mikrotik-2 is single tagged.
If you’ve built pseudowires on OcNOS before you might be familiar with the service-template model. However, when building VPWS circuits and utilizing the same interface this isn’t an option. You have to make a subinterface that is a switchport. This also changes the method for tag operations.
The switchport subinterfaces are then assigned as an access interface matching specific tags. In this case outer tag 3 and inner tag 400-499. On IPI-2 we are expecting single tagged frames instead of double tagged. This is where the rewrite pop comes in. This will remove the outer tag on ingress to the PW and push it back on on egress towards Mikrotik-1.
Since the AC towards Mikrotik-2 is only expecting single tagged packets we just do a simple match on encapsulation.
Let’s do some verification. First we’ll verify that the circuits are up.
ipi-2.lab.jan1.us.ipa.net#show ldp targeted-peers
IP Address Interface
100.127.0.1 xe48
ipi-2.lab.jan1.us.ipa.net#show ldp mpls-l2-circuit
Transport Client VC VC Local Remote Destination
VC ID Binding State Type VC Label VC Label Address
123 xe11.400 UP Ethernet VLAN 25602 26240 100.127.0.1
1234 xe11.4 UP Ethernet VLAN 25603 26241 100.127.0.1
ipi-2.lab.jan1.us.ipa.net#show ldp mpls-l2-circuit detail
PW ID: 123, VC state is up
Access IF: xe11.400,up,AC state is up
Session IF: xe48, state is up
Destination: 100.127.0.1, Peer LDP Ident: 100.127.0.1
Local vctype: vlan, remote vctype :vlan
Local groupid: 0, remote groupid: 0
Local label: 25602, remote label: 26240
Local MTU: 1500, Remote MTU: 1500
Local Control Word: disabled Remote Control Word: Not-Applicable Current use: disabled
Local Flow Label Direction: Disabled, Static: Disabled
Remote Flow Label Direction: Both , Static: Disabled
Local PW Status Capability : disabled
Remote PW Status Capability : disabled
Current PW Status TLV : disabled
PW ID: 1234, VC state is up
Access IF: xe11.4,up,AC state is up
Session IF: xe48, state is up
Destination: 100.127.0.1, Peer LDP Ident: 100.127.0.1
Local vctype: vlan, remote vctype :vlan
Local groupid: 0, remote groupid: 0
Local label: 25603, remote label: 26241
Local MTU: 1500, Remote MTU: 1500
Local Control Word: disabled Remote Control Word: Not-Applicable Current use: disabled
Local Flow Label Direction: Disabled, Static: Disabled
Remote Flow Label Direction: Disabled, Static: Disabled
Local PW Status Capability : disabled
Remote PW Status Capability : disabled
Current PW Status TLV : disabled
We have successfully manipulated the tags and passed traffic end to end. We started with a double tagged packet, pop off the outer tag, placed it into a PW, spit out a single tagged packet, and had end to end reachability.
Conclusion
There are various methods and configurations to manipulate traffic. This is one of the more common examples of tag manipulation that occurs in broadband aggregation. I will explore tag manipulation with service-templates and VPLS in a later post.
Recently, we recorded a webinar to explain a design concept frequently used by iparchitechs.com to build and migrate WISP, FISP and Telco networks – separation of network functions. It centers around simplification of roles within an ISP network. It also explores the use of lower-cost commodity network equipment to maximize the service area for a given ISP footprint while meeting key requirements like scale, redundancy and capacity.
With the current state of the supply chain lead times for networking gear can be astronomical. This led consumers to look at other options for networking equipment forcing the whitebox and disaggregated networking market to become more prevalent.
With full featured operating systems like IP Infusion‘s OcNOS 6.0 and commodity hardware from Ufispace and Edgecore companies have been about to upgrade faster and further than ever before.
We’ll be looking at the ufispace 9600-32s and 9500-30xs in this deployment. This is shaping up to be a great combination for 100g and 10g density. Since both run the same operating system moving between is easy. While the bigger Qumran2c, 9600-32s, doesn’t support breakouts/10g we can aggregate and terminate 100g services here while using a small device to delivery 10g density and breakout.
We’re going to look specifically at VPLS and VPWS delivery in this deployment. Since these deployments typically complement existing deployments we’ll look at interop with a Calix e9-2 ASM 3001 deployment.
I know Calix doesn’t normally come to mind for MPLS deployments but more for FTTX or ERPS. However, they’ve been putting a lot of effort into their MPLS stack on the e9-2 ASM platform which has helped led to this testing.
We also have an Arista 7280CR3K-32P4 acting as a p-router during link failure.
IGP/LDP Setup
We’re going to run isis as an IGP which is typical in a service provider network. This time we’re going to run straight LDP instead of SR-MPLS, however, you can still utilize SR-MPLS with a mapping server if your topology supports it.
Let’s verify IGP/LDP and routing.
ASM3001# show isis neighbors
NEIGHBOR HOLD CIRCUIT
SYSTEM ID TYPE INTERFACE STATE TIME ID
-------------------------------------------------------
0010.0127.0118 L2 la3 UP 23 3
0010.0127.0119 L2 la4 UP 26 4
details
NEIGHBOR HOLD CIRCUIT
SYSTEM ID TYPE INTERFACE STATE TIME ID
-------------------------------------------------------
0010.0127.0118 L2 la3 UP 23 3
Hostname:ufispace-100
SNPA:e8c5.7a77.a655
State Changed:3214
LAN Priority:0
Restart Capable:1
Peer Restart State:1
0010.0127.0119 L2 la4 UP 26 4
Hostname:ARISTA
SNPA:c4ca.2b66.fb6d
State Changed:3152
LAN Priority:0
Restart Capable:1
Peer Restart State:1
-------------------------------------------------------
ASM3001# show mpls ldp neighbors
LOOP INTERFACE
INDEX PEER LDP ID LOCAL LDP ID TYPE SESSION DISTMODE DETECTION TRANS ADD NAME
---------------------------------------------------------------------------------------------------------------
1 100.127.0.118:0 100.127.0.117:0 TARGETED DownstreamUnsolicited Disabled 100.127.0.119 none
2 100.127.0.118:0 100.127.0.117:0 DIRECTED DownstreamUnsolicited Disabled 100.127.0.119 la3
3 100.127.0.119:0 100.127.0.117:0 DIRECTED DownstreamUnsolicited Disabled 100.127.0.119 la4
ASM3001# show ip route all
ROUTE
INDEX PREFIX NEXT HOP TYPE DISTANCE INTERFACE UPTIME
----------------------------------------------------------------------------
1 100.126.2.160/29 100.126.2.161 local 0/0 la3 0:9:47
2 100.126.2.161/32 0.0.0.0 local 0/0 la3 0:9:47
3 100.126.2.168/29 100.126.2.169 local 0/0 la4 0:9:42
4 100.126.2.169/32 0.0.0.0 local 0/0 la4 0:9:42
5 100.126.2.176/29 100.126.2.162 isis 115/20 la3 0:9:33
6 100.126.2.170 isis 115/20 la4 0:9:33
7 100.126.2.184/29 100.126.2.162 isis 115/20 la3 0:9:33
8 100.127.0.117/32 0.0.0.0 local 0/0 loopback1 0:9:53
9 100.127.0.118/32 100.126.2.162 isis 115/20 la3 0:9:33
10 100.127.0.119/32 100.126.2.170 isis 115/20 la4 0:9:33
11 100.127.0.120/32 100.126.2.162 isis 115/30 la3 0:9:33
ASM3001# ping 100.127.0.120
PING 100.127.0.120 (100.127.0.120) 56(84) bytes of data.
64 bytes from 100.127.0.120: icmp_seq=1 ttl=63 time=0.778 ms
64 bytes from 100.127.0.120: icmp_seq=2 ttl=63 time=0.722 ms
OcNOS-SW#show clns neighbors
Total number of L1 adjacencies: 0
Total number of L2 adjacencies: 1
Total number of adjacencies: 1
Tag UNDERLAY: VRF : default
System Id Interface SNPA State Holdtime Type Protocol
ufispace-100 po1 e8c5.7a77.a657 Up 26 L2 IS-IS
OcNOS-SW#ping 100.127.0.117 source-ip 100.127.0.120
Press CTRL+C to exit
PING 100.127.0.117 (100.127.0.117) from 100.127.0.120 : 56(84) bytes of data.
64 bytes from 100.127.0.117: icmp_seq=1 ttl=64 time=0.811 ms
64 bytes from 100.127.0.117: icmp_seq=2 ttl=64 time=0.746 ms
Since we have LDP neighbors and loopback to loopback reachability we can begin to build our services.
100g VPWS
First we’ll build a VPWS service between the E9-2 and ufispace-100 to verify functionality. We’ll utilize a TX300s-100GX test set to push traffic through the service.
First lets look at the ASM configuration for the xconnect.
Below you can see the config for the interface facing the test set. This will put the traffic into the VPWS service.
ASM3001# show running-config interface ethernet 1/2/q7
interface ethernet 1/2/q7
no shutdown
role uni
arp arp-announce any
l2transport
point-to-point 200
!
!
Now we can see the same on the IP infusion side.
ufispace-100#show run mpls
!
service-template TEST
match all
!
mpls l2-circuit TEST-VPWS 200 100.127.0.117
ufispace-100#show run int ce17
!
interface ce17
switchport
mtu 1986
mpls-l2-circuit TEST-VPWS service-template TEST primary
!
Finally lets verify functionality. I did a verbose output of the circuit details to help see all of the details. Some important things to match are the MTU and if it’s a vlan or raw service.
IP Infusion sets the MTU on the attachment circuit while Calix is inherited from the default interface value of 2000 minus some overhead.
ufispace-100#show ldp mpls-l2-circuit detail
PW ID: 200, VC state is up
Access IF: ce17,up,AC state is up
Session IF: po1, state is up
Destination: 100.127.0.117, Peer LDP Ident: 100.127.0.117
Local vctype: vlan, remote vctype :vlan
Local groupid: 0, remote groupid: 0
Local label: 24962, remote label: 26
Local MTU: 1986, Remote MTU: 1986
Local Control Word: disabled Remote Control Word: Not-Applicable Current use: disabled
Local Flow Label Direction: Disabled, Static: Disabled
Remote Flow Label Direction: Disabled, Static: Disabled
Local PW Status Capability : disabled
Remote PW Status Capability : enabled
Current PW Status TLV : disabled
Local VCCV Capability:
CC-Types: None
CV-Types: None
Remote VCCV Capability:
CC-Types: Type 3
CV-Types:
LSP ping
ASM3001# show l2vpn xconnect pw-id 200
l2vpn xconnect pw-id 200
XCONNECT NAME STATE
--------------------------------- ---------------
200 Up
-------------------------------------------------
VPWS Index : 2
VPN Key : 131074
% 1 entries in the table.
AC Details
-------------------------------------------------
INTERFACE VLAN STATE TYPE MTU VPWS-INDEX
------------ --------- --------------- ---------- --------- -----------
1/2/q7 NA Active Tagged 1986 2
% 1 entries in the table.
PW Details
-------------------------------------------------
PW-ID PW-STATE PW-CLASS ENCAPSULATION PROTOCOL ADMIN-STATE REDUNDANCY-STATE VPWS-INDEX
------ ----------------- --------------------- -------------- --------- ------------ ----------------- ----------
200 Up PWE-1 MPLS LDP Up NA 2
-----------------------------------------------------------------------------------------------------------------
PW-INFO LOCAL REMOTE
------------- -------------------- --------------------
Address 100.127.0.117 100.127.0.118
PW ID 200 uNknOwn
PW type Tagged uNknoWn
Label 26 24962
MTU 1986 1986
Control Word Disabled uNknOwn
Status TLV Enabled Disabled
CC Type 4 0
CV Type 2 0
Local Status (PW Status TLV): 0x6
Remote Status (PW Status TLV): 0x0
Create time: 2022-09-10 09:17:23
Last time status changed: 2022-09-10 09:30:28
% 1 entries in the table.
Finally, we can see 95g of traffic across the circuit with the test set.
10g VPLS
Next we will look at a 10g VPLS service delivered off the extension switch. We already saw end to end reachability in the IGP setup so we will start with configuration.
On the ASM you build a bridge domain and tie it to a virtual forwarding instance.
Again, we tie the interface facing the test kit into the bridge-domain. This will put the traffic into the VPLS instance.
ASM3001# show running-config interface ethernet 1/1/x15
interface ethernet 1/1/x15
no shutdown
role uni
arp arp-announce any
l2transport
rewrite-ingress tag add dot1q 220
!
bridge-domain 220
!
!
!
Here we also have to define the peers for targeted hellos in LDP.
OcNOS-SW#show run ldp
!
router ldp
router-id 100.127.0.120
graceful-restart full
targeted-peer ipv4 100.127.0.117
exit-targeted-peer-mode
transport-address ipv4 100.127.0.120
!
Finally, we attached the a port to the service and plug in the test kit.
OcNOS-SW#show run int xe15
!
interface xe15
switchport
mtu 9086
mpls-vpls TEST-VPLS service-template TEST
exit-if-vpls
!
Again, we will look at the verbose output and pay attention to MTU and VPLS type, vlan in this case.
OcNOS-SW#show mpls vpls detail
Virtual Private LAN Service Instance: TEST-VPLS, ID: 220
SIG-Protocol: LDP
Attachment-Circuit :UP
Learning: Enabled
Control-Word: Disabled
Flow Label Status: Disabled, Direction: None, Static: No
Group ID: 0, VPLS Type: Ethernet VLAN, Configured MTU: 9086
Description: none
service-tpid: dot1.q
Operating mode: Tagged
Svlan Id: 0
Svlan Tpid: 8100
Configured interfaces:
Interface: xe15
Service-template : TEST
Match criteria : Accept all
Mesh Peers:
100.127.0.117 (Up)
ASM3001# show l2vpn bridge-domain bd-name 220
l2vpn bridge-domain bd-name 220
BRIDGE DOMAIN NAME STATE
--------------------------------- ---------------
220 Up
-------------------------------------------------
VPLS Index : 3
VPN Key : 65539
MTU : 9086
MAC Learning : ENABLE
MAC Aging Time : 300
MAC Limit Max : 1024
MAC Action : FLOOD
AD Type : NONE
SIG type : NONE
Transport Mode : ETHERNET TAGGED
Control Word : DISABLE
Route Distinguisher: 0x0000000000000000(NULL)
VPLS ID : 0x0000000000000000(DEFAULT)
VE ID : 0
VE Range : 8
% 1 entries in the table.
AC Details
------------ --------------- ---------- ------------ ---------------
DESCRIPTION STATE TYPE VPLS INDEX SPLIT HORIZON
------------ --------------- ---------- ------------ ---------------
1.x15 Active Ethernet 3 Disabled
% 1 entries in the table.
PW Details
PW-ID STATE PW-Class ENCAPSULATION VPLS-INDEX ADMIN-STATE
--------------- ----------- --------------------- -------------- ---------- ------------
220 Up vlan-pwe MPLS 3 Up
------------------------------------------------------------------------------------
------------- -------------------- --------------------
PW LOCAL REMOTE
------------- -------------------- --------------------
Address 100.127.0.117 100.127.0.120
PW ID 220 uNknOwn
PW type Tagged uNknoWn
Label 34 24961
MTU 9086 9086
Control Word Disabled uNknOwn
Status TLV Enabled Disabled
CC Type 4 0
CV Type 2 0
Create time: 2022-09-10 09:55:06
Last time status changed: 2022-09-10 09:58:56
% 1 entries in the table.
Finally, we can see all of the traffic on the test set across the circuit.
Conclusion
Disaggregated networking provides an alternative to traditional vendors and these are real world examples of service deployment for service providers. A special thanks to Sorin Esanu and Race Communications for organizing this test environment as a proof of concept for their deployment.
