We knew it was coming when MikroTik announced the end of production for Tilera chips back in the fall of 2022, but recently (looks like it was last week), MikroTik moved almost all of the CCR1K series (CCR1009, CCR1016, CCR1036) to discontinued officially on the website.
The only CCR1K series that are still listed for sale are:
CCR1009-7G-1C-1S+ CCR1016-12S-1S+ CCR1072-1G-8S+
This likely indicates low or no stock to replenish distributors for the discontinued models, so the CCR1K market is going to get even tighter. Expect the three models still listed to go discontinued in the next 60 to 90 days if not sooner.
It’s a bit of a double-edged sword as operators that haven’t made the jump to ROSv7 will quickly get forced into CCR2K models that only support ROSv7 and while MikroTik has done an awesome job of closing the gap, there is still some work to achieve feature parity with ROSv6.
That said, the upside is that the massive rush to CCR2K that has been happening will help to refine ROSv7 with more bug fixes and improved stability due to the larger user base running it and reporting bugs.
CCR2K series
A worthy line of successors
Luckily we’ve seen rapid development on ROSv7 and the CCR2k line which is arm64 based with several models offering L3 hardware offload using Marvell Prestera chips. arm64
MikroTik has introduced several arm64 chips with the introduction of the CCR2k line. most notably, chips from annapurna labs which is owned by Amazon.
The CCR2116/CCR2216 models are outfitted with a 16 core Annapurna CPU that can move up to 200 Gbps of traffic.
Marvell Prestera
The Marvell series of ASICs has been incredibly successful for MikroTik and other vendors at providing a rich feature set for L2/L3 while maintaining a relatively low price point.
ROSv7
ROSv7 has been on fire this year (in a good way) as far as development is concerned and releases are coming in a timely manner with significant work put into each one fixing bugs. By the end of the year we’ll likely see a long-term release and ROSv6 start to wane in popularity.
New hardware
Tilera created some issues in what MikroTik was able to accomplish because it never became more than a niche CPU architecture. So form factors were limited to what Tilera was able to handle.
Now that arm64 & marvell form the base of CPU/ASIC architecture for MikroTik, the possibilities are broader and there are more hardware platforms on the way.
2023 has been a big year for MikroTik RouterOS v7 development. The OS has matured quite a bit and the number of operators that are using ROSv7 in prod has gone up significantly.
Most of the work this year has been centered around feature parity with ROSv6 and performance improvements.
Although there have been a few new additions like the Back to Home VPN (Which is now scheduled to release with 7.12) and IS-IS (Also slated for 7.12).
This is important because most of the development time by MikroTik has been consumed by stabilization, feature parity and perf improvements. However, as that gap closes, we should see long awaited features like SR-MPLS and EVPN make it into ROS v7 faster as development time frees up.
7.11 contains a long list of fixes and improvements and will likely be adopted quickly by the MikroTik community.
*) bfd – fixed “actual-tx-interval” value and added “remote-min-tx” (CLI only); *) bfd – improved system stability;
Bi-directional forwarding detection or BFD has been a feature gap in ROSv7 for a long time and was a showstopper for some ASNs that wanted to use the new CCR2K hardware as peering/border routers. Now that BFD has been implemented for BGP and OSPF with some stabilization fixes, it’s much easier to deploy a CCR2116 or CCR2216 in that role.
IPv6 support for containers
*) container – added IPv6 support for VETH interface;
Being able to use IPv6 for containers is a great addition given the rapid pace of IPv6 adoption in 2023. It also makes it easier to expose containers directly to the Internet without NAT if desired.
Continued improvement of l3hw offload
*) l3hw – changed minimal supported values for “neigh-discovery-interval” and “neigh-keepalive-interval” properties; *) l3hw – fixed /32 and /128 route offloading after nexthop change; *) l3hw – fixed incorrect source MAC usage for offloaded bonding interface; *) l3hw – improved system responsiveness during partial offloading; *) l3hw – improved system stability during IPv6 route offloading; *) l3hw – improved system stability;
There were a number of l3hw fixes in 7.11 which allows the platforms with Marvell Prestera chips to be put into more roles for L3 switching and lower the cost of wirespeed performance for network operators.
Wifiwave2 development continues
*) wifiwave2 – added “steering” parameters and menu to set up and monitor AP neighbor groups (CLI only); *) wifiwave2 – added more information on roaming candidates to BSS transition management requests (802.11v) and neighbor report responses (802.11k); *) wifiwave2 – added option to filter frames captured by the sniffer command (CLI only); *) wifiwave2 – automatically add wifi interfaces to appropriate bridge VLAN when wireless clients with new VLAN IDs connect; *) wifiwave2 – changed default behavior for handling duplicate client MAC addresses, added settings for changing it (CLI only); *) wifiwave2 – enabled PMK caching with EAP authentication types; *) wifiwave2 – fixed “reg-info” information for several countries; *) wifiwave2 – fixed “security.sae-max-failure” rate not limiting authentications correctly in some cases; *) wifiwave2 – fixed clearing CAPsMAN Common Name when disabling “lock-to-caps-man”; *) wifiwave2 – fixed interface hangs on IPQ6010-based boards (introduced in v7.9); *) wifiwave2 – improved stability when changing interface settings; *) wifiwave2 – improved stability when receiving malformed WPA3-PSK authentication frames; *) wifiwave2 – make info log less verbose during client roaming (some info moved to wireless,debug log); *) wifiwave2 – rename “reg-info” country argument from “Macedonia” to “North Macedonia”; *) wifiwave2 – use correct status code when rejecting WPA3-PSK re-association;
The Wifiwave2 package has gotten a lot of love from the developers since the release of AX hardware and has stabilized quite a bit. The new wireless chips + active development might put MikroTik back in contention for wireless against WISP and WiFi vendors after a long period of limited innovation prior to the 2020s.
MikroTik already has a wealth of VPN options and is introducing another with the Back to Home or BTH VPN service. The framework underneath is based on wireguard which MikroTik began supporting in the early 7.x series of RouterOS.
What makes this notable is the addition of relay servers that MikroTik hosts to facilitate ease of VPN access when behind NAT.
This VPN service appears to be targeted at home users, gamers and others that need a “push button” VPN solution without complex routing or advanced configuration options.
It’s also important to point out that the solution is dual stacked and uses IPv4 and IPv6 by default so it will scale well globally.
BTH tab under the cloud menu
App based remote connectivity.
MikroTik is developing Android & Apple apps to use as clients of the BTH VPN service. While not released yet, there are preview screen shots in the online docs.
Manual Configuration
MikroTik also allows manual configuration of a remote endpoint using either ROS config for wireguard to be pasted in or via a QR code.
One of the biggest misconceptions I had before moving into the service provider space was that all layer 2 operations had been replaced with layer 3. I quickly found out that even if you are routing everything there are still a lot of layer 2 overlays in place, carrier ethernet, or even spanning-tree (STP). Running these technologies, hopefully not STP, is especially important rural broadband, electric co-ops, and telcos to enable things such as IP conservation and subscriber management on a broadband network gateway.
The side effect of layer 2 transport technologies being so heavily used in last mile internet service providers is having to understand vlan tagging operations. The are occasions where one side of the circuit will be double tagged and the other side will be single tagged or everything will be double tagged/single tagged/untagged. Let’s take a quickly look at some simple tag operations on IP Infusions OcNOS SP 6.0.
IGP and MPLS setup
We setup a simple topology of isis and ldp to create VPWS circuits. We are going to build two circuits in this example.
We are going to attach the circuits to the same physical ports and utilize tags to place the traffic into the correct VPWS circuit. Mikrotik-1 is double tagged and Mikrotik-2 is single tagged.
If you’ve built pseudowires on OcNOS before you might be familiar with the service-template model. However, when building VPWS circuits and utilizing the same interface this isn’t an option. You have to make a subinterface that is a switchport. This also changes the method for tag operations.
The switchport subinterfaces are then assigned as an access interface matching specific tags. In this case outer tag 3 and inner tag 400-499. On IPI-2 we are expecting single tagged frames instead of double tagged. This is where the rewrite pop comes in. This will remove the outer tag on ingress to the PW and push it back on on egress towards Mikrotik-1.
Since the AC towards Mikrotik-2 is only expecting single tagged packets we just do a simple match on encapsulation.
Let’s do some verification. First we’ll verify that the circuits are up.
ipi-2.lab.jan1.us.ipa.net#show ldp targeted-peers
IP Address Interface
100.127.0.1 xe48
ipi-2.lab.jan1.us.ipa.net#show ldp mpls-l2-circuit
Transport Client VC VC Local Remote Destination
VC ID Binding State Type VC Label VC Label Address
123 xe11.400 UP Ethernet VLAN 25602 26240 100.127.0.1
1234 xe11.4 UP Ethernet VLAN 25603 26241 100.127.0.1
ipi-2.lab.jan1.us.ipa.net#show ldp mpls-l2-circuit detail
PW ID: 123, VC state is up
Access IF: xe11.400,up,AC state is up
Session IF: xe48, state is up
Destination: 100.127.0.1, Peer LDP Ident: 100.127.0.1
Local vctype: vlan, remote vctype :vlan
Local groupid: 0, remote groupid: 0
Local label: 25602, remote label: 26240
Local MTU: 1500, Remote MTU: 1500
Local Control Word: disabled Remote Control Word: Not-Applicable Current use: disabled
Local Flow Label Direction: Disabled, Static: Disabled
Remote Flow Label Direction: Both , Static: Disabled
Local PW Status Capability : disabled
Remote PW Status Capability : disabled
Current PW Status TLV : disabled
PW ID: 1234, VC state is up
Access IF: xe11.4,up,AC state is up
Session IF: xe48, state is up
Destination: 100.127.0.1, Peer LDP Ident: 100.127.0.1
Local vctype: vlan, remote vctype :vlan
Local groupid: 0, remote groupid: 0
Local label: 25603, remote label: 26241
Local MTU: 1500, Remote MTU: 1500
Local Control Word: disabled Remote Control Word: Not-Applicable Current use: disabled
Local Flow Label Direction: Disabled, Static: Disabled
Remote Flow Label Direction: Disabled, Static: Disabled
Local PW Status Capability : disabled
Remote PW Status Capability : disabled
Current PW Status TLV : disabled
We have successfully manipulated the tags and passed traffic end to end. We started with a double tagged packet, pop off the outer tag, placed it into a PW, spit out a single tagged packet, and had end to end reachability.
Conclusion
There are various methods and configurations to manipulate traffic. This is one of the more common examples of tag manipulation that occurs in broadband aggregation. I will explore tag manipulation with service-templates and VPLS in a later post.
Recently, we recorded a webinar to explain a design concept frequently used by iparchitechs.com to build and migrate WISP, FISP and Telco networks – separation of network functions. It centers around simplification of roles within an ISP network. It also explores the use of lower-cost commodity network equipment to maximize the service area for a given ISP footprint while meeting key requirements like scale, redundancy and capacity.
The pace of development for MikroTik RouterOS version 7 has definitely sped up in 2022 and we are seeing the results in improved stability and features added.
As of August 31st, 2022, MikroTik moved ROS v7.5rc2 into v7.5 stable
dhcpv6-relay – not being able to relay a PD request from a delegating router for IPv6 has been a limitation of MikroTik routers for a while so getting this fixed has a big impact on scaling MikroTik IPv6 deployments
RTSP helper – The addition of a Real Time Streaming Protocol helper is a great addition to ROSv7 to make NAT traversal for realtime applications (IPTV, SIP and IP cameras) easier.
A good overview of the discussion leading up to the addition of RTSP is here: RTSP Helper – MikroTik
l3hw – fixed hw offloaded NAT – This feature still has some issues as IP ArchiTechs recently filed a bug (SUP-91389) where src-nat traffic that carries an H flag in the connection table will die after 1 hour with a 10G load on the router. Once this feature receives further bug fixes and testing, it’s going to be very useful for high capacity but low cost NAT44 gateways.
lte – this category got a significant amount of development work as there are numerous fixes with many relating to the Chateau devices.
wifiwave2 – There was also a significant amount of development in wifi wave 2 which included notable additions like 802.11k for roaming.
vrrp – added “sync-connection-tracking” compatibility with preemption-mode – this is a long awaited feature that showed up early in ROSv7 but did not have pre-emption mode capabilities. The addition of connection synching between routers positions MikroTik routers much closer to traditional enterprise firewall vendors so that failover between devices can include connections.
Changing between network operating systems is one of the most challenging things for new engineers. Most people learned cisco cli due to their extensive training system or got on the job training for whatever their company runs.
We are hoping to make moving back and forth between network operating systems a little easier with some useful show and operational commands for Mikrotik, Juniper, Cisco, and IP Infusion. There are in detail usages of these commands on stubarea51.net.
OSPF Commands
MikroTik
Juniper
Cisco
IP Infusion
routing ospf neighbor print
show ospf neighbor
show ip ospf neighbor
show ip ospf neighbor
routing ospf interface print
show ospf interface
show ip ospf interface
show ip ospf interface
routing ospf instance print detail
show ospf overview brief
show ip ospf 1
show ip ospf 1
routing ospf lsa print
show ospf database
show ip ospf database
show ip ospf database
ip route print where ospf=yes
show route protocol ospf
show ip route ospf
show ip route ospf
routing ospf area-border-router print
show ospf route abr
show ip ospf border-routers
show ip ospf border-routers
routing ospf as-border-router print
show ospf route asbr
show ip ospf border-routers
show ip ospf border-routers
MPLS – LDP Commands
Mikrotik
Juniper
Cisco
IP Infusion
mpls ldp neighbor print
show ldp neighbor
show mpls ldp neighbor
show mpls ldp neighbor
mpls ldp interface print
show ldp interface
show mpls interfaces
show ldp interface
mpls forwarding-table print
show route forwarding-table family mpls
show mpls forwarding-table
show mpls forwarding-table
mpls remote-bindings print
show ldp database
show mpls binding
show mpls ilm-table
mpls local-bindings print
show ldp database
sh mpls ip binding local
show mpls ilm-table
mpls print
show mpls label usage
sh mpls ldp parameters
show mpls label-space 0
BGP Commands
MikroTik
Juniper
Cisco
IP Infusion
routing bgp peer print brief
show bgp summary
show ip bgp summary
show ip bgp summary
routing bgp peer print status
show bgp neighbor
show ip bgp neighbor
show ip bgp neighbors
routing bgp advertisements print peer=peer_name
show route advertising-protocol bgp 172.31.254.2
show ip bgp neighbor 172.31.254.2 advertised-routes
show ip bgp neighbors 172.31.254.2 advertised-routes
ip route print where received-from=peer_name
show route receive-protocol bgp 172.31.254.2
show ip bgp neighbor 172.31.254.2 received-routes
show ip bgp neighbors 172.31.254.2 received-routes
ip route print where bgp=yes
show route protocol bgp
show ip route bgp
show ip route bgp
routing bgp peer refresh peer1
clear bgp neighbor 172.31.254.2 soft-inbound
clear ip bgp 172.31.254.2 soft in
clear ip bgp 172.31.254.2 soft in
routing bgp peer resend peer1
clear bgp neighbor 172.31.254.2 soft
clear ip bgp 172.31.254.2 soft out
clear ip bgp 172.31.254.2 soft out
Let us know what other commands you would like to see in our rosetta stone to make switching network operating systems a breeze.
One of the common questions asked by MikroTik users is how to go about upgrading from ROSv6 to ROSv7.
Before upgrading, always make sure:
– The config is backed up using ‘export’ and ‘backup’ and the files have been moved off the router – Console access is working (if applicable) – A method to netinstall is available in case the upgrade fails for any reason
Understanding config migration
MikroTik added a helpful chart to the support docs that shows what config is automatically upgraded and what needs to be manually adjusted.
BGP config migration has gotten better in the last few versions of v7. For the most part, it works without intervention but occasionally config will need to be removed and readded or edited.
Note the changes below to the structure of BGP menus and peerings as it has changed.
OSPF has come a long way in RouterOS v7 and is stable as well as interoperable with RouterOSv6. Interface templates have replaced network statements to advertise prefixes and form neighbor adjacencies, so be sure to look in that menu after upgrade to work with network statements in v7. Upgrading to v7 for OSPF normally works without issue or intervention.
MPLS is still a work in progress. Like the other protocols it has gotten better but still may need adjustments since it now includes the AFIs for IPv4 and IPv6 with LDP. Be sure to review the syntax pre and post upgrade as well as the operation state and be prepared to delete and re-add the configuration as needed if MPLS is not functional post upgrade. In general, MPLS and VPLS works between ROSv6 and ROSv7
Routing filters are also a work in progress. Most of the functionality and config upgrade works now when moving to v7 but the context sensitive help and tab complete is still being developed and filled in.
Understanding how the MikroTik support process works and how to ask for help can save a lot of time and frustration when you need assistance with features, configurations, hardware or potential bugs.
MikroTik Support…where do I start?
There are a number of ways to get assistance with MikroTik devices and software including: Jira ticket support, documentation, forums, Reddit, Facebook, distributors and professional consulting. One thing to keep in mind for all correspondence with MikroTik is they are based in Riga, Latvia which is GMT+3 in the spring/summer and GMT +2 in the fall/winter.
Current time in Riga, Latvia
MikroTik Documentation
As RouterOS Version 7 was released in Beta, MikroTik began moving to Confluence for documentation instead of the Wiki.
This is probably the best place to start if you need assistance figuring out an issue or what support avenue to use.
The MikroTik forums are a great resource as long as you do a little homework.
The key to getting answers out of the forums is to provide:
– Information that describes the issue and how to repeat it (if possible)
– Configurations (edited for sensitive information)
– Drawings to help clarify your issue.
– Software versions
– Hardware being used and interop with other vendors (if relevant)
Forum members and official MikroTik support members are more likely to provide support when they can read a post and quickly offer a recommendation without having to ask lots of follow up questions.
Tips for getting the most out of the forums
Basics
Use the search feature in the forums to see if your issue has been discussed before
Use google to help with this by adding site:forum.mikrotik.com in your search.
example:
If possible, try all of the latest code versions from the Long Term, Current, Release Candidate and Beta versions to see if it resolves the issue.
Writing your posts
Read MikroTik’s suggestions for writing a forum post which includes text formatting suggestions:
ASCII drawings’ and network drawings using paint or other drawing programs make it more difficult to understand the topology.
Use a program like Visio or lucidchart.com to illustrate your network topology.
Describing the network topology (even a simple one) makes it much harder for people to help you.
If you want answers, draw it out.
Forum Etiquette
Be polite – don’t ‘demand’ answers if nobody has answered your question in a few hours. Sometimes it takes a while to get the right answers.
The more you contribute to the forum, the more likely you’ll be to get answers when you need them.
Users who only ask questions and never provide feedback or help other users don’t tend to get as much help after a while.
For the reason above, the forum should not be considered a resource to address critical and time-sensitive issues – the forums are best for issues that don’t need to be resolved immediately.
Prior to 2020, MikroTik support used e-mail ticketing to work issues which made complex issues a little harder to work on as the chain of discussion was sometimes difficult to follow.
Move to Jira
MikroTik migrated to Jira in 2020 which improved the support experience.
The key to understanding how to interact with MikroTik support is much like the advice for the forums. The more complete and well documented your ticket is, the better chance you have of getting a resolution.
The most important part of opening a ticket is to test the issue you’re experiencing on all versions of RouterOS 6 or 7 (Long Term, Stable, Release Candidate, Beta) and obtain a supout.rif for each of them.
This is very important as it will minimize a follow up e-mail from MikroTik support asking you to upgrade and then test again.
Tips for opening and managing a ticket
Provide detailed information.
– Description of the issue and the steps to repeat it.
– Network drawings.
– Configurations of other devices (if relevant)
– Packet captures (can be very helpful to identify and correct bugs
Be aware of the time difference between where you are and MikroTik (Riga, Latvia) – If you send and respond to support tickets during hours that MikroTik is awake and working, you’ll sometimes see faster responses but there is no guaranteed response time.
Waiting for bug fixes
Understand the limitations of fixing issues in RouterOS – If something can be fixed quickly, MikroTik is pretty good about getting it fixed and released.
Some issues can be patched easily and MikroTik will put them in the list for a future RouterOS release.
Some issues take longer to patch due to complexity and may be a while before they can be tested and released.
Certain issues cannot be fixed due to limitations in the Linux kernel and MikroTik will usually tell you if this is the case although with RouterOS 7 now released, this may happen less often than it did with RouterOS 6.
MikroTik Distributors
MikroTik Distributors can be a great source of support for assistance with setup and configuration as well as issues with hardware.
If you suspect that you have a hardware issue that might require an RMA, try a netinstall first to see if that corrects the issue and if it doesn’t, work with your distributor to replace the device.
The MikroTik Experts group is a fantastic source of information and news about MikroTik.
Many of the same rules as the forums apply in the Facebook group.
Be sure to search the group to see if your question has already been asked, be polite and be as detailed as possible when asking questions to get better answers.
This is actually one of the fastest ways to get answers as the group is rather large and many MikroTik consultants participate in the group and are willing to help newcomers.
Professional MikroTik Consulting
If all of the other resources don’t seem to get you the answer you’re looking for or you don’t have time to wait, consider hiring a professional MikroTik consultant.
MikroTik consultants must hold at least one engineering level certification to be listed on the consulting list.
Participation in the MikroTik forums, Attendance at MikroTik User Meetings and presentation at MikroTik User Meetings all influence the ranking of a MikroTik consultant.
