Blog

IP Infusion OcNOS 6.0: interop VPWS and VPLS

Categories
calix Interop IP Infusion

IP Infusion OcNOS 6.0: interop VPWS and VPLS

With the current state of the supply chain lead times for networking gear can be astronomical. This led consumers to look at other options for networking equipment forcing the whitebox and disaggregated networking market to become more prevalent.

With full featured operating systems like IP Infusion‘s OcNOS 6.0 and commodity hardware from Ufispace and Edgecore companies have been about to upgrade faster and further than ever before.

We’ll be looking at the ufispace 9600-32s and 9500-30xs in this deployment. This is shaping up to be a great combination for 100g and 10g density. Since both run the same operating system moving between is easy. While the bigger Qumran2c, 9600-32s, doesn’t support breakouts/10g we can aggregate and terminate 100g services here while using a small device to delivery 10g density and breakout.

We’re going to look specifically at VPLS and VPWS delivery in this deployment. Since these deployments typically complement existing deployments we’ll look at interop with a Calix e9-2 ASM 3001 deployment.

I know Calix doesn’t normally come to mind for MPLS deployments but more for FTTX or ERPS. However, they’ve been putting a lot of effort into their MPLS stack on the e9-2 ASM platform which has helped led to this testing.

We also have an Arista 7280CR3K-32P4 acting as a p-router during link failure.

IGP/LDP Setup

We’re going to run isis as an IGP which is typical in a service provider network. This time we’re going to run straight LDP instead of SR-MPLS, however, you can still utilize SR-MPLS with a mapping server if your topology supports it.

Let’s verify IGP/LDP and routing.

ASM3001# show isis neighbors
NEIGHBOR                                HOLD  CIRCUIT
SYSTEM ID       TYPE  INTERFACE  STATE  TIME  ID
-------------------------------------------------------
0010.0127.0118  L2    la3        UP     23    3
0010.0127.0119  L2    la4        UP     26    4

 details
NEIGHBOR                                HOLD  CIRCUIT
SYSTEM ID       TYPE  INTERFACE  STATE  TIME  ID
-------------------------------------------------------
0010.0127.0118  L2    la3	 UP	23    3
Hostname:ufispace-100
SNPA:e8c5.7a77.a655
State Changed:3214
LAN Priority:0
Restart Capable:1
Peer Restart State:1
0010.0127.0119  L2    la4	 UP	26    4
Hostname:ARISTA
SNPA:c4ca.2b66.fb6d
State Changed:3152
LAN Priority:0
Restart Capable:1
Peer Restart State:1
-------------------------------------------------------
ASM3001# show mpls ldp neighbors
                                                                          LOOP                      INTERFACE
INDEX  PEER LDP ID      LOCAL LDP ID     TYPE      SESSION DISTMODE       DETECTION  TRANS ADD      NAME
---------------------------------------------------------------------------------------------------------------
1      100.127.0.118:0  100.127.0.117:0  TARGETED  DownstreamUnsolicited  Disabled   100.127.0.119  none
2      100.127.0.118:0  100.127.0.117:0  DIRECTED  DownstreamUnsolicited  Disabled   100.127.0.119  la3
3      100.127.0.119:0  100.127.0.117:0  DIRECTED  DownstreamUnsolicited  Disabled   100.127.0.119  la4
ASM3001# show ip route all
ROUTE
INDEX  PREFIX            NEXT HOP       TYPE   DISTANCE  INTERFACE  UPTIME
----------------------------------------------------------------------------
1      100.126.2.160/29  100.126.2.161  local  0/0       la3        0:9:47
2      100.126.2.161/32  0.0.0.0        local  0/0       la3        0:9:47
3      100.126.2.168/29  100.126.2.169  local  0/0       la4        0:9:42
4      100.126.2.169/32  0.0.0.0        local  0/0       la4        0:9:42
5      100.126.2.176/29  100.126.2.162  isis   115/20    la3        0:9:33
6                        100.126.2.170  isis   115/20    la4        0:9:33
7      100.126.2.184/29  100.126.2.162  isis   115/20    la3        0:9:33
8      100.127.0.117/32  0.0.0.0        local  0/0       loopback1  0:9:53
9      100.127.0.118/32  100.126.2.162  isis   115/20    la3        0:9:33
10     100.127.0.119/32  100.126.2.170  isis   115/20    la4        0:9:33
11     100.127.0.120/32  100.126.2.162  isis   115/30    la3        0:9:33
ASM3001# ping 100.127.0.120
PING 100.127.0.120 (100.127.0.120) 56(84) bytes of data.
64 bytes from 100.127.0.120: icmp_seq=1 ttl=63 time=0.778 ms
64 bytes from 100.127.0.120: icmp_seq=2 ttl=63 time=0.722 ms
OcNOS-SW#show clns neighbors

Total number of L1 adjacencies: 0
Total number of L2 adjacencies: 1
Total number of adjacencies: 1
Tag UNDERLAY:  VRF : default
System Id      Interface   SNPA                State  Holdtime  Type Protocol
ufispace-100   po1         e8c5.7a77.a657      Up     26        L2   IS-IS
OcNOS-SW#show mpls ldp neighbor
IP Address      Mode          Intf Name    Holdtime   LDP-Identifier
100.126.2.185   Interface     po1         15         100.127.0.118:0
fe80::eac5:7aff:fe77:a657Interface     po1         15         100.127.0.118:0
100.127.0.117   Targeted      po1         45         100.127.0.117:0
OcNOS-SW#ping 100.127.0.117 source-ip 100.127.0.120
Press CTRL+C to exit
PING 100.127.0.117 (100.127.0.117) from 100.127.0.120 : 56(84) bytes of data.
64 bytes from 100.127.0.117: icmp_seq=1 ttl=64 time=0.811 ms
64 bytes from 100.127.0.117: icmp_seq=2 ttl=64 time=0.746 ms

Since we have LDP neighbors and loopback to loopback reachability we can begin to build our services.

100g VPWS

First we’ll build a VPWS service between the E9-2 and ufispace-100 to verify functionality. We’ll utilize a TX300s-100GX test set to push traffic through the service.

First lets look at the ASM configuration for the xconnect.

ASM3001# show running-config l2vpn
l2vpn 1
 pw-class PWE-1
  encapsulation mpls
   cc-type        TTL
   transport-mode vlan
  !
 !
 point2point 200
  xconnect-neighbor 100.127.0.118 pw-id 200
   pw-class  PWE-1
   pw-status enable
  !
 !
!

Below you can see the config for the interface facing the test set. This will put the traffic into the VPWS service.

ASM3001# show running-config interface ethernet 1/2/q7
interface ethernet 1/2/q7
 no shutdown
 role     uni
 arp arp-announce any
 l2transport
  point-to-point 200
 !
!

Now we can see the same on the IP infusion side.

ufispace-100#show run mpls
!
service-template TEST
 match all
!
mpls l2-circuit TEST-VPWS 200 100.127.0.117
ufispace-100#show run int ce17
!
interface ce17
 switchport
 mtu 1986
 mpls-l2-circuit TEST-VPWS service-template TEST primary
!

Finally lets verify functionality. I did a verbose output of the circuit details to help see all of the details. Some important things to match are the MTU and if it’s a vlan or raw service.

IP Infusion sets the MTU on the attachment circuit while Calix is inherited from the default interface value of 2000 minus some overhead.

ufispace-100#show ldp mpls-l2-circuit detail
PW ID: 200, VC state is up
Access IF: ce17,up,AC state is up
Session IF: po1, state is up
Destination: 100.127.0.117, Peer LDP Ident: 100.127.0.117
Local vctype: vlan, remote vctype :vlan
Local groupid: 0, remote groupid: 0
Local label: 24962, remote label: 26
Local MTU: 1986, Remote MTU: 1986
Local Control Word: disabled  Remote Control Word: Not-Applicable  Current use: disabled
Local Flow Label Direction: Disabled, Static: Disabled
Remote Flow Label Direction: Disabled, Static: Disabled
Local PW Status Capability : disabled
Remote PW Status Capability : enabled
Current PW Status TLV : disabled
Local VCCV Capability:
 CC-Types: None
 CV-Types: None
Remote VCCV Capability:
 CC-Types:  Type 3
 CV-Types:
 LSP ping
ASM3001# show l2vpn xconnect pw-id 200
l2vpn xconnect pw-id 200



XCONNECT NAME                     STATE
--------------------------------- ---------------
200                               Up
-------------------------------------------------

VPWS Index : 2
VPN Key    : 131074

% 1 entries in the table.


AC Details
-------------------------------------------------

INTERFACE    VLAN      STATE           TYPE       MTU       VPWS-INDEX
------------ --------- --------------- ---------- --------- -----------
1/2/q7       NA        Active          Tagged     1986      2

% 1 entries in the table.


PW Details
-------------------------------------------------

PW-ID  PW-STATE          PW-CLASS              ENCAPSULATION  PROTOCOL  ADMIN-STATE  REDUNDANCY-STATE  VPWS-INDEX
------ ----------------- --------------------- -------------- --------- ------------ ----------------- ----------
200    Up                 PWE-1                 MPLS           LDP       Up           NA                2
-----------------------------------------------------------------------------------------------------------------

PW-INFO       LOCAL                REMOTE
------------- -------------------- --------------------
Address       100.127.0.117        100.127.0.118
PW ID         200                  uNknOwn
PW type       Tagged               uNknoWn
Label         26                   24962
MTU           1986                 1986
Control Word  Disabled             uNknOwn
Status TLV    Enabled              Disabled
CC Type       4                    0
CV Type       2                    0

Local Status (PW Status TLV): 0x6
Remote Status (PW Status TLV): 0x0
Create time: 2022-09-10 09:17:23
Last time status changed: 2022-09-10 09:30:28


% 1 entries in the table.

Finally, we can see 95g of traffic across the circuit with the test set.

10g VPLS

Next we will look at a 10g VPLS service delivered off the extension switch. We already saw end to end reachability in the IGP setup so we will start with configuration.

On the ASM you build a bridge domain and tie it to a virtual forwarding instance.

ASM3001# show running-config l2vpn
l2vpn 1
 pw-class vlan-pwe
  encapsulation mpls
   cc-type        TTL
   transport-mode vlan
  !
 !

Then we define the neighbor or neighbors in the VPLS.

ASM3001# show running-config l2vpn bridge-domain
l2vpn 1
 bridge-domain 220
  mtu 9086
  vfi 220
   neighbor 100.127.0.120 pw-id 220
    pw-class vlan-pwe
   !
  !
 !
!

Again, we tie the interface facing the test kit into the bridge-domain. This will put the traffic into the VPLS instance.

ASM3001# show running-config interface ethernet 1/1/x15
interface ethernet 1/1/x15
 no shutdown
 role     uni
 arp arp-announce any
 l2transport
  rewrite-ingress tag add dot1q 220
  !
  bridge-domain 220
  !
 !
!

Then we build the same on IP Infusion.

OcNOS-SW#show run vpls
!
mpls vpls TEST-VPLS 220
 signaling ldp
  vpls-type vlan
  vpls-peer 100.127.0.117
  exit-signaling
 exit-vpls
!

Here we also have to define the peers for targeted hellos in LDP.

OcNOS-SW#show run ldp
!
router ldp
 router-id 100.127.0.120
 graceful-restart full
 targeted-peer ipv4 100.127.0.117
  exit-targeted-peer-mode
 transport-address ipv4 100.127.0.120
!

Finally, we attached the a port to the service and plug in the test kit.

OcNOS-SW#show run int xe15
!
interface xe15
 switchport
 mtu 9086
 mpls-vpls TEST-VPLS service-template TEST
  exit-if-vpls
!

Again, we will look at the verbose output and pay attention to MTU and VPLS type, vlan in this case.

OcNOS-SW#show mpls vpls detail
Virtual Private LAN Service Instance: TEST-VPLS, ID: 220
 SIG-Protocol: LDP
 Attachment-Circuit :UP
 Learning: Enabled
 Control-Word: Disabled
 Flow Label Status: Disabled, Direction: None, Static: No
 Group ID: 0, VPLS Type: Ethernet VLAN, Configured MTU: 9086
 Description: none
 service-tpid: dot1.q
 Operating mode: Tagged
 Svlan Id: 0
 Svlan Tpid: 8100
 Configured interfaces:
  Interface: xe15
 Service-template : TEST
  Match criteria : Accept all

 Mesh Peers:
   100.127.0.117 (Up)
ASM3001# show l2vpn bridge-domain bd-name 220
l2vpn bridge-domain bd-name 220

BRIDGE DOMAIN NAME                     STATE
--------------------------------- ---------------
220                               Up
-------------------------------------------------

VPLS Index         :  3
VPN Key            :  65539
MTU                :  9086
MAC Learning       :  ENABLE
MAC Aging Time     :  300
MAC Limit Max      :  1024
MAC Action         :  FLOOD
AD Type            :  NONE
SIG type           :  NONE
Transport Mode     :  ETHERNET TAGGED
Control Word       :  DISABLE
Route Distinguisher:  0x0000000000000000(NULL)
VPLS ID            :  0x0000000000000000(DEFAULT)
VE ID              :  0
VE Range           :  8
% 1 entries in the table.
AC Details
------------ --------------- ---------- ------------ ---------------
DESCRIPTION  STATE           TYPE        VPLS INDEX   SPLIT HORIZON
------------ --------------- ---------- ------------ ---------------
1.x15        Active          Ethernet    3            Disabled
% 1 entries in the table.
PW Details
PW-ID           STATE       PW-Class              ENCAPSULATION  VPLS-INDEX  ADMIN-STATE
--------------- ----------- --------------------- -------------- ----------  ------------
220             Up          vlan-pwe              MPLS           3         Up
------------------------------------------------------------------------------------

------------- -------------------- --------------------

PW            LOCAL                REMOTE
------------- -------------------- --------------------
Address       100.127.0.117        100.127.0.120
PW ID         220                  uNknOwn
PW type       Tagged               uNknoWn
Label         34                   24961
MTU           9086                 9086
Control Word  Disabled             uNknOwn
Status TLV    Enabled              Disabled
CC Type       4                    0
CV Type       2                    0

Create time: 2022-09-10 09:55:06
Last time status changed: 2022-09-10 09:58:56


% 1 entries in the table.

Finally, we can see all of the traffic on the test set across the circuit.

Conclusion

Disaggregated networking provides an alternative to traditional vendors and these are real world examples of service deployment for service providers. A special thanks to Sorin Esanu and Race Communications for organizing this test environment as a proof of concept for their deployment.

Categories
Interop IP Infusion

IP Infusion 6.0: L3VPN and 6VPE interop with Juniper

A while back we published a blog on ip infusion‘s OcNOS and MikroTik interop with segment routing mpls and LDP. Today we are going to build on the fundamentals learned there to test VPNv4 and VPNv6 interoperability with Juniper.

We’re going to start with the same topology from the previous post but with juniper instead of mikrotik. Of course juniper could run isis end to end to eliminate the redistribution but we’re going to continue to use OSPF and LDP. OSPF and LDP are largely deployed and it is likely you will come across this scenario in your path towards SR-MPLS.

MPLS/IGP Setup

The first thing to accomplish is end to end reachability between the provider edge (PE) routers.

MPLS only requires the /32s of the loopbacks for functionality so redistribution is limited to the /32 loopbacks of the PE routers. If we ran isis end to end we would not have to perform this redistribution.

ipi-2.lab.jan1.us.ipa.net#
ip prefix-list LDP-PE-LOOPBACKS
 seq 10 permit 100.127.2.0/24 eq 32
!
ip prefix-list SR-PE-LOOPBACKS
 seq 10 permit 100.127.0.0/24 eq 32
!
route-map REDIS-OSPF-TO-ISIS permit 10
 match ip address prefix-list LDP-PE-LOOPBACKS
!
route-map REDIS-ISIS-TO-OSPF permit 10
 match ip address prefix-list SR-PE-LOOPBACKS
!
router isis UNDERLAY
 is-type level-1-2
 metric-style wide
 mpls traffic-eng router-id 100.127.0.2
 mpls traffic-eng level-1
 mpls traffic-eng level-2
 capability cspf
 dynamic-hostname
 fast-reroute ti-lfa level-1 proto ipv4
 fast-reroute ti-lfa level-2 proto ipv4
 net 49.0015.1001.2700.0002.00
 redistribute ospf level-1-2 route-map REDIS-OSPF-TO-ISIS
 segment-routing mpls
!
router ospf
 ospf router-id 100.127.0.2
 redistribute isis UNDERLAY route-map REDIS-ISIS-TO-OSPF
 passive-interface lo enable
 network 100.126.0.0/29 area 0.0.0.0
 network 100.126.0.8/29 area 0.0.0.0
 network 100.127.0.2/32 area 0.0.0.0
!
ipi-1.lab.jan1.us.ipa.net#ping 100.127.2.2 source-ip 100.127.0.1
Press CTRL+C to exit
PING 100.127.2.2 (100.127.2.2) from 100.127.0.1 : 56(84) bytes of data.
64 bytes from 100.127.2.2: icmp_seq=1 ttl=63 time=6.30 ms
64 bytes from 100.127.2.2: icmp_seq=2 ttl=63 time=2.09 ms
64 bytes from 100.127.2.2: icmp_seq=3 ttl=63 time=5.83 ms

Now that we have loopback to loopback reachability we will stitch together the LDP and segment routing (SR) domains.

IPI-1 is going to server as a segment routing mapping server. This will assign labels to the routes in the LDP label space and distribute them to through the SR domain so we can have an end to end label switched path enabling the use of MPLS services.

segment-routing
 mpls sr-prefer
 global block 16000 23999
 mapping-server
  srms preference 100
  prefix-sid-map address-family ipv4
   100.127.2.0/32 4000 range 256
  exit-ms-af
 exit-ms

This will start with prefix 100.127.2.0/32 add 4000 to the segment routing global block starting point (16000 as defined) and be able to label the next 256 routes in order. i.e. 100.127.2.1/32 gets the node sid 20001. IPI-2 shows the stitching in action.

Let’s look at the label space.

ipi-2.lab.jan1.us.ipa.net#show mpls ilm-table
Codes: > - installed ILM, * - selected ILM, p - stale ILM
        K - CLI ILM, T - MPLS-TP, s - Stitched ILM
       S - SNMP, L - LDP, R - RSVP, C - CRLDP
       B - BGP , K - CLI , V - LDP_VC, I - IGP_SHORTCUT
       O - OSPF/OSPF6 SR, i - ISIS SR, k - SR CLI
       P - SR Policy, U - unknown

Code    FEC/VRF/L2CKT    ILM-ID      In-Label    Out-Label   In-Intf    Out-Intf
/VRF       Nexthop                   LSP-Type
   i>   100.127.0.1/32     4           16101       3           N/A        xe48
           100.126.0.1               LSP_DEFAULT
   B>   evpn:1             3           17          Nolabel     N/A        N/A
           127.0.0.1                 LSP_DEFAULT
   B>   evpn:2             1           16          Nolabel     N/A        N/A
           127.0.0.1                 LSP_DEFAULT
   B>   evpn:1             2           640         Nolabel     N/A        N/A
           127.0.0.1                 LSP_DEFAULT
   L>   100.127.0.1/32     8           24961       3           N/A        xe48
           100.126.0.1               LSP_DEFAULT
 s i>   100.127.2.2/32     9           20002       3           N/A        xe47.1
           100.126.0.10              LSP_DEFAULT
   i>   100.127.0.2/32     5           16102       Nolabel     N/A        N/A
           127.0.0.1                 LSP_DEFAULT
   i>   100.126.0.1/32     6           25600       3           N/A        xe48
           100.126.0.1               LSP_DEFAULT
   L>   100.127.2.2/32     10          24962       3           N/A        xe47.1
           100.126.0.10              LSP_DEFAULT

The highlighted label above shows the stitching in action as denoted by the s. IMPORTANT mpls lsp-stitching needs to be enabled on any router in the SR and LDP domain.

IPI-1 only sees isis-sr labels and QFX-2 only sees LDP labels as shown below.

ipi-1.lab.jan1.us.ipa.net#show mpls ilm-table
Codes: > - installed ILM, * - selected ILM, p - stale ILM
        K - CLI ILM, T - MPLS-TP, s - Stitched ILM
       S - SNMP, L - LDP, R - RSVP, C - CRLDP
       B - BGP , K - CLI , V - LDP_VC, I - IGP_SHORTCUT
       O - OSPF/OSPF6 SR, i - ISIS SR, k - SR CLI
       P - SR Policy, U - unknown

Code    FEC/VRF/L2CKT    ILM-ID      In-Label    Out-Label   In-Intf    Out-Intf
/VRF       Nexthop                   LSP-Type
   i>   100.127.0.1/32     4           16101       Nolabel     N/A        N/A
           127.0.0.1                 LSP_DEFAULT
   B>   evpn:1             3           17          Nolabel     N/A        N/A
           127.0.0.1                 LSP_DEFAULT
   B>   evpn:100           1           16          Nolabel     N/A        N/A
           127.0.0.1                 LSP_DEFAULT
   B>   evpn:1             2           640         Nolabel     N/A        N/A
           127.0.0.1                 LSP_DEFAULT
   P>   100.127.0.2/32     7           20          3           N/A        xe48
           100.126.0.2               LSP_DEFAULT
   i>   100.127.2.2/32     8           20002       20002       N/A        xe48
           100.126.0.2               LSP_DEFAULT
   i>   100.127.0.2/32     6           16102       3           N/A        xe48
           100.126.0.2               LSP_DEFAULT
   i>   100.126.0.2/32     5           26240       3           N/A        xe48
           100.126.0.2               LSP_DEFAULT
   B>   VOICE              9           24960       Nolabel     N/A        VOICE
           N/A                       LSP_DEFAULT
[email protected]# run show route table inet.3

inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

100.126.0.0/29     *[LDP/9] 01:14:11, metric 1
                    >  to 100.126.0.9 via xe-0/0/47.0
100.127.0.1/32     *[LDP/9] 01:14:11, metric 1
                    >  to 100.126.0.9 via xe-0/0/47.0, Push 24961
100.127.0.2/32     *[LDP/9] 01:14:11, metric 1
                    >  to 100.126.0.9 via xe-0/0/47.0

L3VPN

It is common to place a voice services into a layer 3 vpn. So first we’ll setup a VPNv4 session. Then build the vrf and test end to end reachability.

ipi-1.lab.jan1.us.ipa.net

ip vrf VOICE
 rd 100.127.0.1:12
 route-target both 65000:12
!
interface xe46
 speed 10g
 ip vrf forwarding VOICE
 ip address 172.16.0.1/24
 ipv6 address 2001:db8::1/64
!
router bgp 65000
 neighbor 100.127.2.2 remote-as 65000
 neighbor 100.127.2.2 update-source lo
 !
 address-family vpnv4 unicast
 neighbor 100.127.2.2 activate
 exit-address-family
!
 address-family ipv4 vrf VOICE
 redistribute connected
 exit-address-family
 !
[email protected]# show routing-instances
VOICE {
    instance-type vrf;
    interface xe-0/0/47.10;
    interface xe-0/0/47.100;
    route-distinguisher 100.127.2.2:1;
    vrf-target target:65000:12;
    vrf-table-label;
}
[email protected]# show interfaces xe-0/0/47
vlan-tagging;
unit 0 {
    vlan-id 1;
    family inet {
        address 100.126.0.10/29;
    }
    family iso;
    family mpls;
}
unit 10 {
    vlan-id 10;
    family inet {
        address 172.16.2.1/24;
    }
    family inet6 {
        address 2001:db8:0:1::1/64;
    }
}
[email protected]# show protocols bgp
group VPN {
    family inet-vpn {
        unicast;
    }
    family inet6-vpn {
        unicast;
    }
    export PS-DIRECT;
    peer-as 65000;
    neighbor 100.127.0.1;
}
local-address 100.127.2.2;
local-as 65000;

We are building the vrf VOICE with route-target import and export 65000:12 for membership.

ipi-1.lab.jan1.us.ipa.net#show ip bgp vpnv4 all summary
BGP router identifier 100.127.0.1, local AS number 65000
BGP table version is 2
1 BGP AS-PATH entries
0 BGP community entries

Neighbor                 V   AS   MsgRcv    MsgSen TblVer   InQ   OutQ    Up/Down   State/PfxRcd
100.127.2.2              4 65000  368        354       2      0      0  02:09:07               2


Total number of neighbors 1

Total number of Established sessions 1
[email protected]# run show bgp summary
Threading mode: BGP I/O
Groups: 1 Peers: 1 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
bgp.l3vpn.0
                       1          1          0          0          0          0
bgp.l3vpn-inet6.0
                       1          1          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
100.127.0.1           65000        319        313       0       7     2:14:50 Establ
  bgp.l3vpn.0: 1/1/1/0
  bgp.l3vpn-inet6.0: 1/1/1/0
  VOICE.inet.0: 1/1/1/0
  VOICE.inet6.0: 1/1/1/0
ipi-1.lab.jan1.us.ipa.net#show ip bgp vrf VOICE
BGP table version is 1, local router ID is 172.16.0.1
Status codes: s suppressed, d damped, h history, a add-path, * valid, > best, i - internal,
              l - labeled, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

    Network          Next Hop            Metric    LocPrf   Weight Path
*> l 172.16.0.0/24    0.0.0.0              0        100       32768  ?
*>i  172.16.2.0/24    100.127.2.2          0        100       0    i
*>i  192.168.0.0      100.127.2.2          0        100       0    i

Total number of prefixes 3
[email protected]# run show route table VOICE.inet.0

VOICE.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.16.0.0/24      *[BGP/170] 01:35:07, localpref 100, from 100.127.0.1
                      AS path: ?, validation-state: unverified
                    >  to 100.126.0.9 via xe-0/0/47.0, Push 24960, Push 24961(top)
172.16.2.0/24      *[Direct/0] 02:08:32
                    >  via xe-0/0/47.10
172.16.2.1/32      *[Local/0] 02:08:32
                       Local via xe-0/0/47.10
192.168.0.0/24     *[Direct/0] 1w0d 00:03:04
                    >  via xe-0/0/47.100
192.168.0.2/32     *[Local/0] 1w0d 00:03:04
                       Local via xe-0/0/47.100

We can see that we are receiving three prefixes from QFX-2 and reachability is confirmed below.

ipi-1.lab.jan1.us.ipa.net#ping 172.16.2.1 vrf VOICE
Press CTRL+C to exit
PING 172.16.2.1 (172.16.2.1) 56(84) bytes of data.
64 bytes from 172.16.2.1: icmp_seq=1 ttl=64 time=3.02 ms
64 bytes from 172.16.2.1: icmp_seq=2 ttl=64 time=6.87 ms

6VPE

6VPE is a great way to take advantage of your already deployed and operational MPLS core. Here we’re going to use the sr-mpls/LDP setup to build an IPv6 service on top, again for vrf VOICE.

Most of the configuration elements are the same except for utilizing VPNv6 and IPv6 addressing. Let’s look specifically at those portions.

ipi-1.lab.jan1.us.ipa.net#show run bgp
!
router bgp 65000
 neighbor 100.127.2.2 remote-as 65000
 neighbor 100.127.2.2 update-source lo
 !
 address-family vpnv6 unicast
 neighbor 100.127.2.2 activate
 exit-address-family
 !
 address-family ipv6 vrf VOICE
 redistribute connected
 exit-address-family
!
[email protected]# show protocols bgp
group VPN {
    family inet-vpn {
        unicast;
    }
    family inet6-vpn {
        unicast;
    }
    export PS-DIRECT;
    peer-as 65000;
    neighbor 100.127.0.1;
}
local-address 100.127.2.2;
local-as 65000;
[email protected]# show protocols mpls
ipv6-tunneling;

I will draw attention to the ipv6-tunneling command on QFX-2 above. This is required to tunnel the IPv6 packets over the MPLS core. Without this you will end up with an unusable next hop.

ipi-1.lab.jan1.us.ipa.net#show ip bgp vpnv6 all summary
BGP router identifier 100.127.0.1, local AS number 65000
BGP table version is 2
1 BGP AS-PATH entries
0 BGP community entries

Neighbor                 V   AS   MsgRcv    MsgSen TblVer   InQ   OutQ    Up/Down   State/PfxRcd
100.127.2.2              4 65000  393        380       2      0      0  02:20:07               1

Total number of neighbors 1

Total number of Established sessions 1
[email protected]# run show bgp summary
Threading mode: BGP I/O
Groups: 1 Peers: 1 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
bgp.l3vpn.0
                       1          1          0          0          0          0
bgp.l3vpn-inet6.0
                       1          1          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
100.127.0.1           65000        319        313       0       7     2:14:50 Establ
  bgp.l3vpn.0: 1/1/1/0
  bgp.l3vpn-inet6.0: 1/1/1/0
  VOICE.inet.0: 1/1/1/0
  VOICE.inet6.0: 1/1/1/0
ipi-1.lab.jan1.us.ipa.net#show ip bgp vpnv6 vrf VOICE
Status codes: s suppressed, d damped, h history, a add-path, * valid, > best, i - internal, l - lab
eled
              S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

     Network          Next Hop                    Metric    LocPrf      Weight       Path
Route Distinguisher: 100.127.0.1:12 (Default for VRF VOICE)
*> l 2001:db8::/64    ::                            0        100       32768  ?
*>i  2001:db8:0:1::/64
                      ::ffff:100.127.2.2            0        100       0    i
 Announced routes count = 1
 Accepted routes count = 1
[email protected]# run show route table VOICE.inet6.0

VOICE.inet6.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2001:db8::/64      *[BGP/170] 01:43:15, localpref 100, from 100.127.0.1
                      AS path: ?, validation-state: unverified
                    >  to 100.126.0.9 via xe-0/0/47.0, Push 24960, Push 24961(top)
2001:db8:0:1::/64  *[Direct/0] 01:49:56
                    >  via xe-0/0/47.10
2001:db8:0:1::1/128*[Local/0] 01:49:56
                       Local via xe-0/0/47.10
fe80::86c1:c100:a32:5032/128
                   *[Local/0] 01:49:56
                       Local via xe-0/0/47.10
ff02::2/128        *[INET6/0] 1w2d 20:22:10
                       MultiRecv

{master:0}[edit]

Now that we have all over our routes and signaling setup lets verify reachability.

ipi-1.lab.jan1.us.ipa.net#ping ipv6 2001:db8:0:1::1 vrf VOICE
Press CTRL+C to exit
PING 2001:db8:0:1::1(2001:db8:0:1::1) 56 data bytes
64 bytes from 2001:db8:0:1::1: icmp_seq=1 ttl=64 time=2.80 ms
64 bytes from 2001:db8:0:1::1: icmp_seq=2 ttl=64 time=3.84 ms

Conclusion

With the new features and support in IP Infusion GA 6.0 stability and interop between vendors is becoming better. This paves the way for deployments of segment-routing while migrating the LDP segments all while maintaining current VPNs.

The 6VPE support allows for rapid adoption of IPv6 utilizing the core already in place.

We’ll be working on more interop testing to include L2VPN with both LDP signaled VPLS and BGP signaled VPLS. Be sure to come back for more.

Categories
MikroTik RouterOSv7

MikroTik RouterOS v7.5 stable released

The pace of development for MikroTik RouterOS version 7 has definitely sped up in 2022 and we are seeing the results in improved stability and features added.

As of August 31st, 2022, MikroTik moved ROS v7.5rc2 into v7.5 stable

MikroTik Routers and Wireless – Software

Noteworthy additions

dhcpv6-relay – not being able to relay a PD request from a delegating router for IPv6 has been a limitation of MikroTik routers for a while so getting this fixed has a big impact on scaling MikroTik IPv6 deployments

RTSP helper – The addition of a Real Time Streaming Protocol helper is a great addition to ROSv7 to make NAT traversal for realtime applications (IPTV, SIP and IP cameras) easier.

A good overview of the discussion leading up to the addition of RTSP is here: RTSP Helper – MikroTik

l3hw – fixed hw offloaded NAT – This feature still has some issues as IP ArchiTechs recently filed a bug (SUP-91389) where src-nat traffic that carries an H flag in the connection table will die after 1 hour with a 10G load on the router. Once this feature receives further bug fixes and testing, it’s going to be very useful for high capacity but low cost NAT44 gateways.

lte – this category got a significant amount of development work as there are numerous fixes with many relating to the Chateau devices.

wifiwave2 – There was also a significant amount of development in wifi wave 2 which included notable additions like 802.11k for roaming.

 vrrp – added “sync-connection-tracking” compatibility with preemption-mode – this is a long awaited feature that showed up early in ROSv7 but did not have pre-emption mode capabilities. The addition of connection synching between routers positions MikroTik routers much closer to traditional enterprise firewall vendors so that failover between devices can include connections.



Categories
network operating systems

Networking CLI Rosetta Stone

Changing between network operating systems is one of the most challenging things for new engineers. Most people learned cisco cli due to their extensive training system or got on the job training for whatever their company runs.

We are hoping to make moving back and forth between network operating systems a little easier with some useful show and operational commands for Mikrotik, Juniper, Cisco, and IP Infusion. There are in detail usages of these commands on stubarea51.net.

OSPF Commands

MikroTikJuniperCiscoIP Infusion
routing ospf neighbor printshow ospf neighborshow ip ospf neighborshow ip ospf neighbor
routing ospf interface printshow ospf interfaceshow ip ospf interfaceshow ip ospf interface
routing ospf instance print detailshow ospf overview briefshow ip ospf 1show ip ospf 1
routing ospf lsa printshow ospf databaseshow ip ospf databaseshow ip ospf database
ip route print where ospf=yesshow route protocol ospfshow ip route ospfshow ip route ospf

routing ospf area-border-router print
show ospf route abrshow ip ospf border-routersshow ip ospf border-routers

routing ospf as-border-router print
show ospf route asbrshow ip ospf border-routersshow ip ospf border-routers

MPLS – LDP Commands

MikrotikJuniperCiscoIP Infusion
mpls ldp neighbor printshow ldp neighborshow mpls ldp neighborshow mpls ldp neighbor
mpls ldp interface printshow ldp interfaceshow mpls interfacesshow ldp interface
mpls forwarding-table printshow route forwarding-table family mplsshow mpls forwarding-tableshow mpls forwarding-table
mpls remote-bindings printshow ldp databaseshow mpls bindingshow mpls ilm-table
mpls local-bindings printshow ldp databasesh mpls ip binding localshow mpls ilm-table
mpls printshow mpls label usagesh mpls ldp parametersshow mpls label-space 0

BGP Commands

MikroTikJuniperCiscoIP Infusion
routing bgp peer print briefshow bgp summaryshow ip bgp summaryshow ip bgp summary
routing bgp peer print statusshow bgp neighborshow ip bgp neighborshow ip bgp neighbors
routing bgp advertisements print peer=peer_nameshow route advertising-protocol bgp 172.31.254.2show ip bgp neighbor 172.31.254.2 advertised-routesshow ip bgp neighbors 172.31.254.2 advertised-routes
ip route print where received-from=peer_nameshow route receive-protocol bgp 172.31.254.2show ip bgp neighbor 172.31.254.2 received-routesshow ip bgp neighbors 172.31.254.2 received-routes
ip route print where bgp=yesshow route protocol bgpshow ip route bgpshow ip route bgp
routing bgp peer refresh peer1clear bgp neighbor 172.31.254.2 soft-inboundclear ip bgp 172.31.254.2 soft inclear ip bgp 172.31.254.2 soft in
routing bgp peer resend peer1clear bgp neighbor 172.31.254.2 softclear ip bgp 172.31.254.2 soft outclear ip bgp 172.31.254.2 soft out

Let us know what other commands you would like to see in our rosetta stone to make switching network operating systems a breeze.

Categories
MikroTik RouterOSv7

MikroTik: Upgrading from ROSv6 to ROSv7

One of the common questions asked by MikroTik users is how to go about upgrading from ROSv6 to ROSv7.

Before upgrading, always make sure:

– The config is backed up using ‘export’ and ‘backup’ and the files have been moved off the router
– Console access is working (if applicable)
– A method to netinstall is available in case the upgrade fails for any reason

Understanding config migration

MikroTik added a helpful chart to the support docs that shows what config is automatically upgraded and what needs to be manually adjusted.

Upgrading to v7 – RouterOS – MikroTik Documentation

Exceptions and notes

BGP config migration has gotten better in the last few versions of v7. For the most part, it works without intervention but occasionally config will need to be removed and readded or edited.

Note the changes below to the structure of BGP menus and peerings as it has changed.


OSPF has come a long way in RouterOS v7 and is stable as well as interoperable with RouterOSv6. Interface templates have replaced network statements to advertise prefixes and form neighbor adjacencies, so be sure to look in that menu after upgrade to work with network statements in v7. Upgrading to v7 for OSPF normally works without issue or intervention.

MPLS is still a work in progress. Like the other protocols it has gotten better but still may need adjustments since it now includes the AFIs for IPv4 and IPv6 with LDP. Be sure to review the syntax pre and post upgrade as well as the operation state and be prepared to delete and re-add the configuration as needed if MPLS is not functional post upgrade. In general, MPLS and VPLS works between ROSv6 and ROSv7

Routing filters are also a work in progress. Most of the functionality and config upgrade works now when moving to v7 but the context sensitive help and tab complete is still being developed and filled in.

For more details, take a look at this article: MikroTik – RouterOSv7 first look – feedback on routing filters – StubArea51.net

User manager has no direct upgrade path available and must be migrated manually.

Categories
MikroTik Network Engineering

Understanding the MikroTik Support process

Understanding how the MikroTik support process works and how to ask for help can save a lot of time and frustration when you need assistance with features, configurations, hardware or potential bugs.

MikroTik Support…where do I start?

There are a number of ways to get assistance with MikroTik devices and software including: Jira ticket support, documentation, forums, Reddit, Facebook, distributors and professional consulting. One thing to keep in mind for all correspondence with MikroTik is they are based in Riga, Latvia which is GMT+3 in the spring/summer and GMT +2 in the fall/winter.

Current time in Riga, Latvia

MikroTik Documentation

As RouterOS Version 7 was released in Beta, MikroTik began moving to Confluence for documentation instead of the Wiki.

RouterOS – RouterOS – MikroTik Documentation


The information in the new documentation is better organized and the visuals are all being updated to give the docs a consistent feel.

Here is an example from the OSPF section under routing protocols:



MikroTik Forums

MikroTik – Forum index

This is probably the best place to start if you need assistance figuring out an issue or what support avenue to use.

The MikroTik forums are a great resource as long as you do a little homework.

The key to getting answers out of the forums is to provide:

  • – Information that describes the issue and how to repeat it (if possible)
  • – Configurations (edited for sensitive information)
  • – Drawings to help clarify your issue.
  • – Software versions
  • – Hardware being used and interop with other vendors (if relevant)


Forum members and official MikroTik support members are more likely to provide support when they can read a post and quickly offer a recommendation without having to ask lots of follow up questions.

Tips for getting the most out of the forums

Basics

Use the search feature in the forums to see if your issue has been discussed before

Use google to help with this by adding site:forum.mikrotik.com in your search.

example:

If possible, try all of the latest code versions from the Long Term, Current, Release Candidate and Beta versions to see if it resolves the issue.

Writing your posts

Read MikroTik’s suggestions for writing a forum post which includes text formatting suggestions:

Getting the most out of the MT forum

ASCII drawings’ and network drawings using paint or other drawing programs make it more difficult to understand the topology.

Use a program like Visio or lucidchart.com to illustrate your network topology.

Describing the network topology (even a simple one) makes it much harder for people to help you.

If you want answers, draw it out.

Forum Etiquette

Be polite – don’t ‘demand’ answers if nobody has answered your question in a few hours. Sometimes it takes a while to get the right answers.

The more you contribute to the forum, the more likely you’ll be to get answers when you need them.

Users who only ask questions and never provide feedback or help other users don’t tend to get as much help after a while.

For the reason above, the forum should not be considered a resource to address critical and time-sensitive issues – the forums are best for issues that don’t need to be resolved immediately.


Official MikroTik support – Jira

Service Management (mikrotik.com)

Prior to 2020, MikroTik support used e-mail ticketing to work issues which made complex issues a little harder to work on as the chain of discussion was sometimes difficult to follow.

Move to Jira

MikroTik migrated to Jira in 2020 which improved the support experience.



The key to understanding how to interact with MikroTik support is much like the advice for the forums. The more complete and well documented your ticket is, the better chance you have of getting a resolution.

The most important part of opening a ticket is to test the issue you’re experiencing on all versions of RouterOS 6 or 7 (Long Term, Stable, Release Candidate, Beta) and obtain a supout.rif for each of them.

This is very important as it will minimize a follow up e-mail from MikroTik support asking you to upgrade and then test again.

Tips for opening and managing a ticket

Provide detailed information.

  1. – Description of the issue and the steps to repeat it.
  2. – Network drawings.
  3. – Configurations of other devices (if relevant)
  4. – Packet captures (can be very helpful to identify
    and correct bugs

Be aware of the time difference between where you are and MikroTik (Riga, Latvia) – If you send and respond to support tickets during hours that MikroTik is awake and working, you’ll sometimes see faster responses but there is no guaranteed response time.

Waiting for bug fixes

Understand the limitations of fixing issues in RouterOS – If something can be fixed quickly, MikroTik is pretty good about getting it fixed and released.

Some issues can be patched easily and MikroTik will put them in the list for a future RouterOS release.

Some issues take longer to patch due to complexity and may be a while before they can be tested and
released.

Certain issues cannot be fixed due to limitations in the Linux kernel and MikroTik will usually tell you if this is the case although with RouterOS 7 now released, this may happen less often than it did with RouterOS 6.

MikroTik Distributors

MikroTik Distributors can be a great source of support for assistance with setup and configuration as well as issues with hardware.

If you suspect that you have a hardware issue that might require an RMA, try a netinstall first to see if that
corrects the issue and if it doesn’t, work with your distributor to replace the device.

Netinstall – RouterOS – MikroTik Documentation

MikroTik Experts (Unofficial) Facebook group

Mikrotik Experts | Facebook

The MikroTik Experts group is a fantastic source of information and news about MikroTik.

Many of the same rules as the forums apply in the Facebook group.

Be sure to search the group to see if your question has already been asked, be polite and be as detailed as possible when asking questions to get better answers.

This is actually one of the fastest ways to get answers as the group is rather large and many MikroTik consultants participate in the group and are willing to help newcomers.

Professional MikroTik Consulting

If all of the other resources don’t seem to get you the answer you’re looking for or you don’t have time to
wait, consider hiring a professional MikroTik consultant.

MikroTik consultants must hold at least one engineering level certification to be listed on the consulting list.

Participation in the MikroTik forums, Attendance at MikroTik User Meetings and presentation at MikroTik
User Meetings all influence the ranking of a MikroTik consultant.

https://www.mikrotik.com/consultants

IP ArchiTechs is the largest MikroTik consulting company in the world and has certified MikroTik consultants on multiple continents.

https://iparchitechs.com/contact-ip-architechs

Categories
Uncategorized

App delivery for an improved pizza experience

It’s been a while since we started work on one of our newest projects.  We have been trying to solve a problem in app location.  It all came from the notion that Little Caesars know where my pizza is, so why can’t the network resolve where the app is?    We also thought it would be novel use of Anycast because the app can be anywhere.
So, what problems specifically have we solved using this design?  Intent based gateways are a signaling mechanism allows the apps to be delivered along with the pizza.  As we can see app Buffalo Wings can reach both the intent based gateway and Fried Pickles using TI-LFA, which strips the fat bits before they reach the gateway.   Our unique caching solution using Tupperware, which are stacked in K8s, allows for the apps to be delivered in a bursty nexthop specific competitive manner.  This has proven to keep the apps warm within the physical layer.
In our example, the Delivery Center Interconnect,  we are doing an east to west Multi Pizza Layered Service that can drop the apps with full BTU into any of the regions.  The apps are unaware of the topping layer and rely on layer 2 media skipping protocol to travel between regions.  As you can see, pineapple pizza is restricted to the west security zone because its traffic is otherwise discarded.  Flows between Brooklyn and deep dish can be maintained because the overlay is based on ZeroTabasco.
A full mesh is achieved with the IGulP maintained by IS-IS.  In this example a choice of which app is being forced so FFR can redirect service when the apps fail to reach the intent based gateway