Mikrotik 40Gb Interface Configuration - 4 ports?

IamSpartacus

Well-Known Member
Mar 14, 2016
2,301
553
113
Can someone with a Mikrotik switch with 40Gb interfaces confirm for me the correct way to configure the 40Gb NIC(s)? The way the ports show up in the switch config is as follows:

Code:
qsfpplus1-1
qsfpplus1-2
qsfpplus1-3
qsfpplus1-4
qsfpplus2-1
qsfpplus2-2
qsfpplus2-3
qsfpplus2-4
I'm connecting these ports to a Mellanox ConnectX-3 VPI dual NIC in a server.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,301
553
113
Guess not too many people have this switch huh?

@Patrick did you run into any issues when testing 40Gb on these Mikrotik switches?
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,968
4,931
113
Thanks for tagging me. No issues but with that 40GbE above you are seeing the four 10Gbps lanes. That is how QSFP+/ 40GbE is architected. So qsfpplusX-Y where X is the port number and Y is the 10GbE channel.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,301
553
113
Thanks for tagging me. No issues but with that 40GbE above you are seeing the four 10Gbps lanes. That is how QSFP+/ 40GbE is architected. So qsfpplusX-Y where X is the port number and Y is the 10GbE channel.
Thanks for the reply @Patrick . Do you remember if you bonded the 4 ports or just configured them as their own separate interfaces?
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,301
553
113
So I did confirm that my switch qsfpplus ports are configured correctly and the switch does detect the links at 40Gbps but when I do iperf3 testing between hosts connected to the two ports I'm seeing only about 2-3Gbps. I confirmed hw switching is enabled on all the ports to ensure the CPU is not being taxed for L2 switching. I did the same iperf3 test between the hosts when I took one of the two cables and direct connected the NICs. You can see the two results below:

Hosts connected via Mikrotik switch
Code:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   277 MBytes   232 Mbits/sec  2050             sender
[  5]   0.00-10.00  sec   276 MBytes   231 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   248 MBytes   208 Mbits/sec  1750             sender
[  7]   0.00-10.00  sec   248 MBytes   208 Mbits/sec                  receiver
[  9]   0.00-10.00  sec   569 MBytes   477 Mbits/sec  4135             sender
[  9]   0.00-10.00  sec   568 MBytes   477 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec   154 MBytes   129 Mbits/sec  1138             sender
[ 11]   0.00-10.00  sec   153 MBytes   129 Mbits/sec                  receiver
[ 13]   0.00-10.00  sec   236 MBytes   198 Mbits/sec  1757             sender
[ 13]   0.00-10.00  sec   235 MBytes   198 Mbits/sec                  receiver
[ 15]   0.00-10.00  sec   175 MBytes   147 Mbits/sec  1275             sender
[ 15]   0.00-10.00  sec   173 MBytes   145 Mbits/sec                  receiver
[ 17]   0.00-10.00  sec   545 MBytes   457 Mbits/sec  3992             sender
[ 17]   0.00-10.00  sec   544 MBytes   456 Mbits/sec                  receiver
[ 19]   0.00-10.00  sec   570 MBytes   478 Mbits/sec  4440             sender
[ 19]   0.00-10.00  sec   569 MBytes   477 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  2.71 GBytes  2.33 Gbits/sec  20537             sender
[SUM]   0.00-10.00  sec  2.70 GBytes  2.32 Gbits/sec                  receiver

Hosts direct connected
Code:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.52 GBytes  4.74 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  5.52 GBytes  4.74 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  5.42 GBytes  4.66 Gbits/sec    0             sender
[  7]   0.00-10.00  sec  5.42 GBytes  4.65 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  5.35 GBytes  4.60 Gbits/sec    0             sender
[  9]   0.00-10.00  sec  5.35 GBytes  4.59 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  5.51 GBytes  4.73 Gbits/sec    0             sender
[ 11]   0.00-10.00  sec  5.51 GBytes  4.73 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  5.48 GBytes  4.71 Gbits/sec    0             sender
[ 13]   0.00-10.00  sec  5.48 GBytes  4.71 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  5.50 GBytes  4.73 Gbits/sec    0             sender
[ 15]   0.00-10.00  sec  5.50 GBytes  4.73 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  5.42 GBytes  4.66 Gbits/sec    0             sender
[ 17]   0.00-10.00  sec  5.42 GBytes  4.66 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  5.38 GBytes  4.62 Gbits/sec    0             sender
[ 19]   0.00-10.00  sec  5.37 GBytes  4.61 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  43.6 GBytes  37.4 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  43.6 GBytes  37.4 Gbits/sec                  receiver
 
  • Like
Reactions: Aluminat

zack$

Active Member
Aug 16, 2018
477
183
43
AFAIK hw switching is supported on certain modes. LACP, I believe, does support it.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,301
553
113
AFAIK hw switching is supported on certain modes. LACP, I believe, does support it.
Are you suggesting that I bond the four qsfpplus ports (qsfpplus1-1,qsfpplus1-2,qsfpplus1-3,qsfpplus1-4)? I don't have them bonded currently.
 
Last edited: