Mikrotik 40Gb Interface Configuration - 4 ports?

[IamSpartacus]

Can someone with a Mikrotik switch with 40Gb interfaces confirm for me the correct way to configure the 40Gb NIC(s)? The way the ports show up in the switch config is as follows:

qsfpplus1-1
qsfpplus1-2
qsfpplus1-3
qsfpplus1-4
qsfpplus2-1
qsfpplus2-2
qsfpplus2-3
qsfpplus2-4

I'm connecting these ports to a Mellanox ConnectX-3 VPI dual NIC in a server.

 

[IamSpartacus]

Guess not too many people have this switch huh?

@Patrick did you run into any issues when testing 40Gb on these Mikrotik switches?

 

[Patrick]

Thanks for tagging me. No issues but with that 40GbE above you are seeing the four 10Gbps lanes. That is how QSFP+/ 40GbE is architected. So qsfpplusX-Y where X is the port number and Y is the 10GbE channel.

 

[IamSpartacus]

Thanks for tagging me. No issues but with that 40GbE above you are seeing the four 10Gbps lanes. That is how QSFP+/ 40GbE is architected. So qsfpplusX-Y where X is the port number and Y is the 10GbE channel.

Thanks for the reply @Patrick . Do you remember if you bonded the 4 ports or just configured them as their own separate interfaces?

 

[IamSpartacus]

So I did confirm that my switch qsfpplus ports are configured correctly and the switch does detect the links at 40Gbps but when I do iperf3 testing between hosts connected to the two ports I'm seeing only about 2-3Gbps. I confirmed hw switching is enabled on all the ports to ensure the CPU is not being taxed for L2 switching. I did the same iperf3 test between the hosts when I took one of the two cables and direct connected the NICs. You can see the two results below:

Hosts connected via Mikrotik switch

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   277 MBytes   232 Mbits/sec  2050             sender
[  5]   0.00-10.00  sec   276 MBytes   231 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   248 MBytes   208 Mbits/sec  1750             sender
[  7]   0.00-10.00  sec   248 MBytes   208 Mbits/sec                  receiver
[  9]   0.00-10.00  sec   569 MBytes   477 Mbits/sec  4135             sender
[  9]   0.00-10.00  sec   568 MBytes   477 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec   154 MBytes   129 Mbits/sec  1138             sender
[ 11]   0.00-10.00  sec   153 MBytes   129 Mbits/sec                  receiver
[ 13]   0.00-10.00  sec   236 MBytes   198 Mbits/sec  1757             sender
[ 13]   0.00-10.00  sec   235 MBytes   198 Mbits/sec                  receiver
[ 15]   0.00-10.00  sec   175 MBytes   147 Mbits/sec  1275             sender
[ 15]   0.00-10.00  sec   173 MBytes   145 Mbits/sec                  receiver
[ 17]   0.00-10.00  sec   545 MBytes   457 Mbits/sec  3992             sender
[ 17]   0.00-10.00  sec   544 MBytes   456 Mbits/sec                  receiver
[ 19]   0.00-10.00  sec   570 MBytes   478 Mbits/sec  4440             sender
[ 19]   0.00-10.00  sec   569 MBytes   477 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  2.71 GBytes  2.33 Gbits/sec  20537             sender
[SUM]   0.00-10.00  sec  2.70 GBytes  2.32 Gbits/sec                  receiver

Hosts direct connected

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.52 GBytes  4.74 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  5.52 GBytes  4.74 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  5.42 GBytes  4.66 Gbits/sec    0             sender
[  7]   0.00-10.00  sec  5.42 GBytes  4.65 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  5.35 GBytes  4.60 Gbits/sec    0             sender
[  9]   0.00-10.00  sec  5.35 GBytes  4.59 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  5.51 GBytes  4.73 Gbits/sec    0             sender
[ 11]   0.00-10.00  sec  5.51 GBytes  4.73 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  5.48 GBytes  4.71 Gbits/sec    0             sender
[ 13]   0.00-10.00  sec  5.48 GBytes  4.71 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  5.50 GBytes  4.73 Gbits/sec    0             sender
[ 15]   0.00-10.00  sec  5.50 GBytes  4.73 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  5.42 GBytes  4.66 Gbits/sec    0             sender
[ 17]   0.00-10.00  sec  5.42 GBytes  4.66 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  5.38 GBytes  4.62 Gbits/sec    0             sender
[ 19]   0.00-10.00  sec  5.37 GBytes  4.61 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  43.6 GBytes  37.4 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  43.6 GBytes  37.4 Gbits/sec                  receiver

 

[zack$]

AFAIK hw switching is supported on certain modes. LACP, I believe, does support it.

 

[IamSpartacus]

AFAIK hw switching is supported on certain modes. LACP, I believe, does support it.

Are you suggesting that I bond the four qsfpplus ports (qsfpplus1-1,qsfpplus1-2,qsfpplus1-3,qsfpplus1-4)? I don't have them bonded currently.

 

[zack$]

Yes, you can try bonding.