bi-direction performance of Mellanox ConnectX-2 cards?

[BLinux]

I just got myself 4 of the $15/each Mellanox ConnectX-2 10Gbe cards. The cards came with 4 3-meter long DAC cables with Cisco branded SFP+ modules. I tested them today with iperf and uni-directional performance was great, showing about 9.8x Gbps - results were same on all 4 cards with all 4 cables. This is using 64bit CentOS 7; no tweaks on anything.

When I did the bi-directional test, I'm only seeing about 5~6Gbps each way. They are showing link at 10000M and Full duplex. When I perform the uni-directional test individually in both directions, I get 9.8x Gbps. So, is this pretty typical for this card as far as bi-directional performance? Is it not able to saturate 10Gbps in both directions at the same time? Or does it require some tweaking?

The system is an 8-core C2758 - I did have to run iperf -P 8 in order to get the 9.8Gbps ... is it possible that the bi-directional test is CPU bound? (since each iperf process is both sending and receiving?)

 

[BLinux]

anyone have bi-directional performance stats to compare with the Mellanox ConnectX-2?

 

[ttabbal]

Using Proxmox on one end, Ubuntu on the other, both the latest versions and fully patched. 2x Mellanox CX2 cards (HP branded), OM3 fiber. I get about 9.7Gbps with both bidirectional and unidirectional tests. I haven't tested with DACs, but can't see why it should matter. I didn't do any tweaking, just whatever settings the OS came with. No jumbo frames etc., IPV4 static addresses on an independant subnet. This was point-to-point, no switches or other traffic on those interfaces.

 

[BLinux]

Using Proxmox on one end, Ubuntu on the other, both the latest versions and fully patched. 2x Mellanox CX2 cards (HP branded), OM3 fiber. I get about 9.7Gbps with both bidirectional and unidirectional tests. I haven't tested with DACs, but can't see why it should matter. I didn't do any tweaking, just whatever settings the OS came with. No jumbo frames etc., IPV4 static addresses on an independant subnet. This was point-to-point, no switches or other traffic on those interfaces.

were your tests done with iperf as well or another tool? with your info, now i'm wondering where my bottleneck is...

 

[ttabbal]

Just iperf. You could watch CPU use to see if that's involved, but I wouldn't think so. Even these older cards have some offload capabilities. The only thing I can suggest is trying to swap parts, as I didn't have to really do anything, I don't have any real troubleshooting suggestions.

 

[Gabriel Mateiciuc]

Use the drivers from the mellanox site vs ofed standard. they make a difference on cpu usage.

 

[BLinux]

Use the drivers from the mellanox site vs ofed standard. they make a difference on cpu usage.

do they supply drivers for Linux that are better performing than what is included in the kernel? is it opensource or a binary blob?

 

[Gabriel Mateiciuc]

The software from Mellanox can be built just as the OFED generic one. What's binary I don't know for sure, I haven't checked the build log. It wouldn't make much sense since they are also releasing the source code for their releases.