Mellanox ConnectX-3 help

[Mkvmike]

Hi,

Im very new to homelabs and just finding my feet, so I appologise in advance if this has been said before. (I have searched but can't find anything specific).

My aim is to direct connect from a W11 machine to an Unraid server, using the cards and this DAC.

Sorry if this comes accross as a bit dumb, but would this cable be ok, or would I need a specific Mellanox cable?

Any other advice to get me started would be greatly appreciated.

Mellanox

Generic DAC

 

[CyklonDX]

You do not need crossover for fiber. Just ensure both sfp's are compatible with your systems. (mellanox is quite chill on using all kinds of sfp's but intel, and some others do not)

 

[i386]

would I need a specific Mellanox cable

If you want to use 56GBE you would need qsfp14 dacs. So far I've seen only rebranded mellanox cables for that.

 

[sko]

If you want to use 56GBE you would need qsfp14 dacs. So far I've seen only rebranded mellanox cables for that.

fiberstore (fs.com) has active and passive DACs for mellanox and 56GbIB / 40GbE.

 

[Philip Brink]

I use a 3 meter 40gb copper DAC from FS.com and am very happy with it. When you select the DAC, there will be a pulldown option for programming the ends to Mellanox.

Pretty quick turn around on the order including shipping to Canada, great support team who will check in to make sure everything is working correctly. Will be ordering a couple more in a month or so.

 

[Stephan]

Release notes of card firmware has a list of certified cables. Beyond 4m I would go with fiber. There are cheapish refurbished fiber options like MC220731V.

 

[Mkvmike]

I ended up getting a set of these, I hope i dont regret it.

MELLANOX MC2206128-005

 

[Mkvmike]

Well it turns out the MELLANOX MC2206128-005 cable won't do 40G as described, it maxes out at 10G. Both cards are up to date and running the latest firmware. I have checked everything else I can think of with both cards.

@Stephan
I can't seem to buy MC220731V anywhere in UK for a reasonable price, so I ended up ordering Mellanox MC2207130-005 56G hoping this will cure the problem.

@Philip Brink
Yeah I looked at a few but tbh I couldnt tell the difference from the one I already bought

 

[BeTeP]

Well it turns out the MELLANOX MC2206128-005 cable won't do 40G as described

There is no way that a working MC2206128-005 won't do 40G. Either it's broken or your problem is not cable related.

 

[Mkvmike]

There is no way that a working MC2206128-005 won't do 40G. Either it's broken or your problem is not cable related.

This is the output, both cards are directly connected and identical (one A3 the other is A5), I agree as Im not sure what else it can be

 

[i386]

it maxes out at 10G.

Like windows shows the link as 10gbe? Or copying data maxes out at something like 1GByte/s?

 

[Mkvmike]

Like windows shows the link as 10gbe? Or copying data maxes out at something like 1GByte/s?

Hi,

With single iperf = 13Gbps and with two instances running 6Gbps ish each

 

[Propaganda]

Hi,

With single iperf = 13Gbps and with two instances running 6Gbps ish each

View attachment 27131

View attachment 27132

What is your iperf command? You probably need to use "-P". Also, the windows stack isn't best to use with iperf on high rate networking. Might want to try linux.

 

[Mkvmike]

Unraid side I tried:
iperf3 -s -p 4500& iperf3 -s -p 4501

W 11 Client side i ran two instances:
iperf3.exe -c 192.168.50.5 -p 4500 -t 50
iperf3.exe -c 192.168.50.5 -p 4501 -t 50


this was recommended on the unraid forum.

 

[Propaganda]

Add "-P 20" to your command note that is a capital not lowercase "P".

 

[Mkvmike]

Tried that and went down to 6Gbps

I have an i5 10600k sat around, so over the next couple of days, i'll swap it out and rule out the cpu, as its currently an i3.

It's so frustrating

 

[i386]

try ntttcp instead of iperf.

I never get good results in iperf. Even simple copy operations over the network via explorer yield higher troughput...

 

[Mkvmike]

Unfortunatley I only have a Windows and unraid machine to hand, I could create a Windows VM on unraid but not sure if it would work as good as bare metal?

 

[Stephan]

Just use ntttcp from Windows, and one for Linux. Should work. Your 40 Gbps link is 10 Gbps times 4 physically. So you need four separate byte streams of whatever to saturate. Also check both machines that specs do not cause high CPU load. You really need 4-core Skylake 65W or better or later to saturate all four links at 10 Gbps. What are the other machine specs? When copying data around, be aware that Windows Defender antivirus will gladly destroy 90% of performance. Maybe try USB thumb drives with Ventoy and some bootable Linux ISO with persistence and ntttcp and a recent kernel like 5.15.x to exclude misbehaving OSs.

Always read https://network.nvidia.com/related-docs/firmware/ConnectX3-FW-2_42_5032-release_notes.pdf page 12f section 1.1.5 "Validated and Supported FDR Cables" and for ConnectX3, always flash to generic Mellanox firmware for good measure. Because as we've learned, HPE firmware is 3rd party transceiver and therefore home lab hostile.

 

[Mkvmike]

Just use ntttcp from Windows, and one for Linux. Should work. Your 40 Gbps link is 10 Gbps times 4 physically. So you need four separate byte streams of whatever to saturate. Also check both machines that specs do not cause high CPU load. You really need 4-core Skylake 65W or better or later to saturate all four links at 10 Gbps. What are the other machine specs? When copying data around, be aware that Windows Defender antivirus will gladly destroy 90% of performance. Maybe try USB thumb drives with Ventoy and some bootable Linux ISO with persistence and ntttcp and a recent kernel like 5.15.x to exclude misbehaving OSs.

Always read https://network.nvidia.com/related-docs/firmware/ConnectX3-FW-2_42_5032-release_notes.pdf page 12f section 1.1.5 "Validated and Supported FDR Cables" and for ConnectX3, always flash to generic Mellanox firmware for good measure. Because as we've learned, HPE firmware is 3rd party transceiver and therefore home lab hostile.

Is the problem maybe my cable? What is the difference between the one I have (FDR10) and the 40GBE one? Does the FDR10 cable split 4x10, and the other doesnt? Could that be my problem?