Mellanox Infiniband Problems

[Rdl]

Castlers

Have you got any further with this? I am seeing the same problem. My card is a connectx-3, I have checked that both cards are mounted in the first pci-e slot and the bios are both set for 16x. Best speed I show is 13.42 gig. What other things have you tried changing? Both machines are AMD on is a tr40 and the other is a x570.

 

[Castlers]

I haven't been able to get above the 20gb/s on any of the systems as I'm unsure what's causing it.

 

[Rdl]

If you are still wanting to do a SX6036 I can give you a hand with it.

 

[Castlers]

I got the SX6036 working, thank you, but I'm still trying to figure out what's causing the performance drop on my ConnectX3 and 4 cards

 

[Rdl]

I didn't know you had connectx-3 cards also are you getting 20gb on them also? I thought that maybe my 13gb might be because they were a 3 series instead of a 4 series. I went and looked at a few things, on my x570 board the bios has setting for different pcie modes (pcie1, pcie2,3) for backward compatability , but also offers 2x8, 1x4,2x4,4x4x4 or auto so want to try different setting for the bandwidth to see if that makes a differance. Looked a little at the linux tuning and it seems to deal with the cpu settings and I know in widows we can assign it to different cpu's under the settings tab so that maybe something to look at also.

 

[Castlers]

I tried quite a lot of different combinations even in QDR/FDR/ETH it still caps around 20 ish. Even when directly connect from card to card I'm not really sure what is wrong. Tried different drivers firmwares abd using my xeon platforms with a lot of available pcie lanes.

 

[Rdl]

tested the pcie and found that connectx 3,4 are both pci3.0 x8 cards so they don't need a x16 connection. Looked at the numa but results look different on the amd vs intel. So not sure what else to check from here.

 

[Rdl]

When you do your test with Iperf-3 if you only use one client you get a very low number. I used this command on my client machine

iperf3 -c 192.168.5.5 -t 60 -P 20

t is for time in secs
Cap P is for number of extra clients

mine came back at about 31 gbs not perfect but much better that 13.5

I did try different numa's but it made no differance on my thruput. I also tried some different setting on the performance tab and those do make a differance. I still need to try it with a teaming setup under Mellanox setting and maybe also with windows teaming. Last but not least is trying with the dac cables which will boost the connections to the 56 gig connections.

 

[acor254]

I just finished putting into production at work a couple months ago a budget Server 2019 Storage Spaces Direct cluster that is using a pair of SX6036's with 56Gb IB and CX-3 Pro's for the backend storage SMB / Cluster network. I maxed out at 60Gbs for a dual port card before we added the CX4 LX's for ethernet, and 4x NVMe for cache. Was able to solidly hit 54Gb-ish single port server to server with Diskspd tool. After adding those components and before full optimization for config settings but pretty good optimization topped out at 44Gb/s on nd_write_bw.exe between the 2 ports on an adapter in loopback test. Likely due to less available PCIe bandwidth.

I think I switched to Diskspd tool as main testing / tuning tool after that and the VMFleet testing for the cluster tuning. Looks like from my testing notes a single VM on the cluster test running Diskspd with vhdx sitting on different server CSV that was not tiered which was doing 4k random 100% read, 12 threads, 32 outstanding hit 1079MB/s, 276k IOPS with Avg latency of 0.925ms. These are from a VM and numbers lower than server to server.

These were on a 4 node cluster with the refurb R730Xd chassis and hit 1.5M IOPs for the total 4 node cluster on 4k random with 6.2GB/s, as well as about 20GB/s on 16k random that was still over 1M IOPs. Its doable to get more speed but I spent a few months testing and tuning to get all the configuration settings optimized with not just the NICs but the IB switches too. If you compare the single VM performance to 1/4 of the total VMFleet (1 node) you can see a single vm even heavily loaded does not quite hit the same perf number as spreading it out across multiple VMs. Max single VM I hit 276K IOPs vs 375K for 1 node.

It really depends on your use case I guess what is acceptable and the best setup here and iPerf may not be enough alone to see what your setup is capable of is my point I guess. Especially since it does not take advantage of RDMA. Just one tuning tool. If you are still fighting with this I can try to pull some of the key settings Monday when I get back in the office.