Which switch for connecting ConnectX2 and ConnectX3 cards ?

[sli]

Hello,


My lab is a HyperV cluster composed of 4 x HP DL170e servers and 1 x Dell RX720XD with Windows Server 2019.

HP Servers > HyperV Hosts
Dell Server > Storage Server (ISCSI and SMB).


I purchased 1 ConnectX3 40GB dual port card for the Dell server (part number MCX354A-QCBT).
HP servers come with ConnectX2 40 GB single port cards (part number MHQH29B-XTR)

I have 5 x Mellanox QSFP+ DAC passive cables (MC2206130-001).

Mellanox MC2206130-001 40G QSFP+ DAC Cable

Check Mellanox MC2206130-001 Compatible 40G QSFP+ DAC Twinax Cable data sheet (1m, passive, QSFP+ to QSFP+, 30AWG) and price list on FS.COM, buy it now!

www.fs.com

.

For testing purpose, i have connected 2 HP servers to the Dell server with the DAC cables and it works.

I bought a Mellanox SX6015 switch and it doesn''t seem to work with my cards.
opensm.exe return "no local port detected" when i launch it in command prompt.

I think this is because it is not a QDR switch but an FDR and FDR10 switch.
And this switch is not compatbible with my QDR Connectx2 and Connectx3 cards.

I am planning to purchase a Mellanox IS-5022 which is apparently a QDR switch with 8 QSFP ports.

Will this switch work with my Connectx2, ConnectX3 cards and my DAC cables?


Thank you.

Jeremy

 

[i386]

Will this switch work with my Connectx2, ConnectX3 cards and my DAC cables?

My guess? No.

The mellanox stuff that I have is not picky about dacs or transceivers. I used connect-x 2,3 and 4 without problems on a 6036 in ethernet mode.
I would recommend to try a cable that's not from fs.com (they seem to have quality problems since the pandemic started, you can find different threads from the last year and half on sth where fs.com cables were the problem) and see if the problems disappear.

Another idea that I got while writing this post: Did you check the windows firewall and if it's blocking opensm.exe?

 

[sli]

Thanks for your answer.

I'm really new to the mellanox and infiniband worlds.

I no longer have the SX6015 switch so I can't try another cable.

My server cards support 40G Infiniband and 10G Ethernet mode.

The 6036 is a managed switch so it can operate in both infiniband and ethernet mode, im i right?
Does ethernet mode on Mellanox switch require to buy a license ?

The SX6015 and IS5022 are unmanaged switch, do they only work in infiniband mode ? (With openSM running on one node directly connected)

 

[i386]

The SX6015 and IS5022 are unmanaged switch, do they only work in infiniband mode ?

Yes, these are "dumb" infiniband switches.

Does ethernet mode on Mellanox switch require to buy a license ?

It requires a license, but there are ways to get/generate them. Search the forum for "ltrace"

 

[sli]

Which low budget refurbiished managed switch (less than 200 $) do you recommend ? (which can support my cards and DAC cables)
It is better to go for a Mellanox switch or will other brands work as well ?

I have seen Mellanox IS5030 under 100$ on ebay.

 

[i386]

Do you need infiniband?

Otherwise check the brocade icx megathread for cheap 10/40gbe switches.
Other recommended brands are arista (eg 7050 series) or mellanox (6036 with the vpi/ethernet license or a modded emc 6012)

 

[sli]

No need to use infiniband.

I have tried building a lan with mellanox stuff (originally come with my HP server) and ended up finding it too complicated to get a stable setup at a low cost (with refurbished boards and switches).

I think I will opt for a 10Gb Ethernet LAN for my lab and buy QSFP+ to SFP+ adapter (MAM1Q00A-QSA-SP) for the cards of my servers and a SFP+ switch like the mikrotik CRS317-1G-16S+RM (16 ports 10 Gb SFP +).

Thank you very much for all your advice.

Jeremy.