Using Mellanox Infiniband DAC Cables for Ethernet, is this possible?

[TonyA]

This previous thread suggests that using Mellanox Infiniband DAC cables, for ethernet instead should work OK.

https://forums.servethehome.com/index.php?threads/qsfp-cables-same-for-ethernet-and-infiniband.18020/

However, as I'm new to all of this stuff I want to try to be as specific as possible to save laying out cash that is for things that won't work.

I picked up some Mellanox MCX4131A-GCAT 50Gbe HBAs, thus far I haven't plugged any cables into them as I'm still in 'trying to understand some of the basics' mode, I can get some Mellanox MCP1600-E003 100Gbe Infiniband DAC Cables.

Both the HBAs and the Cables are rated as QSFP28 compatible.

HBA spec sheet

https://network.nvidia.com/files/doc-2020/pb-connectx-4-lx-en-card.pdf

Cable spec sheet

https://network.nvidia.com/pdf/prod_cables/PB_MCP1600-E0xxEyy_100Gbps_QSFP28_DAC.pdf

As I don't know if the, for want of a more appropriate description, transceivers at the end of the cables, are 'hard wired' only for Infiniband (or if they even can be) I'm cautious about spending money only to find out I've made a beginners mistake.

The previous thread was very helpful and gave me some confidence, but I thought it prudent to ask specific questions and hope to learn from others experience.

 

[aosudh]

It is possible, as long as yours switch have auto negotiation function and have proper portprofile

 

[TonyA]

It is possible, as long as yours switch have auto negotiation function and have proper portprofile

I was thinking of using them point to point right now as I want to try to get experience before investing in a switch.

I think I'll be looking at a 100Gbe switch, however prices seem circa $2.5k on E-Bay, I've been reading about the Brocade ICX switches that can be reflashed, but don't think any of those ones currently support 100Gbe.

 

[aosudh]

I was thinking of using them point to point right now as I want to try to get experience before investing in a switch.

I think I'll be looking at a 100Gbe switch, however prices seem circa $2.5k on E-Bay, I've been reading about the Brocade ICX switches that can be reflashed, but don't think any of those ones currently support 100Gbe.

Those white box or with small factory software switches are not so suitable for home use, they need a complete software system. For home use, I would recommend mellanox's sn27/2100 as your first switch, since you haven't had much experience with these devices before

 

[Castlers]

Since you have 50gbe NICs why don't you go for something like a Mellanox SX6036 switch? (<$200 on ebay) Its 40gbe if you have the "license" and every DAC cable I have gotten has been fine with infiniband or ethernet for that switch and I have a few optical ones from FS.com

 

[piranha32]

According to https://support.mellanox.com/s/article/MLNX2-117-1804kn the 56GbE license is available free of charge, and new releases of the OS have it enabled by default

 

[Castlers]

its more of FDR infiniband and if you HAVE the ethernet license you can get 56gbe as well, the ethernet licensing and the L3 capabilities are not unlocked by default. Do you know what version of the MLNX OS you have?

 

[TonyA]

Since you have 50gbe NICs why don't you go for something like a Mellanox SX6036 switch? (<$200 on ebay) Its 40gbe if you have the "license" and every DAC cable I have gotten has been fine with infiniband or ethernet for that switch and I have a few optical ones from FS.com

Since I'd posted the original question I'd picked up some 100Gbe Optical QSFP28 transceivers (only paid $14 each) and I have some fibre cable that may or may not work, so was looking now to go 50Gbe point to point while I'm still learning. I really don't want to invest in a 40Gbe switch if I'm honest, even a cheap one, I'd much rather keep my eyes open for a 100Gbe bargain, even if that takes a few months.

 

[aosudh]

Since I'd posted the original question I'd picked up some 100Gbe Optical QSFP28 transceivers (only paid $14 each) and I have some fibre cable that may or may not work, so was looking now to go 50Gbe point to point while I'm still learning. I really don't want to invest in a 40Gbe switch if I'm honest, even a cheap one, I'd much rather keep my eyes open for a 100Gbe bargain, even if that takes a few months.

50gbe is a kind of awkward speed rate as if you want to apply rdma(rocev2) in your home network environment, ou have to make sure that the speed rate of each device must be the same. Otherwise, this will result in the packe loss, which is not allow in a roce network. So my recommendation is to go from standard speeds like 40gbe or 100gbe

 

[aosudh]

And I don't see any advantages of purchasing a 50gbe card as a standard 100gb vpi card mellanox cx455a is only at aroud $100

Mellanox MCX455A-ECAT CX455A 100GbE Single Port QSFP28 PCIe ConnectX-4 VPI NIC | eBay

Find many great new & used options and get the best deals for Mellanox MCX455A-ECAT CX455A 100GbE Single Port QSFP28 PCIe ConnectX-4 VPI NIC at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[TonyA]

50gbe is a kind of awkward speed rate as if you want to apply rdma(rocev2) in your home network environment, ou have to make sure that the speed rate of each device must be the same. Otherwise, this will result in the packe loss, which is not allow in a roce network. So my recommendation is to go from standard speeds like 40gbe or 100gbe

I'm not sure if you noticed that I'd said point to point (i.e. Direct connect a couple of PCs) in my post?

 

[aosudh]

I'm not sure if you noticed that I'd said point to point (i.e. Direct connect a couple of PCs) in my post?

Sorry i miss that . But that rule is still in effect here

 

[TonyA]

And I don't see any advantages of purchasing a 50gbe card as a standard 100gb vpi card mellanox cx455a is only at aroud $100

Mellanox MCX455A-ECAT CX455A 100GbE Single Port QSFP28 PCIe ConnectX-4 VPI NIC | eBay

Find many great new & used options and get the best deals for Mellanox MCX455A-ECAT CX455A 100GbE Single Port QSFP28 PCIe ConnectX-4 VPI NIC at the best online prices at eBay! Free shipping for many products!

www.ebay.com

I only paid circa $15 for the 50Gbe cards and about $12 for the 100Gbe transceivers (both QSFP28), so the financial incentive was quite high. I think I'm going to have to invest in some active cooling though as one of the cards is running pretty toasty before it's even under load, I'm just reading up on google what some others have done.

 

[KevinH]

@TonyA could you give an update on using MCP1600-E003 cable for 100Gbe? The specs says it’s for infiniband only and they have a different part number for 100Gbe. Wondering why making 2 part numbers if they work interchangeably.
For 40Gbe, I could confirm InfiniBand cable works well with Ethernet.

 

[klui]

Mellanox Infiniband cables are funny. Their passive coppers should work on everything. But their Infiniband AOCs are more particular on the card. I recall they won't link up for some speeds on Mellanox ethernet-only (MCX....-EN) cards.