QSFP Cables - same for Ethernet and Infiniband?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

smtr

New Member
Dec 22, 2017
1
0
1
44
Hi,

I found cable DAC Mellanox MC2206130-003, this cable on mellanox webpage is dedicated to infiniband.
Question is: Can I use this cable for 40gb ethernet? (7050QX-32 + ConnectX 3 - FCBT)
Is all QSFP passive DAC cables are same, and can be used with ethernet and infiniband?

Thanks
BR,
Sebastian
 

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
Question is: Can I use this cable for 40gb ethernet? (7050QX-32 + ConnectX 3 - FCBT)
Yes.

Is all QSFP passive DAC cables are same
Nope, there are different version of qsfp.

QSFP = 4x 1gbit/s lanes, used for 1gbit/s ethernet
QSFP+ = 4x 10gbit/s lanes, used for 4x 10gbit/s or 1x 40gbit/s ethernet (or infiniband up to qdr speeds)
QSFP14 = 4x 14gbit/s lanes, used mainly for fdr infiniband and as external sas3 connector.
QSFP28 = 4x 28gbit/s lanes, used for 25/50/100gbit/s ethernet (or infiniband up to edr speeds)

can be used with ethernet and infiniband?
For 40gbit/s ethernet you want at least a qsfp+ dac cable. A qsfp14 or qsfp28 would work too (they are backwards compatible), but they are more expensive.
If you want to use fdr infiniband you will need a qsfp14 dac cable an for qdr you will need a qsfp+ cable.

Watch out when you buy them used on ebay. Most "qsfp" cables are qsfp+ cables, but I got once a lot of 5 qsfp cables that don't work for 40gbit/s ethernet.
 

veegee

New Member
Dec 9, 2019
6
1
3
Yes.


Nope, there are different version of qsfp.

QSFP = 4x 1gbit/s lanes, used for 1gbit/s ethernet
QSFP+ = 4x 10gbit/s lanes, used for 4x 10gbit/s or 1x 40gbit/s ethernet (or infiniband up to qdr speeds)
QSFP14 = 4x 14gbit/s lanes, used mainly for fdr infiniband and as external sas3 connector.
QSFP28 = 4x 28gbit/s lanes, used for 25/50/100gbit/s ethernet (or infiniband up to edr speeds)


For 40gbit/s ethernet you want at least a qsfp+ dac cable. A qsfp14 or qsfp28 would work too (they are backwards compatible), but they are more expensive.
If you want to use fdr infiniband you will need a qsfp14 dac cable an for qdr you will need a qsfp+ cable.

Watch out when you buy them used on ebay. Most "qsfp" cables are qsfp+ cables, but I got once a lot of 5 qsfp cables that don't work for 40gbit/s ethernet.
If I buy QSFP28 direct attach copper, will it work at Infiniband FDR speeds (56Gbit/s) on my FDR switch and FDR card? I want my cables to be "future proof".

If I buy 100Gbit/s QSFp28 optical transceivers, will they also work at FDR speeds?
 

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
If I buy 100Gbit/s QSFp28 optical transceivers, will they also work at FDR speeds?
I tried it with ethernet,not infiniband, and I got a 10gbe link without additional configuration (10gbe is the slowest speed supported on the switch and the nic).
I think it could work with fdr speed (or 40gbe in my case) when the ports are properly configured.
 

arnbju

New Member
Mar 13, 2013
26
11
3
If I buy QSFP28 direct attach copper, will it work at Infiniband FDR speeds (56Gbit/s) on my FDR switch and FDR card? I want my cables to be "future proof".
Did you try this out? At fs.com 100G QSFP DAC is cheaper than 56G QSFP+ DAC
 

Labs

Member
Mar 21, 2019
88
16
8
I used a normal Netapp QSFP SAS cable for 40Gbit testing.
Seems to work as it should :)
I can also confirm the NetApp QSFP SAS cable working at 40Gbit which is strange. I checked the EEPROM and at address 128 it has 0Ch which according to MSA specifications it is a QSFP DAC. I didn't see any issues and works fine at 40Gbit.
 

veegee

New Member
Dec 9, 2019
6
1
3
Did you try this out? At fs.com 100G QSFP DAC is cheaper than 56G QSFP+ DAC
Yes. The 100G QSFP+ DAC worked at 40G when hooked up to the switch, but I forgot if they worked at 56G when hooked up directly between the cards, it was a while ago. If you really want to know the answer to that question, I can test it in a few days. I ended up purchasing the genuine Mellanox cables from ebay for a similar price and they worked at 56G with the switch as well as when directly connected between cards.

FS.com offered to reprogram their 100G cables for me if I sent in one of the genuine Mellanox cables for them to extract the data from. I never followed up since this was just for homelab use and I already acquired a bunch of genuine Mellanox cables for a very good price.

Check ebay, you'll almost certainly find them for a good price.
 

arnbju

New Member
Mar 13, 2013
26
11
3
Yes. The 100G QSFP+ DAC worked at 40G when hooked up to the switch, but I forgot if they worked at 56G when hooked up directly between the cards, it was a while ago. If you really want to know the answer to that question, I can test it in a few days. I ended up purchasing the genuine Mellanox cables from ebay for a similar price and they worked at 56G with the switch as well as when directly connected between cards.
Thanks if i can get 40G from 100G cables that is good enough. Will look for some cheap original cables on ebay.
 

marelooke

New Member
Mar 23, 2020
13
1
3
Does it follow that an optical transceiver module that only lists Ethernet support should work for Infiniband, and vice-versa?

Eg. should a 40GBASE-SR4 module work for both 40Gbit Ethernet as well as QDR Infiniband, regardless of what is listed as supported protocols?
 
Jul 19, 2020
51
19
8
DACs are fine, but for AOCs and optical transceivers, you can run in to issues with the specific line rate. For instance, most QSFP28 optical transceivers and AOCs contain CDR circuitry, and in many cases this won't lock if the data rate isn't within a few ppm of 25.78125 Gbps (or a fraction of that, such as 12.890625). In many cases it is possible to disable/bypass the CDRs via the module I2C interface. For 10G modules, I think CDRs are less common. If there is no CDR, then it will probably work fine.
 
  • Like
Reactions: marelooke