Generation difference between ConnectX-4, 5 and 6 ... when all are the same speed (SFP28)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

TrumanHW

Active Member
Sep 16, 2018
256
37
28
Myllenox SFP28 is available as a Connectx-4, Connectx-5 and Connectx-6.

Are there any differences to the average user ..? At the core, they all do the same thing ...
The only difference I can perhaps think of...is the power used under identical loads.

Is there any other reason to prefer one over another..?

I'd assume this is equally true if controlling for 'speed class' (eg., SFP+ vs SFP28 vs QSFP28, etc) for Intel, Chelsio, and so on..?
 

TrumanHW

Active Member
Sep 16, 2018
256
37
28
There's a pretty detailed comparison table by Mellanox

AWESOME. Thank you. That was really informative. Finally, something that explains what NVMe-oF requires in hardware.

NVMe-oF is a feature on ConnectX-5+ adapters and Mellanox Switches of equal or newer gen.

Hardware aside, it's just a set of message-based commands to transfer data between a host computer with NVMe storage device(s), but those commands can be sent over Ethernet, FC, or IB. NVMe commands are tunneled through RDMA yeilding high throughput & low latency.

I'm sure it can't be used with an OS like TrueNAS that abstracts the hardware and presents data ... but ... at least they aren't making dedicated hardware for access to this tech ... but rather treating it as a part of the stack like RoCE, etc.
 

ano

Well-Known Member
Nov 7, 2022
702
302
63
they SHOULD note power usage

we have some ceph clusters now with cx3 and cx4lx, becaus each 2U node goes up 10%! with connectx6, and they are not maxing even a cx3 40gbps.. soo. and for newer 25gbps ones.. yeah power should be in the rows somewhere :|
 

mach3.2

Active Member
Feb 7, 2022
143
104
43
they SHOULD note power usage

we have some ceph clusters now with cx3 and cx4lx, becaus each 2U node goes up 10%! with connectx6, and they are not maxing even a cx3 40gbps.. soo. and for newer 25gbps ones.. yeah power should be in the rows somewhere :|
So the claimed 20W+ power consumption on newer connectx cards are true even if linking at 10GbE/25GbE? :confused:
 

ano

Well-Known Member
Nov 7, 2022
702
302
63
and this is CARD power usage, modules! uses even more, so some of the QSFP28 etc are quite thirsty.
 

i386

Well-Known Member
Mar 18, 2016
4,454
1,665
113
35
Germany
I wanted to write that 25w is the maximum in the pcie specification but then I saw that nvidia mentions in the datasheets that you need pcie slots that can provide more than 25+ watt (x16 slots can provide up to 75watt per pcie specs)