Generation difference between ConnectX-4, 5 and 6 ... when all are the same speed (SFP28)

[TrumanHW]

Myllenox SFP28 is available as a Connectx-4, Connectx-5 and Connectx-6.

Are there any differences to the average user ..? At the core, they all do the same thing ...
The only difference I can perhaps think of...is the power used under identical loads.

Is there any other reason to prefer one over another..?

I'd assume this is equally true if controlling for 'speed class' (eg., SFP+ vs SFP28 vs QSFP28, etc) for Intel, Chelsio, and so on..?

 

[iGene]

There's a pretty detailed comparison table by Mellanox

ESPCommunity

enterprise-support.nvidia.com

 

[TrumanHW]

There's a pretty detailed comparison table by Mellanox

ESPCommunity

enterprise-support.nvidia.com

AWESOME. Thank you. That was really informative. Finally, something that explains what NVMe-oF requires in hardware.

NVMe-oF is a feature on ConnectX-5+ adapters and Mellanox Switches of equal or newer gen.

Hardware aside, it's just a set of message-based commands to transfer data between a host computer with NVMe storage device(s), but those commands can be sent over Ethernet, FC, or IB. NVMe commands are tunneled through RDMA yeilding high throughput & low latency.

I'm sure it can't be used with an OS like TrueNAS that abstracts the hardware and presents data ... but ... at least they aren't making dedicated hardware for access to this tech ... but rather treating it as a part of the stack like RoCE, etc.

 

[i386]

Is there any other reason to prefer one over another..?

Why you would want newer generations:
More features like offloading different protocols, encryption etc.
Smaller lithography

 

[i386]

NVMe-oF is a feature on ConnectX-5+ adapters and Mellanox Switches of equal or newer gen.

Mellanox Interconnect Community

mellanox.my.site.com

this is a post by mellanox about using nvme of with cx3

 

[ano]

they SHOULD note power usage

we have some ceph clusters now with cx3 and cx4lx, becaus each 2U node goes up 10%! with connectx6, and they are not maxing even a cx3 40gbps.. soo. and for newer 25gbps ones.. yeah power should be in the rows somewhere :|

 

[mach3.2]

they SHOULD note power usage

we have some ceph clusters now with cx3 and cx4lx, becaus each 2U node goes up 10%! with connectx6, and they are not maxing even a cx3 40gbps.. soo. and for newer 25gbps ones.. yeah power should be in the rows somewhere :|

So the claimed 20W+ power consumption on newer connectx cards are true even if linking at 10GbE/25GbE?

 

[ano]

So the claimed 20W+ power consumption on newer connectx cards are true even if linking at 10GbE/25GbE?

the older uses less than they say, the newer use MORE.

 

[ano]

and this is CARD power usage, modules! uses even more, so some of the QSFP28 etc are quite thirsty.

 

[i386]

I wanted to write that 25w is the maximum in the pcie specification but then I saw that nvidia mentions in the datasheets that you need pcie slots that can provide more than 25+ watt (x16 slots can provide up to 75watt per pcie specs)