Mellanox Connectx-3 pro 40gbe vs connectx-4 LX 25 gbe
- Thread starter michel333
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
cx-4:
- newer tech (28 GBit/s serdes -> lower latency)
- newer firmware
- newer features (for example roce v2)
- more/better support (for nvidia most of the cx-4 skus are eol)
(- more expensive)
without any background infoirmation I would always say "newer is better" for the mellanox nics
- newer tech (28 GBit/s serdes -> lower latency)
- newer firmware
- newer features (for example roce v2)
- more/better support (for nvidia most of the cx-4 skus are eol)
(- more expensive)
without any background infoirmation I would always say "newer is better" for the mellanox nics
A 100$-ish sx6018/6036 feels like the most used stuff by how many labs you see them in.
Very low power consumption on them also.
i got a bunch of Connectx-4 LX 25 gbe cards but im still using them as just 10 cards due to the cost of 25/100 switches still.
But the price diffrence compared to 10 cards was so low that i just "had to go 25".
I've used both, and they're both pretty performant. The only issue with the CX-3 is lack of esxi 8.0 support, which is pretty stupid IMHO.
Its so old that relevant drivers to put in for it is lacking.
Not like they just decided to drop it for fun.
That its dropped from 8 and cost of cx4 cards is mainly why i got a bit of cisco servers now tho.
The 2x 40gbe cards for cisco servers start at 30$ or so and is esxi8 supported and fit in M4+M5.
But... Europe is also about to be hit by a massive wave of C240 M5 (scalable, r740xd equivalent) for dirt cheap with 2x 40gbe mlom included.
Already starting to see some lots like this one with 10pcs for 1000£, seeing ALOT of sellers starting to list the exact same spec barebone nodes.
Can you do Ethernet on this mellanox sx6018/6036 switchs? Or is only for infiniband switch? And you need some Cpu that supports infiniband?
What about this switch Sun Oracle 24-Port 10G Switch
Modell: ES1-24
P/N: 7315650
20x 10GBASE-T
4x 1/10GbE SFP+ for 300 euros? It is a good deal?
To enable ethernet (and 56gbe if using mellanox card also) is just an optional license that needs to be activated on it, they were sold as both infiniband and infiniband+ethernet.
There are a few large threads on them in the forum here that covers it.
the silicom nic is intel based; intel has (had?) the best driver supports accross the different operating systems.
I don't know much about these cars but if it's 10GBE I assume it's "older" and doesn't support roce or iwarp.
And again without any background information what you plan do to do or what operating systems you want to use I would recommend again the mellanox cx-4 becuase it's newer and has more features (roce v2).
The only reason I would consider the silicom (intel) nic is the driver support
Ethernet is included on the managed switches as they were VPI from the start (multi protocol compatible). The license was only for 56gb speeds (a non standard implementation that squeezed out the extra speed from the 40gb hardware) but that was made free and enabled by default with MLNX-OS 3.4.3002 which was released about a decade ago.
Also, OP is asking about the ConnectX-3 *PRO* and ConnectX-4 *LX* lines. The X-4 LX is somewhat of a step back from regular X-4 and the X-3 Pro is a step up from normal X-3. The LX looks like it was a stop-gap release to address the 25gbe market. It's a pure ethernet product without Infiniband capabilities to fill the product line gap between 10/40gb speed that IB sorta skipped over.
This list compares the generations and you really don't see much major improvement until Gen 5 ConnectX. A bump in speed (or step back in case of LX) with a few incremental improvements and niche features.
As for RoCE v2, that's supported by the ConnectX-3 as it's just software protocol. X-4 LX doesn't even support IB and if latency and RoCE are important to your use case, IB is generally better performing (much lower port to port latency and a lossless protocol for example). The biggest reason to not go with IB is cost and complexity, as ethernet is just a lot cheaper tech and less difficult to implement.Sorta reminds me of FC vs iSCSI where FC is "better" for sure at similar speeds, but it is a lot easier to implement ethernet since it is everywhere...and iSCSI is even easier to setup than FCoE on the software side. That said, in the used market and for a homelab setup, IB is a lot cheaper than 25gbe and often even 10gbe despite providing cross compatibility if you pay attention to model specs and stick to the VPI products.
In the end, it's all a huge step above 10gbe and not really used in a homelab or small production scenario as much as it feels good to see numbers get bigger. 25gbe will serve most people just fine looking for faster ethernet speeds. Infiniband nearly doubles that speed for ethernet and lets you play with "fabrics" and not just networks. IB also tends to be a lot cheaper per unit of speed than ethernet which is a huge benefit for home use.
[EDIT] I also totally forgot one major reason to consider IB right now, GPUDirect over RDMA. With the AI hype train at full steam, playing with nvidia GPUs communicating with each other through RDMA across nvidia owned mellanox IB fabrics is incredibly relevant tech to play with.
Last edited:
I'm pretty sure that's incorrect. So far all the documents/manuals for the managed 6000series list the gateway/ethernet functionality as optional except the 6036G. This matches my experience with a hpe branded 6036 and many more from users from this forum.
Roce v2 is not supported by the regular cx3 and can't be added via custom firmware. Roce v2 was introduced with the cx3 pro that also introduced a newer asic + firmware. That firmware is not compatible with the cx3 (-> no crossflashing possible)
Id expect there to be a sku with the license included/activated.
But they are normally IB only and you need to activate the gateway license to get ethernet.
How to enable gateway/ethernet is the most common topic for them by miles.
Mellanox License Info
Yes, the ROCE v2 is only ConnectX-3 Pro, which is what the OP is asking about. Also, RoCE is literally just "RDMA over Converged Ethernet" which is also known as InfiniBand over Ethernet...so RoCE is a translation of native IB to use omnipresent ethernet infrastructure without needing special hardware (again, same story as the FCoE and iSCSI instead of native fibre channel).
This seems to be an issue with people not paying close attention to which SKU they buy. As far as I know, all 40gb NICs, there are 3 versions of ConnectX based products, pure ethernet, pure IB, and VPI which supports both but must be manually configured and rebooted to switch between the two (by default IB...which is not what most people want so they work "out of the box"). The actual hardware is virtually identical which is why you see cross flashing. For full 56gb speed (IB or ethernet) you need an FDR model to get the extra 16gb speed over the FDR10 models.
For switches, it becomes a little less flexible. Managed Switches in the SX60XX line (even model numbers) are inherently VPI and support both IB and ethernet (it's one of their major marketing points). The separate license is needed for the extra 56gb speeds (I think specifically for ethernet, but possibly IB) but that's a moot point as Mellanox made the license free and set 56gb as the default speed like a decade ago. The complication with switches comes from the unmanaged (and much cheaper) models which DO NOT support VPI and are IB only. On top of that, there are a ton of rebranded mellanox managed switches running different OSes (namely dell EMC surplus) that are sold as genuine Mellanox. These can be reflashed with MLNX-OS but it's not a trivial process at all.
The IB network gear is pretty flexible as a whole and inherently speedy. The reason mellanox is everywhere is because IB is fundamentally zippier than ethernet for HPC applications, and unlike OmniPath, designed to be cross compatible with ethernet, if you put in a little work. The biggest issues are that they aren't all plug and play compatible for ethernet usage and that's what most people want to use them for. You have to pay attention to SKUs and hope resellers are doing the same. Then even if you do get all the proper hardware, all the information for configuring them is spread across a dozen forum threads of people experimenting (successfully). There really aren't any straightforward tutorials or guides on how to use them in a homelab setting beyond the forums and the official HPC/DC oriented technical documentation.
Explain? Like I said above, the rebrands introduce licensing issues (and the rebrands are the majority of what you find cheap). Mellanox firmware itself hasn't required a separate license for ages, but flashing a switch from rebrand to MLNX-OS is "here be dragons" territory. Ironically though, if you want to reflash an EMC SX60XX switch to stock MLNX-OS, you can only easily and publicly find the firmware images under the Lenovo OEM archives. Generally, anytime I see HP/Dell/Lenovo rebrands being used to support the idea a hardware tech is limited, it's 99% not true, HP/Dell/Lenovo usually are disabling features or vendor locking rather than offering the full features. From AMD CPUs to HBAs to NICs. It's just unfortunate that it's primarily the nerfed rebrands that flood the secondary market since they EOL and software nerf perfectly good and stable and essentially modern hardware based on financial cycles.
That license requires the ethernet functionality( =license) that is not free/available out of the box with all but the 6036G of the sx6000 series...