ConnectX-4 compatibility questions

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

unphased

Active Member
Jun 9, 2022
164
39
28
My questions about CX4's are:

  1. Can the cheaper CX4 Lx cards e.g. MCX4131A-GCAT 50Gbit cards operate at 40Gbit when connected directly to ConnectX-3 QSFP 40Gbit NICs (of which I already own 3 or 4)? The CX4 Lx product brief seems to indicate so.
  2. I found that CX4 non-Lx cards have BCAT and GCAT models which support 40/56GbE or 50GbE, and both are QSFP28. I'd like to know a bit more about what that means since the CX4 product brief didn't seem to clarify.
  3. Would the 50GbE models (e.g. MCX414A-GCAT) not support 40GbE? So I would need to grab a MCX414A-BCAT for compatibility with my QSFP 40Gbit networking?
  4. Is 56Gb infiniband only? If so I need to just ignore this as a thing, right? My CX3 VPI cards also do 56Gbit IB i think.
  5. If I get a MCX414A-BCAT would it not be capable of 50Gbit (2 lanes of SFP28??)? So would choosing the 414A BCAT over the GCAT be like choosing to give up the second SFP28 on a QSFP28 port, in order to gain QSFP 40Gbit compatibility? No?
  6. What about 100Gbit? A MCX415A-CCAT is pretty affordable... Is this basically a MCX413A-GCAT but just filling out the second half of the QSFP28? So, maybe same raw bandwidth as a dual port MCX414A but only one port
  7. I was gonna ask about MCX413A-GCAT vs MCX4131A-GCAT but apparently these are just the same thing.
  8. Any benefits for moving up to CX5? I see a MCX515A-GCAT (50Gb) is similar price to MCX415A-CCAT (100Gb)
  9. What about the MCX455A variants? They seem to involve Infiniband so probably what I assumed earlier about IB w.r.t. 56Gbit was wrong.

The thought process motivating this is that I've come to learn that macOS is finally shipping with an mlx5 driver so I need CX4 or better to get out of the box support on macOS, so the idea is to get hardware that will let that point to point operation just work, but I'm also wondering what it would give me in the way of options in the future when SFP28/QSFP28 based infrastructure like switches become affordable. If an option exists where I can have the best of both worlds it would be worth paying a bit more for that I think.

USUALLY it makes the most sense to get the cheapest compatible thing. In this case that would be a $70 MCX413 that is not capable of 50Gb, which saves a few bucks. However, once I move to SFP28 infrastructure such a card will be at a disadvantage compared to their 50 and 100Gb brethren. So it's also reasonable to perhaps spring for a MCX415A-CCAT to get 100Gbit capability which will allow it not to become obsolete quite so early.
 
Last edited:

i386

Well-Known Member
Mar 18, 2016
4,381
1,617
113
35
Germany
- Can the cheaper CX4 Lx cards e.g. MCX4131A-GCAT 50Gbit cards operate at 40Gbit when connected directly to ConnectX-3 QSFP 40Gbit NICs (of which I already own 3 or 4)? The CX4 Lx product brief seems to indicate so.
yes: https://network.nvidia.com/files/doc-2020/pb-connectx-4-lx-en-card.pdf
It should also be possible to crossflash it with the 100GBE firmware (but nic would be limited by the pcie x8 interface)*
- I found that CX4 non-Lx cards have BCAT and GCAT models which support 40/56GbE or 50GbE, and both are QSFP28. I'd like to know a bit more about what that means since the CX4 product brief didn't seem to clarify.
"QSFP28" is an aggregation of multiple specifications (physical connector and receiver, usage of 28GBit/s serdes, etc.). Basically it means the port and the used cables/transceivers can operate up to 28GBit/s per data lane and qsfp offers 4 data lanes (hence the name qsfp => Quad SFP)
- Would the 50GbE models (e.g. MCX414A-GCAT) not support 40GbE? So I would need to grab a MCX414A-BCAT for compatibility with my QSFP 40Gbit networking?
The 50GBE cards should supports all the speed below 50GBE, including 10 and 40GBE
(again: cards with a qsfp28 connector should be crossflashable to the 100GBE firmware*)
- If I get a MCX414A-BCAT would it not be capable of 50Gbit (2 lanes of SFP28??)? So would choosing the 414A BCAT over the GCAT be like choosing to give up the second SFP28 on a QSFP28 port, in order to gain QSFP 40Gbit compatibility? No?
no, it won't support 50GBE. It's a limitation by the firmware and not the hardware. Could probably be resolved by crossflashing to another firmware*.
- What about 100Gbit? A MCX415A-CCAT is pretty affordable... Is this basically a MCX413A-GCAT but just filling out the second half of the QSFP28? So, maybe same raw bandwidth as a dual port MCX414A but only one port
From my understanding all cx-4 cards with qsfp28 connector(s) have all data lanes connected and are limited by firmware.
Besides the dual port version the MCX415A-CCAT is the most "unlocked" version of a ethernet card in the cx-4 version :D
- I was gonna ask about MCX413A-GCAT vs MCX4131A-GCAT but apparently these are just the same thing.
MCX4131A-GCAT: EN LX network card
MCX413A-GCAT: EN
From my understanding the en LX skus were a line that topped at 50GBE while the "normal" EN skus supported the whole range in that generation

*I don't have any cx-4 with qsfp28 connectos and pcie x8 interfaces. My informations comes from different user posts here in the forums
 

unphased

Active Member
Jun 9, 2022
164
39
28
Thank you! I'm pretty familiar with how some of the CX3 cards I got were "10Gb" cards but I was indeed able to do some flashing to obtain 40Gb capability on them. I haven't yet seen (though havent really searched) anecdotes of the same to reach 100Gbit as you explain, but this is quite exciting as it may mean I should be served well by going for whatever is on x16 pcie lanes and is cheap in the short term, and see what i may be able to do with it later on.

Actually since I'm already considering dipping toes into 100Gbit infrastructure, I've been researching how to actually properly leverage that bandwidth for small numbers of hosts in a home network -- I am aware that without RDMA enabled solutions, CPU utilization will prevent me going beyond or even reaching 40Gbit speeds... Still learning... Meanwhile it looks like I should look for deals on CX4 16 lane cards.

Update: from cursory browsing the 8 lane cards are the only ones that are "cheap" so probably still gonna end up with one of those initially. Being limited to 80Gbit (sub 40 tbh for the common 4 lane off an m.2 slot usage) isn't going to be an issue.
 

unphased

Active Member
Jun 9, 2022
164
39
28
Aha!

OK so I just ordered a MCX455A-ECAT which will do 100Gb out of the box. But I just found that industry has been moving toward NVMe-oF and that iSER etc will be left behind. And that CX5 is what I'll need to get NVMe-oF support. Hmm.
 

unphased

Active Member
Jun 9, 2022
164
39
28
Plot thickens, according to Mellanox Interconnect Community,

> NVMEoF can run over any RDMA capable adapter (e.g. ConnectX-4/ConnectX-5) using IB/RoCE link layer.

I'll not cancel my order because CX5 is definitely twice as expensive as CX4 right now and the CX4 should still work well connected via thunderbolt box as a 20gbit networking enabler for macOS.