Mellanox connect X3 vs Connect X4

[Boehamian]

Good day to all.
please go easy on me, just starting to teach myself more in IT after being out of IT for 10+ years so go easy, I may as well be a freshman.

Im in the process of building a home server and am investigating network cards. I have come across the Mellanox connect X3 range (which seems cheaper and has faster transfer rates then the connect X4). I am just wanting to confirm a few things as spec sheets are non existent.

The card in question is a MCX354A-FCBS

Does it work with Hyper-V?
Does it support VLAN 802.1Q?
As I only will be running the 2 ports at 10GB at the moment can I put them into a PCI 3 x 4 lane instead of 8 (physical 16 so it will fit but lanes are only 4 speed)?

New to the whole server game (honestly only did desktop support and basic domain installs so really throwing myself into the deep end here
Or can anyone recommend a cheaper card that will work. Be mindful I am trying to find a dual SFP+/QSFP+ that will work in a PCI gen 3 x 4 slot. Yes I know the bandwidth of the x4 slot is not going to be any good past the 10GB modules but it allows for future expansion if I want to move it into another PCI slot.

40GB not a necessity but hey if you can why no right?

 

[Patriot]

ESPCommunity

enterprise-support.nvidia.com

 

[nexox]

I would not suggest not going with QSFP ports unless you're actually going to run 40 or 100G right now, because that increases the cost to connect to SFP hardware by more than the price of a 10G card. If you want to buy something with headroom I guess a dual 25G ConnectX 4LX (SFP28 is backwards compatible with SFP+) would be alright, but these things are dropping in price so fast it makes sense to wait until you need the hardware to purchase it.

I don't know about Windows support but Solarflare s7120 cards are pretty cheap and work well for me in Linux.

 

[blunden]

Note that the ConnectX-3 line has reached End-of-Life, which probably isn't a huge concern on Linux but definitely can be on Windows as I don't think you can necessarily expect any new driver releases for them.

In other words, the ConnectX-4 NICs might be a better bet as at least some of them are still actively supported last time I checked.

 

[OnnoO]

In my experience the 4 LX needs more power and gets a bit hotter than the 3. Only if you need the extra functionality I would suggest the 4 LX.

 

[BackupProphet]

Mellanox ConnectX 3 works good in Linux, without installing the Mellanox drivers. This included SR-IOV and RDMA/RoCE. Not that there can be issues with both SR-IOV and RDMA/RoCE, which can be resolved with a reboot. Some Mellanox 3 network cards from Dell/HP can have custom settings that you cant override. For example I had a Dell Mellanox card where no matter what I did could not enable more than 8 vfunctions.

 

[i386]

which probably isn't a huge concern on Linux but definitely can be on Windows as I don't think you can necessarily expect any new driver releases for them.

Windows 11 and server 2022 still ships drivers for first generation connectx devices

 

[blunden]

Windows 11 and server 2022 still ships drivers for first generation connectx devices

Oh really? That's good to know.

They use an older driver code base from what I can tell so I'm not sure the drivers will be updated if Microsoft makes changes that require driver changes.

 

[Boehamian]

Well in taking your advice I started to look for further devices that are only SFP+ (don't really need the 28GB functionality). The Mellanox MCX312B-XCCT CX312B seem to fit this bill. The only reason I looked at the above card was because it was far cheaper without having to import a card to Australia (rather buy in aus if possible). Found this one for 145, which will still be cheaper that buying the adaptors from QSFP to SFP+ adapators. Also noticed the first card I mentioned only had a max throughput ot 56GB so was never going to get 40g x 2 anyway.

I had a look at the solarflare cards but there is mixed specs as to if those are PCI gen 2 or gen 3. If they are gen 2 they won't work.
I also was not able to find any specs for supporting SR-IOV and RDMA/RoCE and VLANS. the same goes for the broadcom which I think is only PCI gen 2.

The mellanox seems to fit the bill. The only thing they don't mention is the VLAN. But based on a mate is using a connect X3 card and it has VLAN im not too concerned. It would be pretty pointless after all to have a card designed for server hardware without VLAN tagging support.

 

[Boehamian]

Windows 11 and server 2022 still ships drivers for first generation connectx devices

that's good to know.

 

[Boehamian]

I would not suggest not going with QSFP ports unless you're actually going to run 40 or 100G right now, because that increases the cost to connect to SFP hardware by more than the price of a 10G card. If you want to buy something with headroom I guess a dual 25G ConnectX 4LX (SFP28 is backwards compatible with SFP+) would be alright, but these things are dropping in price so fast it makes sense to wait until you need the hardware to purchase it.

I don't know about Windows support but Solarflare s7120 cards are pretty cheap and work well for me in Linux.

I did find this one MCX4121A-ACAT Mellanox ConnectX-4 Lx 25GbE SFP28 2-port PCIe Ethernet Adapter AU | eBay Do you suggest to run this over the connect X3 based on support is still there?

 

[mach3.2]

I did find this one MCX4121A-ACAT Mellanox ConnectX-4 Lx 25GbE SFP28 2-port PCIe Ethernet Adapter AU | eBay Do you suggest to run this over the connect X3 based on support is still there?

You can flash the 25G firmware (ACAT) on the 10G version (XCAT) if you can manage to find it at a cheaper price.

If there's drivers in the OS I'll be using it on, I'll honestly just buy the cheaper card.

 

[nexox]

I had a look at the solarflare cards but there is mixed specs as to if those are PCI gen 2 or gen 3. If they are gen 2 they won't work.
I also was not able to find any specs for supporting SR-IOV and RDMA/RoCE and VLANS. the same goes for the broadcom which I think is only PCI gen 2.

The 7000 series cards are 3.0, and some posts on here suggest they do SRIOV, but I don't know about RoCE. Since you're not in the US the prices are probably not as attractive, they're under $20US for dual 10G SFP+.

 

[Boehamian]

thanks all. Ended up getting a Mellanox ConnectX4 for 160 AUD. Although more expensive if I had of bought it overseas, it makes returns a little easier if required and it means I can build this server while im off work for another 2 months with a broken leg Thank you all for your help. The learning is about to get deep .

 

[SnJ9MX]

I have some 10G solarflare stuff at home that's getting a bit old and recently looked into connect x3 vs 4 lx as well. If you do some searching, you'll see that the 4 has the ability for the OS to enter lower power states, which can save you a bit of power. The 3 will keep things at a higher power state (even if nothing is happening), which will increase your idle power by at least 10W. Not sure the price of electricity in AUS but for those in EU, the delta can add up.

 

[saivert]

Mellanox Connect-X 4 Lx uses a smaller node shrink controller as well and I wonder how much cooling it requires. All the older Mellanox cards require active cooling of some sort. A fan blowing in its direction is enough to prevent throttling, anything to avoid still air.

 

[mach3.2]

Mellanox Connect-X 4 Lx uses a smaller node shrink controller as well and I wonder how much cooling it requires. All the older Mellanox cards require active cooling of some sort. A fan blowing in its direction is enough to prevent throttling, anything to avoid still air.

I recently swapped out a MCX354-FCBT for a MCX4121-ACUT, ASIC temperature for both cards remained the same at 47°C with the same amount of case airflow.

 

[jei]

Received ConnectX-4 Lx CX412A-ACAT today after waiting almost 3 weeks for the local post to move it 150km.

eBay, USD $78.11 with taxes and shipping from China.

Old card was ConnectX-3 EN CX311A-XCAT.

Same reasons as others have mentioned. Windows 11 OS. CX3 uses WinOF driver, latest update is from june 23, 2020. CX4 uses WinOF-2 driver, latest from november 05, 2023.

CX3 had some issues with sleep (with FS.com 10GBASE-T SFP+ copper module). Waking up "too slowly". CX4 seems to have fixed that. Dunno if it's the newer drivers or better ASIC.

 

[thigobr]

What about ASPM support?