ConnectX 3 vs 4 vs 5 and semi-related optics question

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
jolness

jolness

Member
May 25, 2020
35
10
8
Hey all,
I am going to grab a a 10Gbe NIC for my servers and I think Mellanox seems like where I will go (relatively cheap, even for the SFP28 models if I ever want/need to use 25Gbe and they are transceiver/optic agnostic) but honestly I am hitting decision paralysis.
Prices are
ConnectX-3 $20ish
ConnectX-4 (lx) - $30ish
ConnectX-5 - $60ish
I know the former 2 are EoL (and maybe the connectX-5, seems like they do not support NICs as long as Intel who I am more familliar with) but I am using them with TrueNAS scale and Proxmox so my hunch is it really doesn't matter.
At the moment I am leaning toward the 4. More modern, SFP28 and an extra $10 is not meaningful. An extra $30 isn't much of a jump but if it's a "doesn't matter for my usecase, even in 5 years time" situation then might as well not. If there are benefits to the 5 I am missing, please let me know.
Just want to make sure I am understanding properly.
My big concerns are
1) it being viable in linux based OSes for the next few years at least
2) a 120mm noctua fan at 100% blowing over the PCIe slots being enough airflow since I do not have rack mount chassis (can get a little fan to mount to the heatsink if needed but currently have the fan cooling my HBA and thermals are fine on the card, even under heavy I/O)

Optics:
I have a Sodola 8 Rj45+ 1 SFP+ port unmanaged switch (new style with the two rows of 4 RJ45), as far as I can tell, they do not seem to be the least bit picky about optics but want to make sure before I order the transceivers with the NICs and 1 for the switch.

Thanks in advance!
 
N

nexox

Well-Known Member
May 3, 2023
1,523
734
113
The driver for the 4 and later cards is unified and quite a lot of people run ConnectX 4 in production still, so that should be supported for a really long time. I'd just get the 4 Lx, by the time you have a good reason to use a 5 they will probably be less than $30.
 
  • Like
Reactions: jolness
jolness

jolness

Member
May 25, 2020
35
10
8
@nexox that was what I had gathered but wanted to make certain. I didn't know that about the drivers. Most of my focus was if the large PC OEM branded ones could be flashed to run mellanox firmware (yes without much work it seems). I couldn't find a reason to get the X5 but figured there might be some issue on the 4 or feature on the 5 worth worrying about.
They're agnostic on optics, correct? I haven't found any "mellanox compatible" like with brands who do try to restrict things.

Appreciate the response!
 
N

nexox

Well-Known Member
May 3, 2023
1,523
734
113
The features added to the ConnectX 5 are mostly rather niche and often hard to even use with the publicly released drivers, the transceivers aren't brand locked and I haven't found anything that doesn't work with my SFP28 cards. Do make sure to buy one with the correct bracket for your system, brackets on their own are like 40% of the price of a whole NIC.
 
  • Like
Reactions: jolness
jolness

jolness

Member
May 25, 2020
35
10
8
@nexox Perfect. My needs are basic. I just need an SFP+ compatible card that runs under Linux and doesn't have a ton of bugs.
I found a super cheap low profile bracket clad CX5 but couldn't even find a full width bracket. The ones for the CX4 ended up making the "cheap" one I found no cheaper once I paid for shipping on both items lol. I made that mistake on an HBA many years ago and I lucked out, the seller was super nice and tossed a full width one in a bubble mailer for free. That was 6-7 years ago and I still check that seller first whenever I need something.

Anyways, I appreciate the help. I was 90% sure this was the direction I would go but I've made a few 90% purchases and end up finding I should have checked for that last 10%. o_O
 
  • Like
Reactions: nexox
R

richardm

Member
Sep 27, 2013
50
16
8
Which optics/DACs did you end up with? I'm pricing this stuff out and the stupid modules and/or cables end up costing more than the card itself! There are less expensive parts on eBay but they're not specifically for Dell (or Mellanox).

For every post I find where someone says "anything will work" there's another post where someone ran into what seems like a vendor lock.
 
  • Like
Reactions: homeserver78
N

nexox

Well-Known Member
May 3, 2023
1,523
734
113
I have used all kinds of DACs, AOCs, and RJ45 modules in my ConnectX 4 Lx cards and never found anything that didn't work, my 100G ConnectX4 non-Lx seems a little more picky, but the Lx appears happy with basically anything shaped like an SFP.
 
  • Like
Reactions: pimposh, richardm and MountainBofh
MountainBofh

MountainBofh

Beating my users into submission
Mar 9, 2024
395
289
63
nexox said:
I have used all kinds of DACs, AOCs, and RJ45 modules in my ConnectX 4 Lx cards and never found anything that didn't work, my 100G ConnectX4 non-Lx seems a little more picky, but the Lx appears happy with basically anything shaped like an SFP.
Click to expand...
That's been my experience with the ConnectX-4 LX cards as well. I haven't yet seen anything that doesn't work in them.
 
  • Like
Reactions: pimposh and richardm
R

richardm

Member
Sep 27, 2013
50
16
8
nexox said:
I have used all kinds of DACs, AOCs, and RJ45 modules in my ConnectX 4 Lx cards and never found anything that didn't work, my 100G ConnectX4 non-Lx seems a little more picky, but the Lx appears happy with basically anything shaped like an SFP.
Click to expand...
Nice to hear. Over on FleaBay I've pulled the trigger on a pair of Dell-branded 4121s and a pair of 15m fs.com active optical cables. Only need 5m but these are mis-priced (IMHO) and cheaper than a 5m DAC assembly.

I've noticed cables/optics listed with Cisco equivalent part numbers seem cheaper than listings mentioning Mellanox. Also I've spotted a handful of Cisco part numbers (without mentioning Cisco by name) in some of the Mellanox release notes. I take this as an admission that at least Cisco stuff would work.

I'm a miserable cheapskate and want to pay $5 a pop for optics, not $20. Maybe next time I see something SFP-shaped for $5/ea I'll jump on it.
 
K

kapone

Well-Known Member
May 23, 2015
1,389
823
113
i386 said:
Cx3 roce v1 only
Cx4 First Gen with 28gbit/s serdes (25 & 100gbe), roce v1 & v2
Cx5 second Gen with 28gbit/s serdes, First Gen with pcie 4.0 Support (for dual Port skus)
Click to expand...
Ahhh.....Minor thing...

richardcastle.gif

Cx3 roce v1 only
Cx3 Pro roce v1 & v2
Cx4 First Gen with 28gbit/s serdes (25 & 100gbe), roce v1 & v2
Cx5 second Gen with 28gbit/s serdes, First Gen with pcie 4.0 Support (for dual Port skus)
 
  • Like
Reactions: jode and nexox
You must log in or register to reply here.
Top Bottom