Mellanox 50G 20USD
- Thread starter BackupProphet
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
It's a slightly awkward middle point, faster and lower latency than 40Gb, but you need a 100Gb QSFP28 switch (and wouldn't you want to do 100Gb, then?). But 100Gb cards are like $90 now, and all 40Gb equipment is super cheap.
This is an interesting Deal.
The Mellanox ConnectX-4 Dual Port that I previously bought (MCX416A-CCAT) is 2x100G was about 75 USD / each including VAT & Shipping from the UK to the EU (55 USD / each without VAT & Shipping).
Another Point to keep in Bind is the Limitations in Terms of PCIe Bandwidth:
- PCIe 3.0 x8: maximum 64 gbps Theoretical Bandwidth
- PCIe 3.0 x16: maximum 128 gbps Theoretical Bandwidth
So the 50gbps fits well with the PCIe 3.0 x8 Interface that Card has. Going from 50gbps to 100gbps makes no sense if they kept it at PCIe 3.0 x8 instead of x16, as then it would be bottlenecked at 64gbps, and that would probably look bad in Benchmarks
.
50gbps seems low, but many PCIe Slots on Servers are actually x8 only, so it could fit as a general Purpose NIC.
On the other Hand one can Argue why did Mellanox/NVIDIA save a few Cents on a PCIe 3.0 x8 Slot when they could have gone with x16. Probably to not cannibalize their higher End Cards I guess ?
So the Dual Port Mellanox ConnectX-4 is actually bottlenecked at PCIe 3.0 x16 quite badly. I mean 100G is still plenty but yeah, you will NOT take advantage of the Dual Port a lot. Except for Redundancy/Resiliency Purposes of course.
EDIT 1: I didn't look much into it, but it looks like you need a special 100G (QSFP28) to 2x50G (QSFP28) Cables, which I find a bit weird to be honest (why cannot we just use a normal QSFP28 100G to QSFP28 100G Cable ?)
EDIT 2: given that I currently don't have any 100G Switch, this Card seems to require a "special" 100G to 2x50G Breakout Cable and that ASPM will likely NOT work (it doesn't on the Dual Port ConnectX-4 that I already have), I think it will be a pass for me. Potentially interesting for the Price but many unknowns ...
Plus I'd need to buy some Low Profile Bracket such as this one (which is hopefully compatible with the CX413A) - Price is for 5 Pieces:
The Mellanox ConnectX-4 Dual Port that I previously bought (MCX416A-CCAT) is 2x100G was about 75 USD / each including VAT & Shipping from the UK to the EU (55 USD / each without VAT & Shipping).
Another Point to keep in Bind is the Limitations in Terms of PCIe Bandwidth:
- PCIe 3.0 x8: maximum 64 gbps Theoretical Bandwidth
- PCIe 3.0 x16: maximum 128 gbps Theoretical Bandwidth
So the 50gbps fits well with the PCIe 3.0 x8 Interface that Card has. Going from 50gbps to 100gbps makes no sense if they kept it at PCIe 3.0 x8 instead of x16, as then it would be bottlenecked at 64gbps, and that would probably look bad in Benchmarks
.50gbps seems low, but many PCIe Slots on Servers are actually x8 only, so it could fit as a general Purpose NIC.
On the other Hand one can Argue why did Mellanox/NVIDIA save a few Cents on a PCIe 3.0 x8 Slot when they could have gone with x16. Probably to not cannibalize their higher End Cards I guess ?
So the Dual Port Mellanox ConnectX-4 is actually bottlenecked at PCIe 3.0 x16 quite badly. I mean 100G is still plenty but yeah, you will NOT take advantage of the Dual Port a lot. Except for Redundancy/Resiliency Purposes of course.
EDIT 1: I didn't look much into it, but it looks like you need a special 100G (QSFP28) to 2x50G (QSFP28) Cables, which I find a bit weird to be honest (why cannot we just use a normal QSFP28 100G to QSFP28 100G Cable ?)
EDIT 2: given that I currently don't have any 100G Switch, this Card seems to require a "special" 100G to 2x50G Breakout Cable and that ASPM will likely NOT work (it doesn't on the Dual Port ConnectX-4 that I already have), I think it will be a pass for me. Potentially interesting for the Price but many unknowns ...
Plus I'd need to buy some Low Profile Bracket such as this one (which is hopefully compatible with the CX413A) - Price is for 5 Pieces:
5pcs Low Profile Bracket for CX415A MCX415A CX455A MCX455A CX555A MCX555A | eBay.de
Entdecke 5pcs Low Profile Bracket for CX415A MCX415A CX455A MCX455A CX555A MCX555A in großer Auswahl ✓ Vergleichen Angebote und Preise ✓ Online kaufen bei eBay.de ✓ Kostenlose Lieferung für viele Artikel!
www.ebay.de
Last edited:
You can use 100G cables, or even 40G cables for this. The main advantage though is how ConnectX 4 is superior to ConnectX 3. RoCE is much much easier to setup.
Are there any advantages of ConnectX4 vs ConnectX3 apart from maybe power consumption? Bought half a dozen of MCX354A-FCBT last year for $10 a pop including shipping. They run at 56gb (FDR) and are also X8 Gen 3. FDR switches in the SX60xx series also seem to be relatively cheap. I can quiet my SX6012 by keeping the fan under 40%. Temps stay in the 40s C in my non-air conditioned room.
Supported on later versions of software . Vmware is particularly awful at supporting old hardware.
Nowadays, Vmware is particularly awful. Period.
Yeah the vmware hardware compatibility list is probably the most narrow of anything that home nerds are running. But suddenly less relevant since broadcom was tanking their free and cheap options.
But I've noticed with vmware the themse is the mellanox cards get support dropped quickly. But the broadcom cards still seem to get supported a long time. Heck I have a emulex card from 2011 that is deprecated but still works on esxi 8.0.
I guess BSD is the next most limiting?
But I've noticed with vmware the themse is the mellanox cards get support dropped quickly. But the broadcom cards still seem to get supported a long time. Heck I have a emulex card from 2011 that is deprecated but still works on esxi 8.0.
I guess BSD is the next most limiting?
D
Deleted member 24947
Guest
These cards do not draw a lot of power. 100 gig cards tend to be in the 15-25 watt range. You also don’t have to deal with balancing traffic across a LAG to get more than 25 gig bandwidth.
You can connect 8 of these to a Mikrotik CRS504-4XQ-IN which is relatively inexpensive and very low power. That’s a great little setup - it almost makes me want to trade in my SN2410.
I might pick up a couple more of these. The bracket isn’t quite right, but I just taped it in place.
You can connect 8 of these to a Mikrotik CRS504-4XQ-IN which is relatively inexpensive and very low power. That’s a great little setup - it almost makes me want to trade in my SN2410.
I got my card in yesterday. It fine with a Mellanox 100 gig twinax cable from speeds of 1 to 50 gig. You only need the breakout cable if you want to connect two of these NICs to a single 100 gig port. It’s a feature, not a limitation.
I might pick up a couple more of these. The bracket isn’t quite right, but I just taped it in place.
D
Deleted member 24947
Guest
I’d message the seller and ask. It’s possible spacing is slightly different. I know that when I got a couple Dell-branded Mellanox 25 gig CX4 cards, the spacing between the two SFP28 ports is different than the Mellanox branded CX4. The Mellanox half-height brackets I had didn’t fit the Dell card.
Of course, if you’re willing to do a little sawing and filing you can get it to work.
Relatively Inexpensive ... Looks at 630 EUR current Price in Germany and 670-710 EUR here in Denmark.
Wouldn't the Bracket I linked in my Thread above work ? Isn't this Card compatible with the CX413A ? Visually it would SEEM to work ...
5pcs Low Profile Bracket for CX415A MCX415A CX455A MCX455A CX555A MCX555A | eBay.de
Entdecke 5pcs Low Profile Bracket for CX415A MCX415A CX455A MCX455A CX555A MCX555A in großer Auswahl ✓ Vergleichen Angebote und Preise ✓ Online kaufen bei eBay.de ✓ Kostenlose Lieferung für viele Artikel!
www.ebay.de
The ConnectX4s require 400LFM of airflow across their heatsinks to keep them cool, while the ConnectX3-VPIs require only 200. Keep that in mind if you use them in low-flow/no-flow passively cooled scenarios (i.e. t740/755s or Wyse 5070 “big”).
I have a 1-port QSFP28 CX4 with Dell part number and genuine Mellanox brackets fit. Generic brackets from China also fit.
In the past I had experience with generic brackets from China that don't fit exactly but will if using more force. The issue with them are the metal tabs where the mounting screws are attached were bent at a point where they were too close to the bracket.
In the past I had experience with generic brackets from China that don't fit exactly but will if using more force. The issue with them are the metal tabs where the mounting screws are attached were bent at a point where they were too close to the bracket.
It's been a few weeks so this might have changed... The guidance I've seen for VMware hardware deprecations indicates CX4-Lx cards *are* still supported by the ESX 9.0 mlx5 driver. Which is kind of shocking given their age.
However, the original CX-4 (non-LX models) were dropped.
Even with no official support you should still be able to claim the "raw" PCIe devices for pass-thru to a VM.
However, the original CX-4 (non-LX models) were dropped.
Even with no official support you should still be able to claim the "raw" PCIe devices for pass-thru to a VM.
From my personal Experience with many Sellers, especially from China: they will most likely just dodge the Question and reply with some nonsense that got nothing to do with my original Question
.
As to the bracket question, I pulled the bracket off a CX455A and installed it on one of these perfectly with no issues.