Upgrade to 25GbE (Mellanox ConnectX-4)

[pixelwave]

I am in the market to upgrade my network infrastructure to 25GbE.

Therefore I would like to purchase three PCIe cards (Workstation, Server, Backup Node) and after reading the STH article (https://www.servethehome.com/mellanox-connectx-4-lx-mini-review-ubiquitous-25gbe/) I guess price-wise the Mellanox ConnectX-4 (MCX4121A-ACAT) seems like a good solution.

I used and flashed ConnectX-3 cards in the past and had good results. But it is of course a bit hit and miss with the sellers on ebay.

Are there currently counterfeit ConnectX-4 cards in circulation - is there an (easy) way to identify those? I also see cards "Made in Isreal" and "Made in China". The China ones being the newer dates / revisions (2018 and later)

Any recommendations of sellers shipping to Europe?

 

[TRACKER]

Hi Pixelwave,
i have good experience with this one here
i bought two cards and they arrived for ~2 weeks.
Worked flawlessly with SFP28 modules from FS

 

[pixelwave]

Thanks! I saw those seller ... did you research into the differences between the "Made in Isreal" and "Made in China" cards? I remember there was a debate when I looked into ConnectX-3 series cards a while back. The ones from your reseller mentions "made in Isreal" .. is this also what you received?

"Made in China" version seems to be a lot "newer" (2016 vs 2018/19) ... but newer is not always better.

 

[TRACKER]

I am not sure what's the difference between "made in Israel" and "made in China" cards.
Mine are China ones, manufactured in 05/2020.

 

[i386]

The "made in israel" (or mellanox parts sold from vendors located in israel) could be first revisions or in worst case be engineering samples/prototypes (Mellanox had their r&d near Haifa)

 

[pixelwave]

Thanks for clarification. Ok so I assume most sellers use placeholder photos anyways. Also saw same serial number with different sellers and other weird stuff.

 

[pixelwave]

I ordered three cards from your suggestion.. lets see…

 

[pixelwave]

The cards arrived .. I was quite surprised all three "Made in India". So far I was only aware of "Made in China" and "Made in Isreal" for Mellanox products. One PCB looks quite different then the others ... although same origin, batch, time.

 

[Stephan]

Too many white labels... smells like genuine.

 

[nasbdh9]

One PCB looks quite different then the others

Their PCB suppliers are different, one of the card is Founder PCB, and the other two cards are TTM Technologies

 

[i386]

Their PCB suppliers are different, one of the card is Founder PCB, and the other two cards are TTM Technologies

Where do you find information like that?
Connections? Working in that area?

Edit: ttm logo/signs can be found on one of the pics

 

[nasbdh9]

Where do you find information like that?
Connections? Working in that area?

Edit: ttm logo/signs can be found on one of the pics

This obvious PCB difference will also exist on the memory of the same model but different production weeks (even the same week)

There are also other accessories. example common PCB suppliers for graphics cards include SCC, Victory Giant, Avary Holding, Unimicron, and Foxconn (the PCB quality of different suppliers is different, of course, this is another story)

 

[darthray]

@pixelwave, I'm debating doing this upgrade myself. What are you thinking in terms of 25Gbe switches? Do you have something already or are you researching? Would like a managed one for my home lab.

 

[pixelwave]

My (temp) setup currently is as follows:

Fritzbox 7940 (Router / Internet access) > 1 GbE connection to:

Ubiquiti UniFiSwitch Aggregation (10G Managed Switch, 8x SFP+) with:

• Proxmox Server (ConnectX-4) - DAC
• Proxmox Backup Server (ConnectX-4) - DAC
• Windows 11 Pro Machine (ConnectX-4) - DAC
• Mac Studio (10 GbE built-in) - 10Gbase-T

... with SFP link to Ubiquiti UniFiSwitch 16 Rackmount Gigabit Managed Switch, 16x RJ-45, 2x SFP, PoE+, Gen2 (USW-16-POE).

Something like this (not all devices currently running that is why some do not show up):


Cables I purchased are already one level higher ... so the ConnectX-4 cards are connected with SFP28 cables to the unifi switch. It auto-negotiates down to 10GbE SFP+.

So later I just can change the switch and have 25 GbE. Or I can do a direct link (ConnectX-4 to ConnectX-4) to have full access to 25 GbE speeds if necessary which also is convienient since they have all dual ports.

Currently the SFP28 switch options are not excatly what I am looking for - and for general usage 10GbE is more than enough ... at the moment.

Unifi Controller runs on my Proxmox Server.

 

[Semowett]

Cables I purchased are already one level higher ... so the ConnectX-4 cards are connected with SFP28 cables to the unifi switch. It auto-negotiates down to 10GbE SFP+.

So later I just can change the switch and have 25 GbE. Or I can do a direct link (ConnectX-4 to ConnectX-4) to have full access to 25 GbE speeds if necessary which also is convienient since they have all dual ports.

The latter, only way to achieve 25GBs there is direct connection between Connectx-4s. If your willing to change the switch, you would require a USW-Pro-Aggregation which has (28) 10G SFP+ ports and (4) 25G SFP28 ports.

USW-AGG and USW-16POE (even the USW-PRO models) are all SFP+ (10GB) max.

 

[SnJ9MX]

Does anyone know the power consumption of the Connectx-4 LX? I don't see the figure on the datasheet (https://network.nvidia.com/files/doc-2020/pb-connectx-4-lx-en-card.pdf). Looking to reboot my 10G network. I have 2x solarflare sfn7122f's (datasheet says 5.9W, but they definitely get hot). Need to get at least 2 more 10G NICs. Curious if I should replace those Solarflare NICs with CX-4 LX or just use them in a mixed environment.

Edit: should've done some searching
9.3W - Mellanox ConnectX-4 Lx Dual Port 25GbE DA/SFP NIC Review
7.4W - https://www.servethehome.com/mellanox-connectx-4-lx-mini-review-ubiquitous-25gbe/ (also lol at the CX-6 figure - "contact mallanox")