MELLANOX MNPA19 XTR ConnectX®-2

[wepee]

Hi guys,


I need some advice, I came across this interesting Youtube video about 10GB peer-to-peer networking

I am very new to the world of SFP+ networking, definitely a beginner here.
Yes, these cards are like dinosaurs, but are they worth buying in 2023-2024, as they are affordable to buy in my local market?

Here are my questions:
1) I know the card is very old, and the company: Mellanox had acquired by Nvidia.
Are the drivers already build-in into Linux distros which is based on Debian family including Promox Hypervisor?
How about Windows 11, pfSense or they have deprecated them?

2) Is the card backward compatible? Like, can I hook it up to SFP+ DAC cable from the card to a 2.5Gbps SFP+ switch?

Thank you.

 

[i386]

but are they worth buying in 2023-2024

in my opinion no.
the drivers & firmware were not updated in ages and I believe the 10years+ old components used on the card could be problematic. I would go with conenct-x4 or newer (if you want to use mellanox).

How about Windows 11

windows 11/server 2022 ships with drivers for it, but better install the "official" mellanox drivers (they add menus for firmware/tuning settings).

2) Is the card backward compatible? Like, can I hook it up to SFP+ DAC cable from the card to a 2.5Gbps SFP+ switch?

if the card is connected to a sfp+ 10gbe port on the switch then yes.

Edit: Did some googling about older electronic components and their failures and found this wiki page Failure of electronic components - Wikipedia

 

[wepee]

in my opinion no.
the drivers & firmware were not updated in ages and I believe the 10years+ old components used on the card could be problematic. I would go with conenct-x4 or newer (if you want to use mellanox).

Thanks. I was hoping that you recommend it.

Which model is it do you recommend?
Are you referring to Mellanox CX4121A MCX4121A-ACAT ConnectX-4 25Gigabit Ethernet Card PCI-E 3.0 ?

I think, it is way too overkill.

The reason I choose this card is........
My old motherboard has 2 slots of PCIE v2 x 16 mode, which this card is ideal for my NAS project.

 

[wepee]

Edit: Did some googling about older electronic components and their failures and found this wiki page Failure of electronic components - Wikipedia

Yes all electronic components are subject to hardware failures.
I am sure Mellanox card were made to be durable product for their client like Nvidia before Mellanox was acquired by Nvidia.

 

[mach3.2]

CX2 is pretty damn old with drivers that hasn't been updated in a few years.

CX4 Lx isn't really that expensive (~40-50usd) and you get more recent driver support.

Barring the CX4 Lx, the CX2 is just poor value proposition when CX3s cost a few bucks more.

 

[wepee]

CX2 is pretty damn old with drivers that hasn't been updated in a few years.

CX4 Lx isn't really that expensive (~40-50usd) and you get more recent driver support.

Barring the CX4 Lx, the CX2 is just poor value proposition when CX3s cost a few bucks more.

Ok noted. Any model which has a single SPF+ port that you can recommend?

 

[mach3.2]

take a look at a mellanox connectx-3 EN/4 Lx datasheet

 

[wepee]

CX2 is pretty damn old with drivers that hasn't been updated in a few years.

CX4 Lx isn't really that expensive (~40-50usd) and you get more recent driver support.

Barring the CX4 Lx, the CX2 is just poor value proposition when CX3s cost a few bucks more.

Ok, thanks, I found this information.

 

[BackupProphet]

I still run the Mellanox ConnectX 2 cards, they do work just fine. You can even get RDMA to work on them. They are also power efficient and do not get hot. If you can find them for 10 USD I would consider them fine. But as long you can get ConnectX4 for 50 USD, that is a better deal.

ConnectX 3 does not offer much more. You get working sr-iov, but getting it to work requires some work. You get working RDMA, but with some issues. The OFED drivers are outdated, and Mellanox has removed NFS over RDMA support.

None of these issues for ConnectX 4

 

[wepee]

I still run the Mellanox ConnectX 2 cards, they do work just fine. You can even get RDMA to work on them. They are also power efficient and do not get hot. If you can find them for 10 USD I would consider them fine. But as long you can get ConnectX4 for 50 USD, that is a better deal.

ConnectX 3 does not offer much more. You get working sr-iov, but getting it to work requires some work. You get working RDMA, but with some issues. The OFED drivers are outdated, and Mellanox has removed NFS over RDMA support.

None of these issues for ConnectX 4

Thanks for sharing.

Unfortunately, Connectx 4 option is out of my budget, I looked up the card at ebay.com and aliexpress.com,
all listings are selling dual ports, not single port.

On top of that, it does not come like a kit, which include an used DAC cable.
Hence, I need to fork out a lot more money to buy....... 2 cards, 2 transceivers, DAC cable depends on the length.

I am not from US, so the cost of buying 2nd parts is actually X4.5 more than what you normally pay.

I think at this moment, I am inclined to buy Connectx 3, as it does comes with single port + DAC cable.

Just want to make sure I am in the right track:

Let's say have a NAS storage using NVME, in the NAS I have Connectx-2, or 3, 4 ------------->connect----------> switch: 10Gb

The switch has 4 x 2.5Gb ports------------>connect-----------> PC desktop with 2.5Gb NIC

Can I get the almost 2.5Gbps download & upload speed measuring from my PC desktop?

Thank you.

 

[BackupProphet]

Yes you will get that, it is fast enough for up to 10GbE without needing rdma. I've used it for ages to connect my Steam Hard drive to a iSCSI tcp hard drive and I get over 1GB/s for those games that are optimized to read that fast.

 

[ms264556]

This is good timing... I was just looking at the same cards today, to connect a SFF Win11 PC to SFP+ port on a switch.

There are several forum/reddit postings enthusing over the CX2's low heat output, which was the attraction. Are CX3 / CX4 parts as cool?

 

[BackupProphet]

Yes, they are slightly more energy effective. like 0.5-1 watt.

 

[wepee]

This is good timing... I was just looking at the same cards today, to connect a SFF Win11 PC to SFP+ port on a switch.

There are several forum/reddit postings enthusing over the CX2's low heat output, which was the attraction. Are CX3 / CX4 parts as cool?

Here this what I found CX3 cards, but not sure about CX4 though:

 

[mach3.2]

Here this what I found CX3 cards, but not sure about CX4 though:
View attachment 33037

Slightly higher power usage as specified in the CX4 Lx datasheet, but the ASIC on both the MCX354A-FCBT and MCX4121A-ACUT runs at the same temperature with the same amount of airflow when I swapped the CX3 out for the CX4 Lx. I use both ports at 10GbE, using a DAC cable.

 

[saivert]

Just make sure the network card receives some air circulation. This can easily be fixed by mounting a fan pointed at the card. It is not necessary to mount a fan to the card itself. I experienced throttling with ConnectX-3 before I did so. It was crammed in between the PSU and the motherboard in the lowest PCIe slot so it was far away from any airflow path in my case before I mounted an extra fan.

At least rule out this as a cause of trouble if you don't get line rate with iperf3 between two machines.

 

[blunden]

2) Is the card backward compatible? Like, can I hook it up to SFP+ DAC cable from the card to a 2.5Gbps SFP+ switch?

2.5GbE and 5GbE are newer standards than 10GbE. Therefore, the answer is technically no. It cannot by itself negotiate those lower speeds.

However, if you are talking about a switch with at least one 10GbE SFP+ port, your other devices connected to 2.5GbE ports will be able to communicate with it at their max speed.

 

[ms264556]

How is Connectx-4 compatibility with SFP+ DACs? Can I just grab whatever's cheap and be pretty confident it'll work?

 

[wepee]

Just make sure the network card receives some air circulation. This can easily be fixed by mounting a fan pointed at the card. It is not necessary to mount a fan to the card itself. I experienced throttling with ConnectX-3 before I did so. It was crammed in between the PSU and the motherboard in the lowest PCIe slot so it was far away from any airflow path in my case before I mounted an extra fan.

At least rule out this as a cause of trouble if you don't get line rate with iperf3 between two machines.

If the PC is jammed pack with wires, then airflow will definitely suffer. Unless you the wires are hide underneath the motherboard.
This card are designed to fit into server board, where loud fans are installed to deal with air flow pulling from the front and blow to the back.
However, my take is probably install a low profile fan like maybe Noctua brand will do the extra cooling job, and reduce the fan sound for a home lab
environment, if you are using a normal desktop PC case.

 

[wepee]

2.5GbE and 5GbE are newer standards than 10GbE. Therefore, the answer is technically no. It cannot by itself negotiate those lower speeds.

However, if you are talking about a switch with at least one 10GbE SFP+ port, your other devices connected to 2.5GbE ports will be able to communicate with it at their max speed.

Thanks for clarifying my question.

I had some doubt about Mellanox old CX2 and CX3 card can supports new standard 2.5Gb + 5Gb speed.

I am thinking buying this super cheap 2.5bp + 10Gbp switches, introduced by Patrick:
NEW Sub $69 Fanless 2.5GbE Switches with 10GbE

Just 1 more question, does the DAC cable works on the 10Gbp switch ports?