Mellanox ConnectX-3 Pro EN (MCX314A-BCCT)

[BackupProphet]

With this generation, I highly recommend using these for a single purpose and avoid RoCE. If you want RDMA, use infiniband. If you need SR-IOV, only use virtual functions, and avoid physical functions. Yes they work with RoCE, but sometimes they do not. And since no one knows why, choose the easiest option. Infiniband. So far I've not had any issues with one port running IB, and the other ethernet. But once I enabled a virtual function, the network started to flap. I've been successful with only running virtual functions though. The CX4 generation should be much better here, and RoCE also works much better, and it also supports lossy RoCE.

 

[kapone]

With this generation, I highly recommend using these for a single purpose and avoid RoCE. If you want RDMA, use infiniband. If you need SR-IOV, only use virtual functions, and avoid physical functions. Yes they work with RoCE, but sometimes they do not. And since no one knows why, choose the easiest option. Infiniband. So far I've not had any issues with one port running IB, and the other ethernet. But once I enabled a virtual function, the network started to flap. I've been successful with only running virtual functions though. The CX4 generation should be much better here, and RoCE also works much better, and it also supports lossy RoCE.

Specifically for the Connectx-3 cards, it's how the VFs are created/distributed across the two ports. And heaven forbid if you put them in an LACP bond and then do SR-IOV (the VFs are supposed to be bond aware (with driver support), i.e. if one port fails, the VFs get moved to the second port automatically). It all works great when using the Mellanox OFED drivers, because they included the magic bits in there. The Inbox drivers in Linux don't have those magic bits and kinda work.

And well, the OFED drivers don't work for later kernels.

So, while it works...it's a bit kludgy.

 

[BackupProphet]

Yeah, the drivers. I only use the inbox drivers. The OFED ones are to difficult to install. Same thing with Chelsio, have to use the inbox drivers. Their unified wire drivers only work for outdated distributions.

 

[luckylinux]

Yeah, the drivers. I only use the inbox drivers. The OFED ones are to difficult to install. Same thing with Chelsio, have to use the inbox drivers. Their unified wire drivers only work for outdated distributions.

What are "Inbox Drivers" ? Do you mean "Mainline Kernel Drivers" (i.e. Drivers that are Part / shipped together with the Linux Kernel) ?

I never heard that Term "Inbox Drivers" . I mean "Inbox" makes me think about email so .

 

[BackupProphet]

Inbox drivers are not exactly Mainline Kernel drivers, but Mainline distro drivers, which can be an upstream from the Linux Mainline with minor patches

 

[Schemer]

Gotcha this is good info... I think im going to do a fresh install of proxmox with just mlxconfig/mstflint and see if it works better without ofed drivers in the way. its a hassle to remove the ofed drivers and get to a clean untainted state(let me know if you have good ideas on how to do this). I may take a crack at compiling my own OFED for debian13/PVE9 but thats a way latter thing. I think I should probably just give up on RoCE and use NVMe/TCP which is probably good enough honestly.

 

[luckylinux]

IIRC I read that above say 25gbps (or maybe 40gbps ?) RDMA/ROCE was basically REQUIRED, otherwise realistically you wouldn't be able to achieve the required Bandwidth.

 

[Schemer]

IIRC I read that above say 25gbps (or maybe 40gbps ?) RDMA/ROCE was basically REQUIRED, otherwise realistically you wouldn't be able to achieve the required Bandwidth.

Interesting, I think I should have been fine in that case? I think I was able to hit 30-40+ during benchmarks for the most part, but who knows how bogged down it could get

 

[luckylinux]

Interesting, I think I should have been fine in that case? I think I was able to hit 30-40+ during benchmarks for the most part, but who knows how bogged down it could get

With which CPU and which CPU Load is also the Question ?

The other Question is: full Duplex (e.g. 40 gbps Send AND 40gbps Receive at the same Time) or half Duplex (only 40gbps Send OR Receive) ?

I testediperf3 on a Mellanox ConnectX-4 100gbps using several CPU parallel Threads (IIRC each with several Parallel Streams) at between Port1 and Port2 (need a "Trick" to put them in different Namespaces, so you force traffic to go from one Port to the Other).

That was on an Intel Xeon E3-1240 v5 I believe.

Probably capped at 64gbps (due to PCIe Slot) and IIRC even lower at 40gbps by the QSFP+ Cable I have (I don't have QSFP28 DACs ... yet), measured was probably 38 gbps IIRC.

 

[kapone]

IIRC I read that above say 25gbps (or maybe 40gbps ?) RDMA/ROCE was basically REQUIRED, otherwise realistically you wouldn't be able to achieve the required Bandwidth.

No it's not. It'll just take more CPU to do the same.

 

[BackupProphet]

I've been testing with and without. There is really no contest, use RDMA if you can. Its like going from a 386 to Pentium 3

 

[Schemer]

I would really like to get it working and I might take a few more days to see if its possible. Just haven't been able to get it working, i've got my vlans and PFC setup but I believe it couldn't find the GUIDs? Not entirely familiar with everything yet. I would consider doing half IB and half Ethernet but cables are 30 dollars a pop minimum for the equivalent of what I bought for 18-20 and for 5 NICs it might be better for me to consider biting the bullet and just buying CX-4s. I'm going to do a fresh install proxmox again on one of my lenovo p520s that will soon be a set of Mirrored SANs serving a cluster of three lenovo m720qs.... If I can get RoCE or NVMe/TCP. Hopefully just using native drivers might give me a fighting chance. Man it would be insane value if someone was able to get CX-3 pros to function with their full set of features in 2025

 

[luckylinux]

I would really like to get it working and I might take a few more days to see if its possible. Just haven't been able to get it working, i've got my vlans and PFC setup but I believe it couldn't find the GUIDs? Not entirely familiar with everything yet. I would consider doing half IB and half Ethernet but cables are 30 dollars a pop minimum for the equivalent of what I bought for 18-20 and for 5 NICs it might be better for me to consider biting the bullet and just buying CX-4s. I'm going to do a fresh install proxmox again on one of my lenovo p520s that will soon be a set of Mirrored SANs serving a cluster of three lenovo m720qs.... If I can get RoCE or NVMe/TCP. Hopefully just using native drivers might give me a fighting chance. Man it would be insane value if someone was able to get CX-3 pros to function with their full set of features in 2025

Not sure that is a fair Comparison given the Price Difference:

ConnectX-3 Pro are listed at ~ 15-18 USD / Piece for dual Ports, sellers will probably accept Offers at ~ 10 USD / Piece

ConnectX-4 are > 50 USD /Piece all the Way up to 100 USD / Piece for dual Ports, sellers will probably NOT accept much lower Offers

From a quick Search on eBay US, QSFP+ DAC and QSFP28 DAC are all approx. 20 USD / Piece (more if longer). I saw a few Offers in "Lot" for around 10 USD / each for QSFP28.

Pretty sure in Europe there were some good "Lot" (of 10-40 Pieces) Auctions for QSFP+ DACs and it came down to like 7 EUR / Piece or so (cannot remember, I didn't win that one ).

I also got some QSFP28 CWDM4 Transceivers that were like 5-8 USD / Piece so if you get a Deal on OS2 (Single Mode) Fibers that's also a cheap possibility. Those QSFP28 Transceivers are probably CHEAPER than the QSFP+ Transceivers you can find around (unless you find a super good Deal for a Lot of 50-100 QSFP+), at least in my Experience.

So while you could get QSFP28 DAC/Transceivers instead of QSFP+, I'd say that the Cost Difference on the NIC itself is still quite high (Factor 3x ... 7x).

 

[Schemer]

I might be having some better luck with infiniband on a fresh install.... im going to have to

Not sure that is a fair Comparison given the Price Difference:

ConnectX-3 Pro are listed at ~ 15-18 USD / Piece for dual Ports, sellers will probably accept Offers at ~ 10 USD / Piece

ConnectX-4 are > 50 USD /Piece all the Way up to 100 USD / Piece for dual Ports, sellers will probably NOT accept much lower Offers

From a quick Search on eBay US, QSFP+ DAC and QSFP28 DAC are all approx. 20 USD / Piece (more if longer). I saw a few Offers in "Lot" for around 10 USD / each for QSFP28.

Pretty sure in Europe there were some good "Lot" (of 10-40 Pieces) Auctions for QSFP+ DACs and it came down to like 7 EUR / Piece or so (cannot remember, I didn't win that one ).

I also got some QSFP28 CWDM4 Transceivers that were like 5-8 USD / Piece so if you get a Deal on OS2 (Single Mode) Fibers that's also a cheap possibility. Those QSFP28 Transceivers are probably CHEAPER than the QSFP+ Transceivers you can find around (unless you find a super good Deal for a Lot of 50-100 QSFP+), at least in my Experience.

So while you could get QSFP28 DAC/Transceivers instead of QSFP+, I'd say that the Cost Difference on the NIC itself is still quite high (Factor 3x ... 7x).

Yea I bought all longer cables haha, im really only comparing to the 3M cables/exact model that previously bought that I know should be able to hit 56Gbps when I was doing my initial research. I also agree that cx4s still really aren't "that" affordable but some others in the homelab discord would disagree hahaha, up to probably at least 80 dollars per NIC is insane to me. I would guess that there are probably cheaper cards that can be flashed to better firmware but I have no clue what those would be. Overall really wish these things would "just work" lmao. I am having a bit better luck with ROCE using inbox drivers and not using sr-iov but I still need to test its connection to another node and, i'll have to wipe it to do that... inbox drivers seem to act funky afterfucking around with ofed drivers on it for so long

 

[luckylinux]

I might be having some better luck with infiniband on a fresh install.... im going to have to


Yea I bought all longer cables haha, im really only comparing to the 3M cables/exact model that previously bought that I know should be able to hit 56Gbps when I was doing my initial research. I also agree that cx4s still really aren't "that" affordable but some others in the homelab discord would disagree hahaha, up to probably at least 80 dollars per NIC is insane to me. I would guess that there are probably cheaper cards that can be flashed to better firmware but I have no clue what those would be. Overall really wish these things would "just work" lmao. I am having a bit better luck with ROCE using inbox drivers and not using sr-iov but I still need to test its connection to another node and, i'll have to wipe it to do that... inbox drivers seem to act funky afterfucking around with ofed drivers on it for so long

Which Cards are you currently testing right now ?

 

[Schemer]

Which Cards are you currently testing right now ?

I'm currently using 5 MCX314A-BCCT. Three in a lenovo m720q(i5-8500t) and two in my SAN lenovo p520(Xeon w-2135) in a generic sense they work pretty well, but using special features is a little bit harder. I think I got these for roughly 13 bucks each in March

 

[luckylinux]

I'm currently using 5 MCX314A-BCCT. Three in a lenovo m720q(i5-8500t) and two in my SAN lenovo p520(Xeon w-2135) in a generic sense they work pretty well, but using special features is a little bit harder. I think I got these for roughly 13 bucks each in March

I tried to offer 9 USD / Piece to the Seller for 10 Pieces. Let's see .

 

[luckylinux]

Seller accepted 9 USD / Piece for 10 Pieces .

I'm kinda tempted to order more of them now though and try even lower .

 

[kapone]

Seller accepted 9 USD / Piece for 10 Pieces .

I'm kinda tempted to order more of them now though and try even lower .

Stockpiling old…EOL…nics….That’s new to me.

Are you actually going to put them to use…or just a shopping spree??

 

[luckylinux]

Stockpiling old…EOL…nics….That’s new to me.

Are you actually going to put them to use…or just a shopping spree??

The ConnectX-4 QSFP28 are > 50 USD / Piece and it seemed like the ConnectX-3 Pro still work "OK" according to what @BackupProphet and @Schemer said.

I was looking for a SX6036 Switch Yesterday but either they are 100 USD or so (I tried offering 40 ... 60 USD / Piece for 5 x, all auto rejected) or they are 40 USD or so with an Insane Amount for Shipping .

Plan is to use them, question is just the Timeline. Pretty sure I bought Infiniband ConnectX-2 (or even ConnectX) VPI together with the Voltaire 4036 and never deployed them due to the Noise and all of the moves I've been going through. So yeah, I definitively need to get it done this Time.

EDIT 1: and yeah, maybe better to just take these 10 and see how it goes and how many more I need