Mellanox ConnectX-3 Pro EN (MCX314A-BCCT)

[kapone]

Yeah I saw that already. But again ... IIRC it needs to be a Mellanox Switch, NOT a Dell branded one, that was my only Remark on that front .


So low Power ???


OK so only Stock Firmware basically, probably very old Version and unsupported ...

The Thing is I wouldn't mind settling to 40 gbe, but to get 25gbps like many NICs I have, I'd need QSFP28 and then a QSFP28 to 4xSFP28 Breakout. There is no QSFP+ to SFP28 Adapter, not cheap at least. 25gbe also supports ASPM which most 40gbps/QSFP+ (minus maybe Intel XL710) and surely 100gbps/QSFP28 do NOT.


"Only" RDMA ? No RDMA2 / ROCE2 Support ?

If 25gbps is your target, the SX6036 is not for you.

As far as RDMA is concerned...RDMA is RDMA. i.e. The switch supports PFC/DCB etc for a lossless fabric. What you run over it, depends on your NIC/software stack. Yes, ROCEv2 works.

 

[luckylinux]

If you are looking at rack mounted stuff old mellanox sfp+/qsfp+ switches are fairly cheap.

I still got a sx1024 in use (48x sfp+ 12x qsfp+), those regularly appear on ebay in the 100-150$ area.
sx1016 with 64x sfp+ is maybe around 200$ and the popular sx6036 with 36x qsfp+ (and breakouts for your sfp+) is 30-100$ with plenty available.

sx1000 series and sx6000 are the same units, but 1000 is ethernet while 6000 is for infiniband with optional ethernet as added license.
(There are some threads about the sx6036 on here with information regarding enabling ethernet.)

I've seen some Arista or Celestica "reasonably" Priced switches at around 400-500 USD for QSFP28 (100gbe), but some have better support than others.

The Mellanox SX6036 could be a good Option for those of us that don't really need the Full Bandwidth and are on the Cheap. 40gbe realistic Target, cheap DAC & cheap Tranceivers. Personally I think QSFP28 Tranceivers are actually cheaper ... Look at this UK Seller, he has been selling lots of QSFP28 Transceivers at like 5 GBP / Piece, sometimes less. I don't think it's worth to expect / plan for 56gbps for a special Case of Transceivers, DACs, NICs, etc. Sure if you can get it for cheaper or same Price why not, but don't spend 10x more on it.

Question then becomes Transceivers/DAC Compatibility, because I read that QSFP28 Transceivers are NOT always compatible with QSFP+ Speeds ... unless they are plugged in a QSFP+ Port, which of course would apply to the Mellanox SX6036. Basically would a QSFP28 Transceiver work at 40gbps (is that Data Rate "supported") if plugged into a 40gbps QSFP+ Port ? I agree those QSFP28 CWDM4 Transceivers might be a bit "Special" (4 Light Wavelengths multiplexed into a single SMF Fiber), but for the point of view of the External Equipment, I guess that should be Transparent.

What about the SW Support of the Mellanox SX6036 though ? Running stock Mellanox Firmware or Flashed to SONIC / other ? Running "pure" Ethernet or "pure" Infiniband or "mixed" (1 Port Ethernet, 1 Port Infiniband) ? If running Infiniband I guess you need to run your own Subnet (?) Manager, right ?

Back then (1gbe-10gbe Ethernet vs 40gbps Infiniband) I recall Infiniband was definitively preferred for Block Storage and MUCH lower Latency, but I guess the Gap narrowed down when you compare 40gbe Ethernet / 40gbps Infiniband let alone 100gbe Ethernet / 100gbps Infiniband, so maybe we can get reasonably good Performance using much simpler Ethernet (TCP/IP).

My main Issue is probably going to be Noise though. There's a Reason why I never deployed my Voltaire Infiniband 4036 ... I tried to do a Fan Mod but I guess Priorities shifted around and ... 10 years later it's still not deployed with the Fan Module probably sitting somewhere I don't remember .

Are these Mellanox SX6036 easy to water cool ? I never did Watercooling until now but that seems to be the Only Option. For the HEDGE EPYC we just discussed in the other Thread the Cooling Plate would cost twice as much as the Server so ... nope . But maybe these Mellanox could work with a Water Block ? Unsure ...

EDIT 1: based on what I read in another Thread and IIRC, make sure that it's a REAL Mellanox SX6036 and NOT a Dell/etc branded one, otherwise I think the Ethernet License will NOT be easy to enable .

 

[kapone]

And I guess there are no cheap 25gbe Options since it's relatively new and QSFP28 Switches are > 400 USD ...

Not really.

 

[Cruzader]

And I guess there are no cheap 25gbe Options since it's relatively new and QSFP28 Switches are > 400 USD ...

I belive cheapest is cisco nexus 92160 that is often on ebay in the 250-300 usd area, but gotta ship it from US also.
(if i dont remember too wrong i paid 260+shipping for mine)
That is 48x 25gbe 4x 40/100gbe 2x 40gbe.

The mellanox models ive been eyeing are still 750-1000 area.

 

[luckylinux]

@luckylinux - As far as the SX6036 is concerned...

1. You're making this more complicated than it needs to be. There's a long thread on this forum about them. Read it carefully, I mean it, all your questions, especially *cough* license keys, will be answered.

Yeah I saw that already. But again ... IIRC it needs to be a Mellanox Switch, NOT a Dell branded one, that was my only Remark on that front .

2. The SX6036 supports software control of the stock fans. Spun down to the lowest setting, they're barely audible. Not silent, and wouldn't work under desk, but if your rack is even a few feet away, it's a non-issue. Mine is almost fully populated with DACs and the fans still don't spin up...and it's still consuming ~60w...for 36x 40/56gbe ports.

So low Power ???

3. As far as the firmware is concerned, again read the thread, but no Sonic or anything else for these switches. Once the license keys are in place...you essentially have 36x 40/56gb ETH ports, that can also be broken out into a whole bunch of 10gb ports with the right breakout cables.

OK so only Stock Firmware basically, probably very old Version and unsupported ...

The Thing is I wouldn't mind settling to 40 gbe, but to get 25gbps like many NICs I have, I'd need QSFP28 and then a QSFP28 to 4xSFP28 Breakout. There is no QSFP+ to SFP28 Adapter, not cheap at least. 25gbe also supports ASPM which most 40gbps/QSFP+ (minus maybe Intel XL710) and surely 100gbps/QSFP28 do NOT.

4. And they support RDMA... Slightly painful to setup and configure but not too much. RDMA at 56gb is likely more than enough for 99% (if not more) of homelabs.

"Only" RDMA ? No RDMA2 / ROCE2 Support ?

 

[Cruzader]

Isn't Cisco an Issue with regards to always requiring to have a License ?

These are honor based licensing, they do not need to call home to verify if there is active support etc type stuff.

 

[kapone]

But I mean, no Updates then, right ?

So what?

 

[luckylinux]

If 25gbps is your target, the SX6036 is not for you.

As far as RDMA is concerned...RDMA is RDMA. i.e. The switch supports PFC/DCB etc for a lossless fabric. What you run over it, depends on your NIC/software stack. Yes, ROCEv2 works.

And I guess there are no cheap 25gbe Options since it's relatively new and QSFP28 Switches are > 400 USD ...

 

[BackupProphet]

Yes, I run one port as IB and the other one as Eth just fine.
There may be issues with SR-IOV though.

 

[BackupProphet]

MCX354A-FCCT
Linux support is great. I use Almalinux 10. RHEL has dropped the driver. But the driver is in the official Linux kernel. SR-IOV is not great for that generation.

 

[BackupProphet]

SR-IOV save cpu cycles, and comes with improved latency, throughput. And you can give your VM's RDMA with full performance.

 

[luckylinux]

I belive cheapest is cisco nexus 92160 that is often on ebay in the 250-300 usd area, but gotta ship it from US also.
(if i dont remember too wrong i paid 260+shipping for mine)
That is 48x 25gbe 4x 40/100gbe 2x 40gbe.

The mellanox models ive been eyeing are still 750-1000 area.

Isn't Cisco an Issue with regards to always requiring to have a License ?

 

[Schemer]

SR-IOV save cpu cycles, and comes with improved latency, throughput. And you can give your VM's RDMA with full performance.

Would you happen to know if other people have had issues enabling SR-IOV and RoCE v2? They seem to work separately but when I enable SR-IOV i'm not able to enable RoCE v2 for whatever reason. Im running Mellanox CX-3 Pro MCX314-BCCT flashed to the latest version. All on proxmox pve9, I was curious if you knew of general OFED versions that generally work. Also using mft-4.25.1-11-406 if its relevant. 5 other cards connected through a SX6036. I'm ultimately trying to setup NVMe/RoCE so i'm trying to make sure I get all the individual pieces working

 

[BackupProphet]

With this generation, I highly recommend using these for a single purpose and avoid RoCE. If you want RDMA, use infiniband. If you need SR-IOV, only use virtual functions, and avoid physical functions. Yes they work with RoCE, but sometimes they do not. And since no one knows why, choose the easiest option. Infiniband. So far I've not had any issues with one port running IB, and the other ethernet. But once I enabled a virtual function, the network started to flap. I've been successful with only running virtual functions though. The CX4 generation should be much better here, and RoCE also works much better, and it also supports lossy RoCE.

 

[luckylinux]

These are honor based licensing, they do not need to call home to verify if there is active support etc type stuff.

But I mean, no Updates then, right ?

 

[kapone]

With this generation, I highly recommend using these for a single purpose and avoid RoCE. If you want RDMA, use infiniband. If you need SR-IOV, only use virtual functions, and avoid physical functions. Yes they work with RoCE, but sometimes they do not. And since no one knows why, choose the easiest option. Infiniband. So far I've not had any issues with one port running IB, and the other ethernet. But once I enabled a virtual function, the network started to flap. I've been successful with only running virtual functions though. The CX4 generation should be much better here, and RoCE also works much better, and it also supports lossy RoCE.

Specifically for the Connectx-3 cards, it's how the VFs are created/distributed across the two ports. And heaven forbid if you put them in an LACP bond and then do SR-IOV (the VFs are supposed to be bond aware (with driver support), i.e. if one port fails, the VFs get moved to the second port automatically). It all works great when using the Mellanox OFED drivers, because they included the magic bits in there. The Inbox drivers in Linux don't have those magic bits and kinda work.

And well, the OFED drivers don't work for later kernels.

So, while it works...it's a bit kludgy.

 

[BackupProphet]

Yeah, the drivers. I only use the inbox drivers. The OFED ones are to difficult to install. Same thing with Chelsio, have to use the inbox drivers. Their unified wire drivers only work for outdated distributions.

 

[luckylinux]

Would the Mellanox MCX354A-FCCT also work or does it need to be the Mellanox MCX314A-BCCT ?

Will both Infiniband and Ethernet work on these NICs though ? Is it possible to configure one Port as IB and the other as Ethernet ?

I was considering getting this (but let's hope Linux Support won't be dropped too soon):

Pardon Our Interruption...

 

[BackupProphet]

Inbox drivers are not exactly Mainline Kernel drivers, but Mainline distro drivers, which can be an upstream from the Linux Mainline with minor patches

 

[Schemer]

Gotcha this is good info... I think im going to do a fresh install of proxmox with just mlxconfig/mstflint and see if it works better without ofed drivers in the way. its a hassle to remove the ofed drivers and get to a clean untainted state(let me know if you have good ideas on how to do this). I may take a crack at compiling my own OFED for debian13/PVE9 but thats a way latter thing. I think I should probably just give up on RoCE and use NVMe/TCP which is probably good enough honestly.

 

[luckylinux]

Yes, I run one port as IB and the other one as Eth just fine.
There may be issues with SR-IOV though.

Thank you for your Quick Reply .

Do you mean this for the Mellanox MCX354A-FCCT or Mellanox MCX314A-BCCT specifically ?

Is Linux Kernel Support still good with the latest Kernels (e.g. Kernel 6.14 on Proxmox VE) ?

 

[Schemer]

IIRC I read that above say 25gbps (or maybe 40gbps ?) RDMA/ROCE was basically REQUIRED, otherwise realistically you wouldn't be able to achieve the required Bandwidth.

Interesting, I think I should have been fine in that case? I think I was able to hit 30-40+ during benchmarks for the most part, but who knows how bogged down it could get

 

[kapone]

IIRC I read that above say 25gbps (or maybe 40gbps ?) RDMA/ROCE was basically REQUIRED, otherwise realistically you wouldn't be able to achieve the required Bandwidth.

No it's not. It'll just take more CPU to do the same.

 

[luckylinux]

MCX354A-FCCT
Linux support is great. I use Almalinux 10. RHEL has dropped the driver. But the driver is in the official Linux kernel. SR-IOV is not great for that generation.

Thank you .

What is the Issue with SR-IOV exactly though ? I still didn't start using it and not even sure I will need it ... So far I've been using Linux Bridges in Proxmox VE without any direct PCIe Passthrough into OPNSense VM. If I understood correctly, SR-IOV is basically a Way to have direct NIC Access both on e.g. Proxmox VE Host as well as one or several VMs (e.g. OPNSense) via these "Virtual Functions" / VFs instead of just the "Physical Function" (PF).

It seems some NICs (not only Mellanox) have Issues when using e.g. SR-IOV and VLANs in Combination.

 

[BackupProphet]

I've been testing with and without. There is really no contest, use RDMA if you can. Its like going from a 386 to Pentium 3

 

[Schemer]

I would really like to get it working and I might take a few more days to see if its possible. Just haven't been able to get it working, i've got my vlans and PFC setup but I believe it couldn't find the GUIDs? Not entirely familiar with everything yet. I would consider doing half IB and half Ethernet but cables are 30 dollars a pop minimum for the equivalent of what I bought for 18-20 and for 5 NICs it might be better for me to consider biting the bullet and just buying CX-4s. I'm going to do a fresh install proxmox again on one of my lenovo p520s that will soon be a set of Mirrored SANs serving a cluster of three lenovo m720qs.... If I can get RoCE or NVMe/TCP. Hopefully just using native drivers might give me a fighting chance. Man it would be insane value if someone was able to get CX-3 pros to function with their full set of features in 2025

 

[luckylinux]

SR-IOV save cpu cycles, and comes with improved latency, throughput. And you can give your VM's RDMA with full performance.

Interesting, even saving CPU Cycles ? I thought that would be thanks to RDMA / ROCE v2 instead.

So ... what is NOT working with this Generation of Cards and SR-IOV then ?

 

[Schemer]

I might be having some better luck with infiniband on a fresh install.... im going to have to

Not sure that is a fair Comparison given the Price Difference:

ConnectX-3 Pro are listed at ~ 15-18 USD / Piece for dual Ports, sellers will probably accept Offers at ~ 10 USD / Piece

ConnectX-4 are > 50 USD /Piece all the Way up to 100 USD / Piece for dual Ports, sellers will probably NOT accept much lower Offers

From a quick Search on eBay US, QSFP+ DAC and QSFP28 DAC are all approx. 20 USD / Piece (more if longer). I saw a few Offers in "Lot" for around 10 USD / each for QSFP28.

Pretty sure in Europe there were some good "Lot" (of 10-40 Pieces) Auctions for QSFP+ DACs and it came down to like 7 EUR / Piece or so (cannot remember, I didn't win that one ).

I also got some QSFP28 CWDM4 Transceivers that were like 5-8 USD / Piece so if you get a Deal on OS2 (Single Mode) Fibers that's also a cheap possibility. Those QSFP28 Transceivers are probably CHEAPER than the QSFP+ Transceivers you can find around (unless you find a super good Deal for a Lot of 50-100 QSFP+), at least in my Experience.

So while you could get QSFP28 DAC/Transceivers instead of QSFP+, I'd say that the Cost Difference on the NIC itself is still quite high (Factor 3x ... 7x).

Yea I bought all longer cables haha, im really only comparing to the 3M cables/exact model that previously bought that I know should be able to hit 56Gbps when I was doing my initial research. I also agree that cx4s still really aren't "that" affordable but some others in the homelab discord would disagree hahaha, up to probably at least 80 dollars per NIC is insane to me. I would guess that there are probably cheaper cards that can be flashed to better firmware but I have no clue what those would be. Overall really wish these things would "just work" lmao. I am having a bit better luck with ROCE using inbox drivers and not using sr-iov but I still need to test its connection to another node and, i'll have to wipe it to do that... inbox drivers seem to act funky afterfucking around with ofed drivers on it for so long

 

[Schemer]

Which Cards are you currently testing right now ?

I'm currently using 5 MCX314A-BCCT. Three in a lenovo m720q(i5-8500t) and two in my SAN lenovo p520(Xeon w-2135) in a generic sense they work pretty well, but using special features is a little bit harder. I think I got these for roughly 13 bucks each in March