[SOLVED] [Rec] Inexpensive Dual-Port 40GbE PCIe

eduncan911

The New James Dean
Jul 27, 2015
637
483
63
eduncan911.com
What are some inexpensive recommendations for Dual-Port 40GbE PCIe cards? The switch is Brocade so there's no vendor locking. Searching Fleabay is turning up a wide range, and some searches here hasn't turned up much.

Are there any "combo" card options, like Dual-Port 40GbE + Dual-Port 1GbE? I'm currently running most of these combo-types with dual-10G + dual-1G. (Proxmox + Ceph require a tun of NICs!)

Single ports are also an option on some servers, but my old equipment are mostly 1Us with only 1 PCIe riser - which needs the 3 to 4 NIC ports.

I guess Quad-40GbE is an option as well instead of the combo cards, and I see those. But waaay out of my price range!

I really only could use a single 40GbE port for the new Ceph pools. The rest could be combined over a single 40Gbps w/vlans. So dual I guess is the most cost effective route.
 

eduncan911

The New James Dean
Jul 27, 2015
637
483
63
eduncan911.com
A lot of the listings for Mellanox CX354A show them as 10G. However, the pics are clearly QSFP ports.

Are they all QSFP ports? Or is there licensing or restricted models at 10G?

Also, these prices ($40) is the same I just paid for a bunch of dual 10G cards. Argh... Heh.
 

NablaSquaredG

Well-Known Member
Aug 17, 2020
667
286
63
Yes, MCX354A-QCBT only support 10GBe or QDR Infiniband, but are apparently physically identical to MCX354A-FCBT, which Support 40GBe/56GBe and FDR 56G Infiniband.

CX354A-QCBT can easily be crossflashed to CX354A-FCBT (you can find many guides in the forum)
 
  • Like
Reactions: eduncan911

eduncan911

The New James Dean
Jul 27, 2015
637
483
63
eduncan911.com
Ah, I got it. So if I pick up one of the QCBTs, I can cross-flash to FCBT to get 40G.

Niiiiiiiiice. Exactly what I was looking for.
 

gtcdma

New Member
Oct 29, 2020
19
7
3
Can MCX354A-QCBT/FCBT be used with QSFP-to-4-SFP breakout cables? Is it possible to get it to work as 8 independent 10gbe ports this way?
 

NablaSquaredG

Well-Known Member
Aug 17, 2020
667
286
63
CX3 Pro is a different silicon.
As far as I have heard you can flash the Pro Firmware on non-pro cards, but you will lose 56G connectivity

@Rand__
 

Rand__

Well-Known Member
Mar 6, 2014
6,189
1,535
113
The 544+ should be a cx3-pro so it should have appropriate FW.

If oyu need more of them its probably easier to just get some,. they are not that much more expensive than CX3's, but yes it shhould be possible to flash. You need a modified flash tool (not sure if it was flint or mlxburn, something that is (or used to be) available as code on github or similar so you can patch out the psid check.

I have not done it myself, only used the modified flash tool (ages ago). Not sure if the converted CX3s were able to run roce v2, never tried since I was annoyed by the fact that they couldnt do FDR14 (for whatever reason), so i replaced them with regular CX3 again.
 

AllenAsm

Member
Jul 30, 2018
53
3
8
The 544+ should be a cx3-pro so it should have appropriate FW.

If oyu need more of them its probably easier to just get some,. they are not that much more expensive than CX3's, but yes it shhould be possible to flash. You need a modified flash tool (not sure if it was flint or mlxburn, something that is (or used to be) available as code on github or similar so you can patch out the psid check.

I have not done it myself, only used the modified flash tool (ages ago). Not sure if the converted CX3s were able to run roce v2, never tried since I was annoyed by the fact that they couldnt do FDR14 (for whatever reason), so i replaced them with regular CX3 again.
Good point, I bought the cx3 PRO cards. I was super careful to only get those for the RDMA and such.