Mellanox ConnectX3 MCX353A-QCAT? -FCBS?

[unphased]

What are the difference between these and the -FCBT?
I have some 354A-FCBT and -QCBT dual port NICs and I like them because they work and I'm familiar with their management now.

I have need for the even smaller profile single slot QSFP+ cards like the above, which I see on EBay, and I'd like to know if I can get away with the cheaper QCAT and FCBS options. I saw in Nvidias data sheet the specs for the FCBT and FCBS are the same, but I'd like to know what the difference is if any.
Maybe I should just grab the QCBT since it seems likely to work as well as my existing two port QCBTs at 40Gbit.

Notice how the card impinges on my ability to establish a proper deshroud on the GPU, the single port NIC would work perfectly here.
That said, the dual port does allow the case panel to be fully closed. And GPU cooling is already adequate with the stock fans.

 

[i386]

S and T in the suffix of the sku usually refers to the preinstalled bracket

 

[Stephan]

Note... card needs positive air pressure from a fan. If a thin cloth thread (like from those hotel quick repair packs) ~4 cm in length dangling near the black heatsink isn't moving, card will overheat imho.

 

[unphased]

Note... card needs positive air pressure from a fan. If a thin cloth thread (like from those hotel quick repair packs) ~4 cm in length dangling near the black heatsink isn't moving, card will overheat imho.

Good call! Thank you! I will prepare a 40mm fan and thermal sensor on the heat sink for these. Might need to set one up for the LSI card in the other rig too since that feels even hotter than these.

What's a hotel quick repair pack though?

 

[unphased]

S and T in the suffix of the sku usually refers to the preinstalled bracket

Thanks! Still wanna know what a QCAT is tho!

 

[mach3.2]

Thanks! Still wanna know what a QCAT is tho!

I've seen people having trouble getting QCAT cards to do 40GbE ethernet, so caveat emptor. I'll stick to QCBT cards, those go for around $25 before shipping if you stay patient and camp hard enough.

 

[Stephan]

"hotel quick repair pack" = threads on a spool like 1m long and in a couple of colors, and a needle so you can fix clothes if torn or loose button. I use about 4cm of such a thread dangling near a heatsink to gauge if there is enough airflow from a fan.

Edit: "Oracle 7046442" (usually low profile, cheap high profile slot metal from Aliexpress, my seller managed to screw up the screws, sent wrong ones) or "649281-B21" or "Dell 6RKNM" should work all good. Get a Rev A3 or A4. A2-A5 should be ok.

Edit: Here are cables which work at 56 Gbps. I tested Mellanox MC2207130-00A and EMC 038-004-067-01 so far. EMC with slightly strange SX6012 GUI output (I think length was missing) but link went up when forced to 56 Gbps FDR.

038-004-066-01 EMC FDR QSFP+ to QSFP+ copper cable 2M
038-004-067-01 EMC FDR QSFP+ to QSFP+ copper cable 3M
038-900-027-01 EMC FDR QSFP+ to QSFP+ copper cable 5M
038-900-030-01 EMC FDR QSFP+ to QSFP+ copper cable 8M
038-004-236-01 EMC FDR QSFP+ to QSFP+ copper cable 0.5M
038-004-065-01 EMC FDR QSFP+ to QSFP+ copper cable 1M
038-004-069-01 EMC FDR QSFP+ to QSFP+ copper cable 5M
MC2207126-004 Mellanox Passive Copper Cable VPI UP TO 56GB/S QSFP 4M
MC2207128-003 Mellanox Passive Copper Cable VPI UP TO 56GB/S QSFP 3M
MC2207128-0A2 Mellanox Passive Copper Cable VPI UP TO 56GB/S QSFP 2.5M
MC2207130-001 Mellanox Passive Copper Cable VPI UP TO 56GB/S QSFP 1M
MC2207130-002 Mellanox Passive Copper Cable VPI UP TO 56GB/S QSFP 2M
MC2207130-00A Mellanox Passive Copper Cable VPI UP TO 56GB/S QSFP 0.5M
MC2207130-0A1 Mellanox Passive Copper Cable VPI UP TO 56GB/S QSFP 1.5M
MC2207310-XXX Mellanox Active Fiber Cable VPI UP TO 56GB/S QSFP from 3M up to 100M
MC2207312-XXX Mellanox Active Fiber Cable VPI UP TO 56GB/S QSFP from 3M up to 300M
MC220731V-XXX Mellanox Active Fiber Cable VPI UP TO 56GB/S QSFP from 3M up to 100M
MC2207411-SR4L Mellanox Optical Module IB FDR 56GB/S QSFP MPO 850NM UP TO 30M
IBM 00W0061 0.5M
IBM 00W0049 1M
IBM 00W0057 3M

 

[i386]

Thanks! Still wanna know what a QCAT is tho!

MCX354A-QCAT
Dual port qdr + 10GBE nic, a0 revision (see ic part number)

The "legacy" skus were usually a0 revisions aka "rtm" versions that were replaced with fixed/updated skus

 

[Sean Ho]

I have a couple QCAT cards (rev A0), working fine at 10GbE with QSA, haven't tried 40GbE yet. Different PSID, latest fw it takes is 2.33.

If all you're after is single-port, any 353 will work. To be safe, get rev B -- QCBT, TCBT, or FCBT.

 

[MichalPL]

QCBT are 10GbE, you need to flash them to achieve 40 or 56GbE like in the FCB. About cooling - small fan or massive Geforce RTX nearby

 

[unphased]

Thanks for the tips all!

 

[unphased]

Just a quick follow-up.

I did eventually snag a $26 MCX353A-QCBT and was able to very easily use a PopOS 20.04 live USB boot session to get it flashed to FCBT after installing it into this SFF gaming rig here. It's the perfect size, being even lower profile a card than low profile.

I did also learn about the Brocade icx6610 which seems like the way to grow out the high speed network! I do feel like the high power consumption would hold me back mentally on that until I regularly make use of the bandwidth. I wish it didn't suck down 100 watts sitting there idle. If I were to get one I'd definitely do some modding to put some large fans in it and fake an RPM signal or two.

For now I think I'll try to bridge interfaces on my workstation-NAS box to allow internet to flow on these ad hoc connections.

couple snags so far but I have faith that they can be resolved:

1. would like to have wake-on-lan capability. Not sure if this will be hampered by being connected ad-hoc. I have yet to enter the mellanox cards' bios yet. I think I can enable WoL in there?
2. Seems like I can map a network drive (so I can have a SMB shared zfs dataset as a drive letter and use it as a fiber-attached-NVMe steam library on account of not having enough PCIe lanes) but upon booting up again, the drive letter is defunct. It errors out when trying to access with "the local device name is already in use". Couldn't figure that one out and wonder if that will be addressed by setting up bridging to get DHCP. Hopefully one of the suggestions here might work: How do I resolve the error "The local device name is already in use."? Looks like some bone headed legacy windows networking behavior.

 

[Sean Ho]

Glad the CX353 is working out so far! For WoL I'd just send the magic packet to the SFF box's regular gigabit NIC.

 

[unphased]

Glad the CX353 is working out so far! For WoL I'd just send the magic packet to the SFF box's regular gigabit NIC.

That's highly nontrivial though since I do not have an Ethernet run to there, I did the fiber run and that really ought to be all I need. Would pretty much rather dedicate a raspberry pi running 24/7 to implement this if it comes to that. (On Wi-Fi & use GPIO to control power button)