Dell cx94x/24gfd BCM57414 2*25GbE NIC - RoCE performance and FW updates

[NPS]

Hi,

I got a pair of Dell cx94x FH (low profile is 24gfd) BCM57414 NICs used. They work but FW is old and not the same on both cards. The last couple of month I was checking out various ways of building a NAS that is fast for single-user access (there will be multiple users, but most of the time not concurrent) but not to energy hungry (yeah thats vague by design ). I tried TrueNAS/NFS, Proxmox/Ceph and Ubuntu/ZFS/NFS/RoCE. Yes, I did read the odyssey of forum member Rand_ As I found no performance gains from using NFSoRDMA over NFSoTCP even using a tmpfs ramdisk, I invested quite some time in trying to update the FW of my two NICs.

So what are my findings?

- Broadcom does not support Ubuntu. Do not try to install anything you can download at broadcom.com on your Ubuntu. It either does not work, or even breaks something. The Ubuntu Inbox drivers/tools are quite nice however! Whats not working properly is the bnxtnvm tool for fw-updates/-settings. Seems Ubuntu is missing the bnxt_lfc part even in the hirsute developement release.

- Next one Fedora 33. This is a litte better. There is no support for official Broadcom drivers (should be no problem), but the bnxtnvm FW tool does not throw warnings about bnxt_lfc. FW update is not working non the less "Invalid package file. Package installation failed" (got the same message plus warnings on Ubuntu). So this is where I started to worry that I can not flash the Dell cards with Broadcom tools despite the fact, that the Part Number, the tool is reading from the card, is the same. Yes I tried all the combinations of various tool/fw versions I count think of. So I started to dig at Dell for FW again. Tried that before but failed because you can not search for the NIC you have at Dell. You need to guess what systems where sold with it and then search for drivers/fw for that system. Found the FW updating tool but as I kind of expected, the result was "This Update Package is not compatible with your system configuration.".

- So let's try CentOS 8.2 since RHEL 8.2 is officially supported by the Dell packages. Driver installation: works! FW update: you allready guessed it. "This Update Package is not compatible with your system configuration."

My question:

Has anybody any idea what to try next out of buying a Dell socket 3647 or SP3 server?
- Do I even need to bother beeing nice to some people who maybe could let me use a older Dell machine to do my FW updates? How crazy is that whole "this device is not supported in combination with that system" at Dell?
- Did i miss something? Is there another way of getting this FW update done?
- Is NFSoRDMA/RoCE simply slow and I don't need to bother? Is Mellanox faster? How much faster is Connectx-6 Dx compared to Connectx-4 Dx? (6 needs more power than 4)
- I really like the idea of putting a BCM957504-P425G 4*SFP28 NIC in my NAS, so I can connect all my machines without the need for a loud/energy-hungry/expensive 25GbE-switch, but if I do not find significantly more performance using 25GbE, I will settle for 10GbE (without RoCE as there is no PFC in silent switches).


So this post has become a litte longer... thanks for reading if you made it here! I will happily answer questions about the hardware/software/settings but thought I should not make this even longer right from the start.

 

[LodeRunner]

If you have Windows workstation, plug them in there and try Dell's DUP packages?

 

[NPS]

Thanks for the suggestion! Tried both NICs in my Win10 gaming PC. No luck. Same error "This Update Package is not compatible with your system configuration."

 

[LodeRunner]

With the Windows DUP package, you can open it with 7-zip and pull the firmware and the write utility out that way and may be able to run them manually; skipping whatever manifest check is in the DUP.

I think you can do the same with the Linux packages? The firmware is in the payload directory of the BIN. Not sure which file is the utility to flash it though.

Here's a possibility: Broadcom NetXtreme-E Network Device Firmware and Configuration 21.6.2 | Driver Details | Dell US
57414 PCIe addin card is listed under "Applies to"

I just manually specified I was looking for support for a R740, then went to drivers and search Broadcom. Should narrow the field a bit for you.

 

[NPS]

Thanks a lot!
Motivated by your words I gave it another try. Coupled with a thought I had myself, I finally managed to flash the current Dell FW. While fiddling with the update packages, I remembered, that I never checked the drivers on Windows10. No Win10 can not find drivers for this card on its own... No wonder the FW DUP said "incompatible" (but strange that it did not find the card on Linux). The Dell driver pack for Windows Server did not work first try (unsupported OS), but the DUP has an "extract" button and in the payload folder is a Broadcom driver installer executable. After installing the driver, the FW DUP worked right away. Thanks again!

So the FW part is done. As one can imagine the new FW works no wonders. Linux performance seems to be about the same. There seem to be some barriers in single-thread/sync NFSoRDMA performance. Maybe in general, but at least with these NICs in my setup. Small block sizes are faster using RDMA/RoCE vs. TCP, but strange enough, at large blocksizes, TCP is sometimes faster than RDMA/RoCE. I observed the same to an even larger extend with a pair of directly connected Mellanox MCX311A-XCAT and MCX312A-XCBT NICs. (FW updates where easy with these ) Interestingly the pure RDMA message rate (ib_send_bw) is higher on small package sizes with the 10GbE Mellanox NICs than with 25GbE Broadcom cards.

 

[TrumanHW]

NFSoRDMA -- Single-threaded/sync performance barriers ... with these NICs :

• small blocks = faster via RDMA/RoCE vs. TCP
• large blocks = TCP is sometimes faster than RDMA/RoCE

And...to an even larger extent via direct-connected Mellanox:

(MCX311A-XCAT -- to -- MCX312A-XCBT)
FW updates were easy with mellanox, also​

10GbE Mellanox pure RDMA message rate with small package sizes...
...(ib_send_bw) was faster than 25GbE Broadcom.

Dude, that's great info. Networking is a world of knowledge that takes being immersed in it like a language to understand at the level of the experts here, so I know I'm as likely to gain that level of knowledge as I am to learn languages I like ... but the information you pointed out was really interesting, even to an NSL 'Network as a Second Language' guy like myself.

Likewise, I enjoyed that economics-point you addressed ... that you could skip buying both an expensive and LOUD SFP28 switch with a 4-port SFP28 card!

For me -- the issue would be my NSL proficiency ... and I can only imagine how much more difficult it'd be to administrate Peer-to-Peer setups, no GUI (ever) ... and -- I think I got a good deal on a Dell Networking S5048F-ON 48 x 25GbE SFP28 for around $300 which needs PSU + Fans, which 1 of the PSU ports were bad on ... that said, since I always have employees who do SMD level repair (and I can do light SMD repair) ... I fully expect to see the obvious point that failed (it's probably an in-rush limiting diode or something -- whatever it is, it'll probably be burnt obviously enough to see without even diagnosing -- and will no doubt be a standard package).

After that it's project 2 -- replacing fans to see if I can make it quiet enough to tolerate ...or if not, make a few bucks off it as I continue replacing fans on my other SFP+ switches to find ONE that's not bad. (I have similar plans for the other enterprise gear) ...

Anyway, again, thank you so much. It was a treat reading through all the knowledge and your ideas for using it.

Likewise -- you're a very good communicator; I'm sure some company's happy to have you on their team.

If you have any suggestions between either a

• Mellanox Connectx-4
• Broadcom BCM57414. (which STH actually reviewed and found it's avg draw was 5W ...)
• QLogic FastLinQ 41262
• Intel XXV710 .....
• Dell (C63DV) Intel [X520 (2x) 10GbE (SFP+)] + [I350 (2x) 1GbE] (or I'd only have the iDRAC 8 port for Gbic backup)
• Dell (6VDPG) Intel [X710 (2x) 10GbE (SFP+)] + [I350 (2x) 1GbE] (or I'd only have the iDRAC 8 port for Gbic backup)
  (all of which make daughter cards for the Dell R730xd) ... please, LMK.

Thus far, my 'take away is' ...
I'd need all Mellanox for IB advantages to be substantial.
And, Broadcom is perhaps slow & difficult to flash, but efficient ..?