Have you been able to flash ConnectX-3 VPI controllers with ConnectX-3 Pro firmware?

[AveryFreeman]

Hey,

So this is totally anecdotal, but I thought I saw someone who said they were able to flash their ConnectX-3 VPI with ConnectX-3 Pro firmware

I've seen other threads where people said they were NOT able to do this

Looking over the PDFs for the three models, they don't really look any different as far as their feature sets. Can someone be a tie breaker and let me know whether or not they've gotten this to work?

I was able to flash some QCBTs (10/40) with FCBT (40/56) firmware easy enough, so if someone says they were able to go all the way to the FCCF (RoCE v2) firmware, I might have to try it...

The main advantage would be in ESXi the "regular" CX3s can't do RoCE v2 in ESXi, while the "pro" ones can, apparently...

The models I'm talking about are:

VPI:
MCX353A-QCBT (single QDR) / 354A -QCBT(dual QDR) - the 10GbE/40Gb IB models
MCX353A-FCBT (single QDR) / 354A -FCBT(dual QDR) - the 40GbE/56Gb IB models

Pro:
MT27524A1-FCCF-FV (single QDR) / MT27528A1-FCCF-FV (dual QDR) - these are also 40GbE/56Gb IB like FCBT afaik but support RoCE v2 and may or may not have a different ASIC (? Can someone confirm?)

 

[i386]

Have you seen this post from @fohdeesha:https://forums.servethehome.com/ind...rnet-dual-port-qsfp-adapter.20525/post-198961 ?

 

[NablaSquaredG]

I once had a CX3 from @Rand__ that was supposedly flashed to CX3-Pro software.

I don't know whether it actually was the Pro firmware or whether the Pro features worked, but I do know that with this firmware, even though it was the right variant, 56G did not work (40G did)

 

[Rand__]

There is a reference somewhere where a user said taht he modded the open source version of flint or mlxburn to work around the checks on what hw the pro fw could be burned.

I never found if the CX3 really was running the pro features (RoCE v2) since the downgrade to 40G that @NablaSquaredG mentioned was annoying me so i didnt persue it (since i didnt have the other infra set up for RoCE)

 

[i386]

There is a reference somewhere where a user said taht he modded the open source version of flint or mlxburn to work around the checks on what hw the pro fw could be burned.

I used flint with the "bb" command (instead of burn), but it bricked my cx-3 and I had to use the recovery mode to burn the old firmware

 

[Rand__]

well whatever he did back then it worked, but he has not been on for years

 

[necr]

https://www.mellanox.com/sites/default/files/doc-2020/pb-connectx-3-pro-silicon-en.pdf has different silicon compared to non-Pro cards. Seems it's only in HW revision C onwards (like MCX353A-FCCT), according to https://www.mellanox.com/pdf/firmware/ConnectX3Pro-FW-2_42_5000-release_notes.pdf. You can check your revision on the board label or in the VPD.

 

[AveryFreeman]

I once had a CX3 from @Rand__ that was supposedly flashed to CX3-Pro software.

I don't know whether it actually was the Pro firmware or whether the Pro features worked, but I do know that with this firmware, even though it was the right variant, 56G did not work (40G did)

56Gb is IB only, the same variants that do 56Gb IB only do 40Gb ethernet. Both the pro and the "regular" vpi. And the ethernet-only MCX313-314a models, too. So, that's normal.

Almost nobody uses IB because not much software supports it these days, on account of it being a PITA to set up (at least, that's my understanding of it, stop me if you've heard different...)

 

[AveryFreeman]

Have you seen this post from @fohdeesha:https://forums.servethehome.com/ind...rnet-dual-port-qsfp-adapter.20525/post-198961 ?

Hi, yeah I was just looking at that the other day, but before that I had a reply (that now I can't find) where someone said they'd flashed their QCBTs with PRO firmware, so I just wanted to put some feelers out there in case I wasn't dreaming.

RoCEv2 would be pretty awesome, but I'm thinking I might be able to get it with pvrdma (host-only) even without HCAs that support it (?) I'm going to put up another post asking if someone can split these definition hairs, there's a lot of lit out there that's confusing for people that aren't so experienced (e.g. I had to look up "HCA" because all the tech blurbs I was reading didn't bother to define it, so I thought CX3s were "NICs"...)

 

[AveryFreeman]

There is a reference somewhere where a user said taht he modded the open source version of flint or mlxburn to work around the checks on what hw the pro fw could be burned.

I never found if the CX3 really was running the pro features (RoCE v2) since the downgrade to 40G that @NablaSquaredG mentioned was annoying me so i didnt persue it (since i didnt have the other infra set up for RoCE)

Downgrade? You know the 40/56 numbers are Ethernet/IB, right? Are you saying it would only do 40Gb IB?

If we're just talking IB speeds, those speeds also correlate with QDR(40) vs FDR(56) speeds, which are capabilities of different cables (same connectors, dif fabric) ... Could you have had QDR instead of FDR cables, if you were trying to do 56Gb IB (?) (sorry, I figure you know all this stuff, just probing for answers...)

Do you think the mods were done with a hex dump program? My guess is it'd be easier to remove checks from the firmware if that's where they're at, the program I'd have no idea. If you know anyone who can do either, though, I have like 4 HP-branded HBAs that are stuck at v1.98 because HP never goes past 2-3 years of upgrades, when their orig. vendor (Microsemi) is up to v4 on the same cards now....

I need to go hit up that site I found when I was looking for Thinkpad BIOS whitelists for WIFI cards back in the day...

 

[klui]

56Gb is IB only, the same variants that do 56Gb IB only do 40Gb ethernet.

My CX3s (Mellanox and HP-branded) can link at 56G in Ethernet mode.

 

[AveryFreeman]

My CX3s (Mellanox and HP-branded) can link at 56G in Ethernet mode.

No shit? How'd you manage that? I'm not above admitting when I'm wrong.

 

[klui]

Nothing special. Just connected to a Mellanox switch and forced the port to ethernet and 56G.

 

[Rand__]

Downgrade? You know the 40/56 numbers are Ethernet/IB, right? Are you saying it would only do 40Gb IB?

No, as @klui mentioned the CX3 can run 40 and 56 in both ETH and IB. A proprietary MLX upgrade from 10 bit (QDR) to 14 bit (FDR).

If we're just talking IB speeds, those speeds also correlate with QDR(40) vs FDR(56) speeds, which are capabilities of different cables (same connectors, dif fabric) ... Could you have had QDR instead of FDR cables, if you were trying to do 56Gb IB (?) (sorry, I figure you know all this stuff, just probing for answers...)

O/c all parts need to be compatible to be able to run FDR instead of QDR, but no, it was not a cable/transceiver/switch incompatibility. I ran 56G Ethernet before flashing the cards to Pro, afterwards they only linked at 40.

Do you think the mods were done with a hex dump program? My guess is it'd be easier to remove checks from the firmware if that's where they're at, the program I'd have no idea.

No, as mentioned it was a patched version of one of the MLX tools from github(?) where the HW ID check was patched out before compiling it. I searched my archives but have not been able to find it.

If you know anyone who can do either, though, I have like 4 HP-branded HBAs that are stuck at v1.98 because HP never goes past 2-3 years of upgrades, when their orig. vendor (Microsemi) is up to v4 on the same cards now....

Usually CX3 cards are cross-flashable to original Mellanox (or any other vendor) with the appropriate flag, so not sure what your issue with these is?

 

[AveryFreeman]

Nothing special. Just connected to a Mellanox switch and forced the port to ethernet and 56G.

Wait ... is that like "regular" ethernet, or RoCE? (does it make a difference?) And did you have to pay special attention to getting FDR-capable cables, or are they QDR? I'm just trying to figure out how much I thought I knew about this subject will turn out to be bullshit.

Do you think it depends on having a switch, or can you force the port to 56GbE without one? For me, in vCenter the fastest I can set my mnlx4_en driver-using cards to is 40Gb, where one sets port speed, duplex, autoneg, etc.

Thanks

 

[AveryFreeman]

No, as @klui mentioned the CX3 can run 40 and 56 in both ETH and IB. A proprietary MLX upgrade from 10 bit (QDR) to 14 bit (FDR).


O/c all parts need to be compatible to be able to run FDR instead of QDR, but no, it was not a cable/transceiver/switch incompatibility. I ran 56G Ethernet before flashing the cards to Pro, afterwards they only linked at 40.


No, as mentioned it was a patched version of one of the MLX tools from github(?) where the HW ID check was patched out before compiling it. I searched my archives but have not been able to find it.


Usually CX3 cards are cross-flashable to original Mellanox (or any other vendor) with the appropriate flag, so not sure what your issue with these is?

Cool, thanks for clearing some of that up for me, still getting my legs for wading through the sea of jargon. It's tidal. yarrr.

This doesn't look at all promising, does it? GitHub - BeTeP-STH/mft-scripts: Mellanox firmware files and MFT related scripts

... or maybe in the list of forks? Forks · Mellanox/mstflint

Long shot, but figured I'd send 'em your way just in case ... (maybe they forked it and you can recognize their username)

 

[klui]

Wait ... is that like "regular" ethernet, or RoCE? (does it make a difference?) And did you have to pay special attention to getting FDR-capable cables, or are they QDR? I'm just trying to figure out how much I thought I knew about this subject will turn out to be bullshit.

Do you think it depends on having a switch, or can you force the port to 56GbE without one? For me, in vCenter the fastest I can set my mnlx4_en driver-using cards to is 40Gb, where one sets port speed, duplex, autoneg, etc.

Thanks

That was a quick test using Windows. No other setting changes required. Not using RoCE but no doubt will work. I am not using VMware for RoCE at this time as my fiber buildout is not complete. Honestly I am taking it a step at a time. Get regular ethernet working then think about what stacks to use to get more performance. @Rand__ is correct. One needs to make sure all components work. CX3s are on the outs with newer VMware versions and I am migrating away from that to Proxmox.

 

[fohdeesha]

the pro and non-pro CX3 cards are slightly different silicon (pro is codenamed bental), you can see some of the differences if you download the firmware sources and poke through the ini and mlx build files, the silicon ID and other idents are different. they can't be (successfully) crossflashed. I'd be wary of anyone who claimed to modify mellanox flashing tools and crossflashed it and had a working card - for starters, you don't need to modify the tools. Mellanox's low level tools like Flint has a built in argument "burn block" that bypasses all checks (it won't even check if the given PCIe ID is a NIC) and dumps the specified firmware file straight to the card. I did this, read the flash back and compared it to a real pro to verify it flashed correctly, and it did (identically) - it will not initialize the ASIC and stays in a fault state. feel free to try yourself but it *will* brick your card:

flint -d /dev/mst/mt4099_pci_cr0 -i 3xpro.bin bb
flint -d /dev/mst/mt4099_pci_cr0 -guid 0x24be05ffff8b97a0 -mac 0x24be058b97a1 sg

 

[Rand__]

I can just say that I got a modified firmware at some point with the explanation that the creator had modified some checks - cant remember if it was the burning tool or maybe the firmware itself now that you mention it.

It flashed fine, worked fine (visibly all except the 40G link issue) but didn't convince me.
I flashed most of them back to CX3 except one apparently that I later sold to @NablaSquaredG.

I just iterate to say its possible.
Is it advisable? probably not. Would I recommend? No. Do I think it creates a fully working Pro? No

 

[nasbdh9]

No longer recommend any cx3/cx3pro products, because PFC and other roce-related functions are flawed
Contemporaneous products, using sx2's sx6xxx and sn17xx switches will also have such a situation, you will encounter the problem that the traffic cannot be fully loaded, or even the data is interrupted.
cx4lx is cheap enough, it’s time to put down these EOL products