Crossflash Oracle CX556A to Mellanox OEM?

[Stephan]

The point is to have modules loadable cleanly, for any kernel you might run, because they are properly compiled just for it and are not precompiled for some older Centos or Debian kernel, which you have to exactly hit. As can be seen in your post #16, you seem to have this problem at least. Also all MFT tools will be available from /usr/bin without resorting to /opt. To build and install, just do

cd (change into your user home)
mkdir tmp
cd tmp
wget https://seitics.de/files/mft/PKGBUILD
makepkg -Ccfi
(-i will also install, needs root password)

Then do
mst start (this will load the kernel modules)
mst status

I cross-flashed four CX3 from EMC to Mellanox recently using the package on Arch. Not saying using a "supported" Linux version is any better or worse, provided it works. If you can't get it to work, I'm just giving you another option.

 

[jpmomo]

Thanks for the info and detailed steps. I will give it a try and update this thread so that others might benefit. You gotta luv STH!

 

[jpmomo]

I think I may be running into issues with centos 8 vs centos 8 stream but not sure.

I tried to follow your steps above but got stuck at the " makepkg -Ccfi " my system couldn't find the makepkg command.

I then tried to install makepkg and also tried to install pacman but neither were found. It is probably a basic linux command/concept that I am missing.
Thanks again for any tips,
jp

 

[Stephan]

$ pacman -Qo /usr/bin/makepkg
/usr/bin/makepkg is owned by pacman 6.0.1-5

You need to install Arch Linux for real, not dabble with Centos. Maybe you have an old SSD laying around.

 

[jpmomo]

Will install another distro. In the meantime, is there much difference between your method and running the install.sh from mellanox? It looks like the install script from mellanox will also use the specific kernel modules.

 

[Stephan]

I never looked at Mellanox' install.sh, because it is of no use on Arch. I saw they did not even bother to natively port their package from RPM to DEB, just used "alien" for Debian. This can go well and not so well.

So I wrote my own package and made sure it integrates cleanly and works. I also made sure to dynamically build the two kernel modules required, for whatever kernels that are currently installed. If "mst start" does not load kernel modules, you have that kernel module problem.

 

[Hushhh]

Can this method be used on HPE 547FLR? Sorry for replying to a post from two years ago

 

[Stephan]

Can this method be used on HPE 547FLR? Sorry for replying to a post from two years ago

Card can do InfiniBand and Ethernet, no need to cross-flash? Just use HPE own updater, e.g. SPP for ProLiant. Or did you install this FlexibleLOM card via PCIe adapter in another machine?

 

[Hushhh]

Thank you for your reply. It works normally, but I would like to give it a try. Perhaps this card can enable 100G and PCIe Gen4

 

[darthray]

Replying to this old thread just to say thank you for all the great info and to add some info on the HPE card I have that may be useful to someone else.

I have a "HPE InfiniBand EDR 100Gb 1-port 841QSFP28 Adapter" (part number 872725-B21). Following the instructions here (thank you @chicken-of-the-cave!), I tried flashing the Mellanox MCX555 firmware on it to enable 100Gb Ethernet, but it didn't work. It flashed to completion, but upon reboot the card would not show up anywhere in the system, even in the BIOS (it shows up there for me when it works). I'm guessing it didn't work because the HPE card expects a "secure" firmware but the one from Mellanox is not (not sure, this is honestly a guess based on what I read elsewhere). Perhaps there's a way to disabled that, but I haven't looked into it yet.

Having said that, I was able to cross-flash it to the 2-port HPE EDR/Ethernet version (the "HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter", part number 872726-B21) and it worked! That makes the OS to show 2 ports, with the second one obviously not working since it's not physically present, but the first port seems to work just fine. I ended up setting the LINK_TYPE on port 1 to ETH(1) and left port 2 as IB(2) so it wouldn't show as an ethernet adapter to avoid confusion.

root@pve:~# lspci | grep Mellanox
86:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
86:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]

root@pve03:~# flint -d 86:00.0 q
Image type:            FS4
FW Version:            16.35.4030
FW Release Date:       27.6.2024
Product Version:       16.35.4030
Rom Info:              type=UEFI version=14.29.15 cpu=AMD64
                       type=PXE version=3.6.902 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             xxxxxxxxxxxxxxx        4
Base MAC:              xxxxxxxxxxx            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  HPE0000000009
Security Attributes:   secure-fw
root@pve03:~#

The latest 872726-B21 firmware (16.35.4030 as of today) is here.

I'll see if I can find a way to make the card not require a secure firmware to see if I can get the Mellanox firmware to work.

Hope this helps.

 

[dliao]

Replying to this old thread just to say thank you for all the great info and to add some info on the HPE card I have that may be useful to someone else.

I have a "HPE InfiniBand EDR 100Gb 1-port 841QSFP28 Adapter" (part number 872725-B21). Following the instructions here (thank you @chicken-of-the-cave!), I tried flashing the Mellanox MCX555 firmware on it to enable 100Gb Ethernet, but it didn't work. It flashed to completion, but upon reboot the card would not show up anywhere in the system, even in the BIOS (it shows up there for me when it works). I'm guessing it didn't work because the HPE card expects a "secure" firmware but the one from Mellanox is not (not sure, this is honestly a guess based on what I read elsewhere). Perhaps there's a way to disabled that, but I haven't looked into it yet.

Having said that, I was able to cross-flash it to the 2-port HPE EDR/Ethernet version (the "HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter", part number 872726-B21) and it worked! That makes the OS to show 2 ports, with the second one obviously not working since it's not physically present, but the first port seems to work just fine. I ended up setting the LINK_TYPE on port 1 to ETH(1) and left port 2 as IB(2) so it wouldn't show as an ethernet adapter to avoid confusion.

root@pve:~# lspci | grep Mellanox
86:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
86:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]

root@pve03:~# flint -d 86:00.0 q
Image type:            FS4
FW Version:            16.35.4030
FW Release Date:       27.6.2024
Product Version:       16.35.4030
Rom Info:              type=UEFI version=14.29.15 cpu=AMD64
                       type=PXE version=3.6.902 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             xxxxxxxxxxxxxxx        4
Base MAC:              xxxxxxxxxxx            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  HPE0000000009
Security Attributes:   secure-fw
root@pve03:~#

The latest 872726-B21 firmware (16.35.4030 as of today) is here.

I'll see if I can find a way to make the card not require a secure firmware to see if I can get the Mellanox firmware to work.

Hope this helps.

Hi ，I am interested in this card and want to buy it recently. Can you help answer my 2 questions about this card please?
1. Does this card only support IB model in OEM firmware ? (I see you said you want to switch firmware to enable it)
2. After cross-flash to 872726-B21 firmware, the working port can switch model ? (IB/ETH)

 

[darthray]

Hi ，I am interested in this card and want to buy it recently. Can you help answer my 2 questions about this card please?
1. Does this card only support IB model in OEM firmware ? (I see you said you want to switch firmware to enable it)

As far as I know, yes, 872725-B21 is an IB-only card.

2. After cross-flash to 872726-B21 firmware, the working port can switch model ? (IB/ETH)

That's correct. I have port 1 as ETH and port 2 (not present) as IB.