Flash OEM (IBM) Mellanox ConnectX-3 EN with stock firmware?

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,823
113
29
fohdeesha.com
hmmm, the only thing I can think of at this point - are you sure it's a CX3 Pro? This is the exact same behavior I got when I flashed a non-pro with pro firmware - firmware query in recovery mode would show the flash is perfectly fine and had a firmware image in it, but booting without the jumper, the code just couldn't boot the card.

or maybe if it is a cx3pro card, are you sure you're flashing the right firmware? Showing proper flash and firmware loaded but not initializing the card all points to incorrect FW to me but I could be wrong

Is that IBM-00RX859.ini ini in the zip you posted dumped from BEFORE you flashed the card? Because that definitely calls it out as

Code:
Name = MCX312B-XCC_Ax
Description = ConnectX-3 Pro EN network interface card; 10GigE; dual-port SFP+; PCIe3.0 x8 8GT/s; RoHS R6
PRS_name    = cx3pro_MCX312B_2mb.prs
which is definitely what you're trying to flash, so I dunno. It is also IDENTICAL to the MCX312B-XCC_Ax.ini that gets used to build the official image you're flashing, so I dunno why it wouldn't work. I guess the card could be bad, it does happen
 
  • Like
Reactions: arglebargle

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Yeah, the .ini and .bin in the tarball are dumps taken before I touched the flash at all. I wasn't expecting to need to dump the entire flash content or I'd have done that before touching the card.

Here's IBM's current Mellanox reference: RG_IBM.pdf -- I'm fairly certain this is a CX3 Pro.

Welp, I figured for $55 I could roll the dice on this one. I wasn't expecting IBM's firmware to be dug in like a tick.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
The only thing I can think of now is finding someone with an identical CX3 Pro card and having them dump their flash, maybe there's something in one of the other segments that I'm missing.

Code:
root@ted:~# !483
flint -d /dev/mst/mt504_pciconf0 verify

    FS2 failsafe image. Start address: 0x80000. Chunk size 0x80000:

    NOTE: The addresses below are contiguous logical addresses. Physical addresses on
          flash may be different, based on the image start address and chunk size

    /0x00000038-0x0000065b (0x000624)/ (BOOT2) - OK
    /0x0000065c-0x0000297f (0x002324)/ (BOOT2) - OK
    /0x00002980-0x00003923 (0x000fa4)/ (Configuration) - OK
    /0x00003924-0x0001d14b (0x019828)/ (ROM) - OK
    /0x0001d14c-0x0001d18f (0x000044)/ (GUID) - OK
    /0x0001d190-0x0001d317 (0x000188)/ (Image Info) - OK
    /0x0001d318-0x0002a6af (0x00d398)/ (DDR) - OK
    /0x0002a6b0-0x0002b713 (0x001064)/ (DDR) - OK
    /0x0002b714-0x0002bae3 (0x0003d0)/ (DDR) - OK
    /0x0002bae4-0x00069eb3 (0x03e3d0)/ (DDR) - OK
    /0x00069eb4-0x0006ed37 (0x004e84)/ (DDR) - OK
    /0x0006ed38-0x0007332b (0x0045f4)/ (DDR) - OK
    /0x0007332c-0x00073e23 (0x000af8)/ (DDR) - OK
    /0x00073e24-0x000a3fc7 (0x0301a4)/ (DDR) - OK
    /0x000a3fc8-0x000a7b73 (0x003bac)/ (DDR) - OK
    /0x000a7b74-0x000bd067 (0x0154f4)/ (DDR) - OK
    /0x000bd068-0x000bd16f (0x000108)/ (DDR) - OK
    /0x000bd170-0x000c21f3 (0x005084)/ (DDR) - OK
    /0x000c21f4-0x000c39ef (0x0017fc)/ (Configuration) - OK
    /0x000c39f0-0x000c3a63 (0x000074)/ (Jump addresses) - OK
    /0x000c3a64-0x000c405b (0x0005f8)/ (FW Configuration) - OK
    /0x00000000-0x000c405b (0x0c405c)/ (Full Image) - OK

-I- FW image verification succeeded. Image is bootable.
 

Freebsd1976

Active Member
Feb 23, 2018
228
33
28
Thanks! This worked for me as well against the HP branded cards from a recentish thread (HP 649281-B21 ConnectX-3 VPI MCX354A-FCBT $35 + $8 shipping OBO ebay). I did not need to short anything.
my post here - https://forums.servethehome.com/ind...x-3-to-arista-7050-no-link.18369/#post-178015

has the source packages (with mlx files, you don't need the ini files as those are generated by you from your card during the process) for both normal and pro cards, and also instructions for compiling new fw to flash to your specific card (pretty much the same instructions as izx's)

Regarding compiling new firmware using your cards specific INI, versus just flashing a pre-made BIN from mellanox, it depends on the card. If the card has been "customized" in any way other than just different firmware, you need to compile a new firmware file using the process linked above. Cards like mezzanine cards, custom made cards for certain vendors, etc. If you use a stock BIN on these cards you'll have weird problems like LEDs not working or the second port not working

This compiling process can also be used to turn the vanilla stock 10gbE CX3 cards into 40gbE CX3 cards (compile an image for it using the INI file of the 40gbE card, eg the FCBT ini file, then flash the resulting image to the "10gbE" (QCBT) card. This obviously only works for the CX3 cards that support 40gb infiniband out of the box, but have restricted ethernet to 10gbE. (the -QCBT models)
i crossflash three Mellanox MCX354A-qcbt (not Oem version)to fcbt using self complied firmware,two card is ok both port work in eth mode .but one card only has one port work (in Windows it recognized two port),the second port always show cable disconnect.
I carefully look through the three crad ,they should seems.

Is this problem due to cross flash or bad hardware?

Now I can try following two way ,which way is better?
1 using Mellanox originall fcbt firmware to try
2 using Mellanox original qcbt firmware to try
Btw I back up the original ini file but now I delete it
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
i crossflash three Mellanox MCX354A-qcbt (not Oem version)to fcbt using self complied firmware,two card is ok both port work in eth mode .but one card only has one port work (in Windows it recognized two port),the second port always show cable disconnect.
I carefully look through the three crad ,they should seems.

Is this problem due to cross flash or bad hardware?

Now I can try following two way ,which way is better?
1 using Mellanox originall fcbt firmware to try
2 using Mellanox original qcbt firmware to try
Btw I back up the original ini file but now I delete it
Try the Mellanox QCBT image and see if both ports work, if both ports work then you know that the card functions.

I'd then try the FCBT image and verify that both ports still work, then you can try and find the problem with your custom image.

One thing, check the hardware revision of all three cards. IIRC the Rev A1 cards can't run at the higher clock rate that is required for VPI functionality -- I suspect that if 2 of 3 cards work for you and all are undamaged then the last may be a Rev A1.
 

Freebsd1976

Active Member
Feb 23, 2018
228
33
28
Try the Mellanox QCBT image and see if both ports work, if both ports work then you know that the card functions.

I'd then try the FCBT image and verify that both ports still work, then you can try and find the problem with your custom image.

One thing, check the hardware revision of all three cards. IIRC the Rev A1 cards can't run at the higher clock rate that is required for VPI functionality -- I suspect that if 2 of 3 cards work for you and all are undamaged then the last may be a Rev A1.
Checked the three card, all are rev 01 on the label ,product date different but all producted in 2014, I will try further
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Flash the QCBT firmware and verify that both ports work, then flash the FCBT firmware and test both ports again. If only one port works with QCBT the card is damaged. If only one port works with FCBT then I'm at a loss.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
I made a custom firmware file using the MCX314A-BCBT ini file, then flashed the exact same image to a variety of IBM, EMC, and stock VPI cards, all worked perfectly. Then ran mlxup after a reboot, it detected them as MCX314A-BCBT and updated them all to the latest Mellanox firmware as normal. They all work afterwards too at 40G on both ports. Happy days.

I haven't yet tried crossflashing them to a Pro, but I have a genuine Pro arriving in the next week or to compare, might then try and upgrade a non Pro and see what happens. Will report back either way.
A couple of members tried forced conversion of regular CX-3 cards to Pro and didn't have any success. The cards don't initialize on boot, you have to use the flash not present jumper to boot them in recovery mode to see them at all.
 

Freebsd1976

Active Member
Feb 23, 2018
228
33
28
Flash the QCBT firmware and verify that both ports work, then flash the FCBT firmware and test both ports again. If only one port works with QCBT the card is damaged. If only one port works with FCBT then I'm at a loss.
test on windows further today,using one mellanox dac cable connect two port , flash to qcbt and change to eth mode both port work, but what strange are on ib or vpi mode not work (opensm started but always said port down).
flash to mellanox fcbt or compiled firwmware only one port work

seems card is ok on qcbt firmware but don’t know why it not work on ib or vpi mode, maybe casused by one dac connect two port?
i am total confusing , the dac i used is mc2207130-001 .
now i have two questions:
1 why it can work under qcbt firmware on eth but not on ib or vpi mode
2 why this card only one port work on fcbt firmware ,the other two card have same revision both port work
 
Last edited:

Freebsd1976

Active Member
Feb 23, 2018
228
33
28
more test , use two card and same dac to test .
the bad card only work on eth mode both port (qcbt firmware) on 10gb, on ib mode only one port work (qcbt firmware) on 40gb.
the bad card(fcbt firmware) work only one port on eth or ib mode on 56gb.

maybe bad lucky , this is a bad card? one port only work on eth mode?
 

TeleFragger

Member
Oct 26, 2016
147
19
18
47
I got an OLD hp branded connect-x card to flash. HP only went to 2.8 for firmware and I force flashed 2.9.1000 and got it working in esxi 6.7, freenas 11.1, and windows 10. this card is the old cx4 port (I think)… and would love to know what switch works with these... but going to try and do pier to pier for now...
 

Stereodude

Active Member
Feb 21, 2016
412
65
28
USA
Just as a heads up the Mellanox WinMFT tools (v 4.11.0) for Windows don't come with mic.exe anymore so you'll have to download an older version. I downloaded 4.3.0 for the .bin creation step (mlxburn -fw fw-ConnectX3-rel.mlx -conf MCX312A-XCB_A2-A6.ini -wrimage mlnx_firmware.bin) because that's what I had used back in 2016. However, the old 4.3.0 version doesn't work under Windows 10, for all the other steps so I re-installed 4.11.0. I ended up using 4.3.0 for only the .bin creation step.

I used mlxup.exe after I crossflashed from the OEM version to stock in order to get the latest stock FW on the CX-3 EN since the tarballs are no longer seem to be available and I only had an older tarball.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,823
113
29
fohdeesha.com
Just as a heads up the Mellanox WinMFT tools (v 4.11.0) for Windows don't come with mic.exe anymore so you'll have to download an older version. I downloaded 4.3.0 for the .bin creation step (mlxburn -fw fw-ConnectX3-rel.mlx -conf MCX312A-XCB_A2-A6.ini -wrimage mlnx_firmware.bin) because that's what I had used back in 2016. However, the old 4.3.0 version doesn't work under Windows 10, for all the other steps so I re-installed 4.11.0. I ended up using 4.3.0 for only the .bin creation step.

I used mlxup.exe after I crossflashed from the OEM version to stock in order to get the latest stock FW on the CX-3 EN since the tarballs are no longer seem to be available and I only had an older tarball.

for regular cx3 cards (eg not niche mezzanine versions of the cards) you don't need to compile your own image from the old source tarballs - indeed you don't WANT to because they are quite old versions of the driver. You can directly flash the latest firmware files right off their site (exact firmware files and commands may differ from windows to linux and differing cx3 models): https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015
 

Stereodude

Active Member
Feb 21, 2016
412
65
28
USA
for regular cx3 cards (eg not niche mezzanine versions of the cards) you don't need to compile your own image from the old source tarballs - indeed you don't WANT to because they are quite old versions of the driver. You can directly flash the latest firmware files right off their site (exact firmware files and commands may differ from windows to linux and differing cx3 models): https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015
Good to know. I'll keep that in mind should I buy more of them that need crossflashing.
 

Blue)(Fusion

Member
Mar 1, 2017
86
14
8
I just picked up 4 of these IBM1080110023 dual port CX3 cards. They are all running firmware 2.33.5000. I see IBM has the latest firmware at 2.40.5048 on their website. I downloaded the binary file, but it appears to be a linux program, not an image I can flash to the card using mstflint.

I have only mstflint to upload the latest firmware. No mlxburn package available (is it even avail for linux?) What exactly do I do to get the latest firmware on my cards?
 

edrozenberg

New Member
Nov 26, 2017
4
0
1
120
If it's a something.bin file, that's not a Linux binary. That's the firmware image. There's lots of info in this thread and others on SVTH on using flint (part of MFT tools) or mlxup to update the firmware. If you're burning a newer version of the existing firmware, it's straightforward with either of these tools.

http://www.mellanox.com/page/firmware_HCA_FW_update

If you're trying to burn firmware to change from say IBM or Dell OEM to Mellanox, it requires some additional magic such as the '--allow-psid-change' mstflint flag (care required to make sure to use the correct firmware) to force burning of an image it would normally not allow you to flash.

...I downloaded the binary file, but it appears to be a linux program, not an image I can flash to the card using mstflint.

I have only mstflint to upload the latest firmware. No mlxburn package available (is it even avail for linux?) What exactly do I do to get the latest firmware on my cards?
 

Blue)(Fusion

Member
Mar 1, 2017
86
14
8
I wish it were the binary firmware file, but according to the IBM instructions where I downloaded it, it is to be executed in some proprietary software to perform the update. mstflint does not recognize it. Peaking a little at the .bin file, it is an executable, not a firmware image.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Mellanox OFED is available on their website and includes all of the standard tools for manipulating the cards. I've installed Ubuntu to a USB drive before to update firmware, you might try something like that if IBM provides a linux binary. Worst case scenario if you get a linux system up and running you can just use OFED and cross flash the cards to standard firmware and go from there.