Flash OEM (IBM) Mellanox ConnectX-3 EN with stock firmware?

ncpjim

New Member
Sep 25, 2017
3
0
1
50
I have successfully flashed all my cards. now i have on issue. i didn't notice that the second port was not there after a reboot. any insight on this?
 

edrozenberg

New Member
Nov 26, 2017
4
0
1
120
I have successfully flashed all my cards. now i have on issue. i didn't notice that the second port was not there after a reboot. any insight on this?
Make sure you've rebooted the machine after flashing or changing firmware.

It depends on what you mean by port is not there. Maybe it's not yet configured with IPv4 or v6 settings, or maybe there is no cable plugged into it, or maybe the port was automatically set to IB (inifiniband) mode rather than ethernet because it wasn't connected to ethernet when the card booted.

On Linux you can run:

$ mlxconfig -d /dev/mst/mt<yourdevice> query

to view card info.

If you don't have any /dev/mst/ items you'll need to first start mst with:
$ mst start

On Windows there should be a similar mlxconfig.exe command to view device info.

In the output from mlxconfig you will hopefully see something like:

LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)

If you don't have a P2 item, then indeed there is only 1 port recognized by the firmware.
If that's the case, you may have flashed a firmware for a different card model with only 1 port.

With the usual disclaimers and warnings about flashing firmware :), you might want
to try identifying the correct firmware for your card with 2 ports and flashing that.
 

segfault

New Member
Apr 5, 2016
26
11
3
39
hasno.info
Thanks! This worked for me as well against the HP branded cards from a recentish thread (HP 649281-B21 ConnectX-3 VPI MCX354A-FCBT $35 + $8 shipping OBO ebay). I did not need to short anything.

Here is what I did after I could not find the custom firmware files. This worked for a Dell branded card. Note the below instructions do not include the steps of using a shorting wire to short the holes on the card. See the info in the posts above for those details. Just remember to remove the short after you update the fw.

Step 1 - download and install
MLNX_firmware_tool_WinMFT_x64_4_7_0_42.exe

Step 2 - download:
fw-ConnectX3-rel-2_40_7000-MCX312A-XCB_A2-A6-FlexBoot-3.4.746.bin.zip
Extract .bin file and rename in to fw.bin
copy it to:
C:\Program Files\Mellanox\WinMFT>

Step 3 - open command prompt as administrator
cd to directory:
C:\Program Files\Mellanox\WinMFT>

Run:
0) mst status
1) mlxfwmanager
2) flint -d mt26448_pci_cr0 -i fw.bin burn -allow_psid_change

where "mt26448_pci_cr0" is the name output from commands 0 or 1

The -allow_psid_change switch is dangerous and I make no guarantees that this will not brick your card. YMMV!!! But it worked perfectly for my Dell branded card.

Good luck.
 

alltheasimov

Member
Feb 17, 2018
59
11
8
29
Is the mlx file and ini necessary, or can just a bin work? It looks like izx's instructions require the mlx and ini, while kiteboarder's go straight to the bin. It looks like Mellanox has removed all of their custom firmware downloads from all sites now :( , so I can't find the mlx and ini files.

I'm specifically looking for the CX354a-FCB and -QCB firmware to flash some EMC or Sun branded CX354A cards. I can download the mellanox .bin files, but I haven't tried just flashing the bin's with allow_psid_change yet.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,000
1,816
113
29
fohdeesha.com
my post here - https://forums.servethehome.com/ind...x-3-to-arista-7050-no-link.18369/#post-178015

has the source packages (with mlx files, you don't need the ini files as those are generated by you from your card during the process) for both normal and pro cards, and also instructions for compiling new fw to flash to your specific card (pretty much the same instructions as izx's)

Regarding compiling new firmware using your cards specific INI, versus just flashing a pre-made BIN from mellanox, it depends on the card. If the card has been "customized" in any way other than just different firmware, you need to compile a new firmware file using the process linked above. Cards like mezzanine cards, custom made cards for certain vendors, etc. If you use a stock BIN on these cards you'll have weird problems like LEDs not working or the second port not working

This compiling process can also be used to turn the vanilla stock 10gbE CX3 cards into 40gbE CX3 cards (compile an image for it using the INI file of the 40gbE card, eg the FCBT ini file, then flash the resulting image to the "10gbE" (QCBT) card. This obviously only works for the CX3 cards that support 40gb infiniband out of the box, but have restricted ethernet to 10gbE. (the -QCBT models)
 
Last edited:
Mar 24, 2017
1
0
1
41
+1 on the question regarding if anyone has been able to bump a ConnectX-3 (ie, MCX354A-FCBT) up to ConnectX-3 Pro (ie, FCX354A-FCCT). Even after using the C-X3 Pro custom firmware package (thanks for the mirror!) and mlxburn, I get:

# flint -d /dev/mst/mt4099_pci_cr0 -i custom.bin -allow_psid_change burn
Current FW version on flash: 2.42.5000
New FW version: 2.40.5030
Note: The new FW version is older than the current FW version on flash.

Do you want to continue ? (y/n) [n] : y

-E- Burning FS2 image failed: Device/Image mismatch: FW image file cannot be programmed to device ConnectX-3 A1, it is intended for: ConnectX-3Pro only

I see others suggesting this is doable, so I must be missing something.

-jd
 

Nick85

New Member
Jul 13, 2018
6
1
3
Bournemouth
I made a custom firmware file using the MCX314A-BCBT ini file, then flashed the exact same image to a variety of IBM, EMC, and stock VPI cards, all worked perfectly. Then ran mlxup after a reboot, it detected them as MCX314A-BCBT and updated them all to the latest Mellanox firmware as normal. They all work afterwards too at 40G on both ports. Happy days.

I haven't yet tried crossflashing them to a Pro, but I have a genuine Pro arriving in the next week or to compare, might then try and upgrade a non Pro and see what happens. Will report back either way.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Here's a challenge: I've come across an IBM Power architecture CX3 Pro with some weird firmware behavior. If I boot the card normally it doesn't show up in lspci at all, if I boot with the Firmware Not Present jumper closed I can flash the card, but the original firmware is still present after a cold boot:

Code:
Image type:            FS2
FW Version:            2.42.5000
FW Version(Running):   5988.47812.10365
FW Release Date:       5.9.2017
Product Version:       02.42.50.00
Rom Info:              type=PXE version=3.4.752
Device ID:             4103
Description:           Node             Port1            Port2            Sys image
GUIDs:                 ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
MACs:                                       248a07eefc80     248a07eefc81
VSD:                
PSID:                  MT_1200111023
After a cold boot the card is gone again. Booting with the jumper closed gives the above output from flint.

Anyone know how to flash something like this?

Edit: Here's the product page for the card: IBM Knowledge Center

Edit: This is either an 00RX859 or an 00RX856 -- it shipped with a high profile bracket so I think it's an 859.
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,000
1,816
113
29
fohdeesha.com
Have you tried zeroing out the entire flash chip while in recovery mode? It seems it's booting from a second recovery partition or something that still has the original firmware in it. Make sure in recovery mode when flashing you're programming back in the GUID's as well (they seem blank in that output). can't remember the flash erase command offhand but I know it's in the MFT manual

out of curiosity before erasing anything, can you dump the entire flash?

flint -d /dev/mst/mt4099_pci_cr0 rb 0 2097152 before.bin

that dumps the entire flash size of the regular cx3, but I'd imagine the cx3 pro uses the same flash size. If you want to be sure, there's a command that shows flash IC info in mft including the size, you just need to put that in the command in bytes. Post the dump here if you can, I'm curious
 
  • Like
Reactions: arglebargle

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Have you tried zeroing out the entire flash chip while in recovery mode? It seems it's booting from a second recovery partition or something that still has the original firmware in it. Make sure in recovery mode when flashing you're programming back in the GUID's as well (they seem blank in that output). can't remember the flash erase command offhand but I know it's in the MFT manual

out of curiosity before erasing anything, can you dump the entire flash?

flint -d /dev/mst/mt4099_pci_cr0 rb 0 2097152 before.bin

that dumps the entire flash size of the regular cx3, but I'd imagine the cx3 pro uses the same flash size. If you want to be sure, there's a command that shows flash IC info in mft including the size, you just need to put that in the command in bytes. Post the dump here if you can, I'm curious
So here's the flash IC info:

Code:
root@ted:~# flint -d /dev/mst/mt504_pciconf0 hw query
HW Info:
  HwDevId               503
  HwRevId               0x0
Flash Info:
  Type                  M25PXxx
  TotalSize             0x200000
  Banks                 0x1
  SectorSize            0x1000
  WriteBlockSize        0x10
  CmdSet                0x80
It's exactly the same size as the flash on the CX-2 in one of my other machines.

I'll pull the image now.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Here's the full dump -- I've already flashed stock CX3 Pro En on to one of the slots at this point. Also included are a dump of the FW bin before that and the .ini.

00RX859.tgz

Also, the GUIDS are empty because there aren't any 10GbE CX-3 Pro VPI cards, just EN.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Before I go ahead and zero the firmware completely what's my recovery command to flash the before dump back?

bb ?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,000
1,816
113
29
fohdeesha.com
Looks like there's definitely multiple images in that dump.

To write back an image raw to the flash:

wb <data-file> <addr> :

flint -d /dev/mst/mt4099_pci_cr0 wb before.bin 0

If I remember right the only flash erase command is sector by sector, so to zero out the flash you might need to use that wb command to write a 2097152 byte file full of FF's to it. I've attached one for your erasure pleasure
 

Attachments

  • Like
Reactions: arglebargle

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Also, unrelated but @arglebargle do you have any 12th gen dell servers (R720 etc)? Working on something interesting with IDRAC7 https://i.imgur.com/hCobBkZ.png
Unfortunately I don't, I'm 100% consumer hardware over here atm.

So this is getting interesting:
Code:
flint -d /dev/mst/mt504_pciconf0 wb zero.bin 0
Success, flash is gone. I shut it down and cold booted for good measure.
Code:
flint -d /dev/mst/mt504_pciconf0 -i fw-ConnectX3Pro-rel-2_42_5000-MCX312B-XCC_Ax-FlexBoot-3.4.752.bin -mac 248a07eefc80 --nofs b
Success, sort of?
Code:
root@ted:~# flint -d /dev/mst/mt504_pciconf0 q
Image type:            FS2
FW Version:            2.42.5000
FW Version(Running):   5988.47812.10365
FW Release Date:       5.9.2017
Product Version:       02.42.50.00
Rom Info:              type=PXE version=3.4.752
Device ID:             4103
Description:           Node             Port1            Port2            Sys image
GUIDs:                 0000000000000000 0000000000000000 0000000000000000 0000000000000000 
MACs:                                       248a07eefc80     248a07eefc81
VSD:                   
PSID:                  MT_1200111023
Ok, maybe that's just the failsafe FW.. Shutdown, pull the jumper, boot: card is gone again.

Shutdown, replace jumper, boot:
Code:
root@ted:~# flint -d /dev/mst/mt504_pciconf0 q
Image type:            FS2
FW Version:            2.42.5000
FW Version(Running):   5988.47812.10365
FW Release Date:       5.9.2017
Product Version:       02.42.50.00
Rom Info:              type=PXE version=3.4.752
Device ID:             4103
Description:           Node             Port1            Port2            Sys image
GUIDs:                 0000000000000000 0000000000000000 0000000000000000 0000000000000000
MACs:                                       248a07eefc80     248a07eefc81
VSD:                   
PSID:                  MT_1200111023
I'm not really sure where to go from here.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Yeah, here's the whole process one more time for luck:

Code:
root@ted:~/tmp/mlx/cx3pro# flint -d /dev/mst/mt504_pciconf0 wb zero.bin 0
Writing Block:   - OK
root@ted:~/tmp/mlx/cx3pro# flint -d /dev/mst/mt504_pciconf0 q
-E- Failed to query Device: /dev/mst/mt504_pciconf0.No valid image found
root@ted:~/tmp/mlx/cx3pro# flint -d /dev/mst/mt504_pciconf0 -i fw-ConnectX3Pro-rel-2_42_5000-MCX312B-XCC_Ax-FlexBoot-3.4.752.bin -mac 248a07eefc80 --nofs b
    You are about to change the Guids/Macs/Uids on the device:
                        New Values              Current Values
        Node  GUID:     0000000000000000              N/A
        Port1 GUID:     0000000000000000              N/A
        Port2 GUID:     0000000000000000              N/A
        Sys.Image GUID: 0000000000000000              N/A
        Port1 MAC:          248a07eefc80              N/A
        Port2 MAC:          248a07eefc81              N/A
 Do you want to continue ? (y/n) [n] : y
-W- Can not extract VSD/PSID info from flash.
    To use a specific VSD, abort and re-burn specifying the
    needed info (using command line flags -vsd / -use_image_ps).
    You can also continue burn using blank VSD.
 Do you want to continue ? (y/n) [n] : n
-E- Aborted by user
root@ted:~/tmp/mlx/cx3pro# flint -d /dev/mst/mt504_pciconf0 -i fw-ConnectX3Pro-rel-2_42_5000-MCX312B-XCC_Ax-FlexBoot-3.4.752.bin -mac 248a07eefc80 --nofs --use_image_ps --dual_image b
    You are about to change the Guids/Macs/Uids on the device:
                        New Values              Current Values
        Node  GUID:     0000000000000000              N/A
        Port1 GUID:     0000000000000000              N/A
        Port2 GUID:     0000000000000000              N/A
        Sys.Image GUID: 0000000000000000              N/A
        Port1 MAC:          248a07eefc80              N/A
        Port2 MAC:          248a07eefc81              N/A
 Do you want to continue ? (y/n) [n] : y
    Current FW version on flash:  N/A
    New FW version:               2.42.5000
Burn process will not be failsafe. No checks will be performed.
ALL flash, including the Invariant Sector will be overwritten.
If this process fails, computer may remain in an inoperable state.
 Do you want to continue ? (y/n) [n] : y
Burning FS2 FW image without signatures - OK
Restoring signature                     - OK
root@ted:~/tmp/mlx/cx3pro# flint -d /dev/mst/mt504_pciconf0 q
Image type:            FS2
FW Version:            2.42.5000
FW Version(Running):   5988.47812.10365
FW Release Date:       5.9.2017
Product Version:       02.42.50.00
Rom Info:              type=PXE version=3.4.752
Device ID:             4103
Description:           Node             Port1            Port2            Sys image
GUIDs:                 0000000000000000 0000000000000000 0000000000000000 0000000000000000
MACs:                                       248a07eefc80     248a07eefc81
VSD:                   n/a
PSID:                  MT_1200111023
root@ted:~/tmp/mlx/cx3pro#
Cold booting leaves the device undetected again, re-installing the jumper puts me right back where I am with FW 5988.* running on the card.
 
  • Like
Reactions: Freebsd1976