Flash OEM (IBM) Mellanox ConnectX-3 EN with stock firmware?

Discussion in 'Networking' started by Stereodude, Feb 21, 2016.

  1. ncpjim

    ncpjim New Member

    Joined:
    Sep 25, 2017
    Messages:
    3
    Likes Received:
    0
    I have successfully flashed all my cards. now i have on issue. i didn't notice that the second port was not there after a reboot. any insight on this?
     
    #21
  2. edrozenberg

    edrozenberg New Member

    Joined:
    Nov 26, 2017
    Messages:
    2
    Likes Received:
    0
    Make sure you've rebooted the machine after flashing or changing firmware.

    It depends on what you mean by port is not there. Maybe it's not yet configured with IPv4 or v6 settings, or maybe there is no cable plugged into it, or maybe the port was automatically set to IB (inifiniband) mode rather than ethernet because it wasn't connected to ethernet when the card booted.

    On Linux you can run:

    $ mlxconfig -d /dev/mst/mt<yourdevice> query

    to view card info.

    If you don't have any /dev/mst/ items you'll need to first start mst with:
    $ mst start

    On Windows there should be a similar mlxconfig.exe command to view device info.

    In the output from mlxconfig you will hopefully see something like:

    LINK_TYPE_P1 ETH(2)
    LINK_TYPE_P2 ETH(2)

    If you don't have a P2 item, then indeed there is only 1 port recognized by the firmware.
    If that's the case, you may have flashed a firmware for a different card model with only 1 port.

    With the usual disclaimers and warnings about flashing firmware :), you might want
    to try identifying the correct firmware for your card with 2 ports and flashing that.
     
    #22
  3. segfault

    segfault New Member

    Joined:
    Apr 5, 2016
    Messages:
    26
    Likes Received:
    11
    Thanks! This worked for me as well against the HP branded cards from a recentish thread (HP 649281-B21 ConnectX-3 VPI MCX354A-FCBT $35 + $8 shipping OBO ebay). I did not need to short anything.

     
    #23
  4. alltheasimov

    alltheasimov Member

    Joined:
    Feb 17, 2018
    Messages:
    58
    Likes Received:
    10
    Is the mlx file and ini necessary, or can just a bin work? It looks like izx's instructions require the mlx and ini, while kiteboarder's go straight to the bin. It looks like Mellanox has removed all of their custom firmware downloads from all sites now :( , so I can't find the mlx and ini files.

    I'm specifically looking for the CX354a-FCB and -QCB firmware to flash some EMC or Sun branded CX354A cards. I can download the mellanox .bin files, but I haven't tried just flashing the bin's with allow_psid_change yet.
     
    #24
  5. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    980
    Likes Received:
    716
    my post here - https://forums.servethehome.com/ind...x-3-to-arista-7050-no-link.18369/#post-178015

    has the source packages (with mlx files, you don't need the ini files as those are generated by you from your card during the process) for both normal and pro cards, and also instructions for compiling new fw to flash to your specific card (pretty much the same instructions as izx's)

    Regarding compiling new firmware using your cards specific INI, versus just flashing a pre-made BIN from mellanox, it depends on the card. If the card has been "customized" in any way other than just different firmware, you need to compile a new firmware file using the process linked above. Cards like mezzanine cards, custom made cards for certain vendors, etc. If you use a stock BIN on these cards you'll have weird problems like LEDs not working or the second port not working

    This compiling process can also be used to turn the vanilla stock 10gbE CX3 cards into 40gbE CX3 cards (compile an image for it using the INI file of the 40gbE card, eg the FCBT ini file, then flash the resulting image to the "10gbE" (QCBT) card. This obviously only works for the CX3 cards that support 40gb infiniband out of the box, but have restricted ethernet to 10gbE. (the -QCBT models)
     
    #25
    Last edited: Jun 21, 2018
  6. alltheasimov

    alltheasimov Member

    Joined:
    Feb 17, 2018
    Messages:
    58
    Likes Received:
    10
    Awesome! Thanks!

    The pro firmware link is broken, but the normal one worked. I mirrored it, along with all of the cx354/3a bin's from the Mellanox website, on my google drive.
     
    #26
  7. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    980
    Likes Received:
    716
    oops, missed a capital P. pro link fixed in original post
     
    #27
  8. alltheasimov

    alltheasimov Member

    Joined:
    Feb 17, 2018
    Messages:
    58
    Likes Received:
    10
    Awesome, I added it to my mirror, too.
     
    #28
  9. Jason A. Diegmueller

    Joined:
    Mar 24, 2017
    Messages:
    1
    Likes Received:
    0
    +1 on the question regarding if anyone has been able to bump a ConnectX-3 (ie, MCX354A-FCBT) up to ConnectX-3 Pro (ie, FCX354A-FCCT). Even after using the C-X3 Pro custom firmware package (thanks for the mirror!) and mlxburn, I get:

    # flint -d /dev/mst/mt4099_pci_cr0 -i custom.bin -allow_psid_change burn
    Current FW version on flash: 2.42.5000
    New FW version: 2.40.5030
    Note: The new FW version is older than the current FW version on flash.

    Do you want to continue ? (y/n) [n] : y

    -E- Burning FS2 image failed: Device/Image mismatch: FW image file cannot be programmed to device ConnectX-3 A1, it is intended for: ConnectX-3Pro only

    I see others suggesting this is doable, so I must be missing something.

    -jd
     
    #29
  10. Nick85

    Nick85 New Member

    Joined:
    Jul 13, 2018
    Messages:
    1
    Likes Received:
    0
    I made a custom firmware file using the MCX314A-BCBT ini file, then flashed the exact same image to a variety of IBM, EMC, and stock VPI cards, all worked perfectly. Then ran mlxup after a reboot, it detected them as MCX314A-BCBT and updated them all to the latest Mellanox firmware as normal. They all work afterwards too at 40G on both ports. Happy days.

    I haven't yet tried crossflashing them to a Pro, but I have a genuine Pro arriving in the next week or to compare, might then try and upgrade a non Pro and see what happens. Will report back either way.
     
    #30
  11. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    315
    Likes Received:
    84
    Here's a challenge: I've come across an IBM Power architecture CX3 Pro with some weird firmware behavior. If I boot the card normally it doesn't show up in lspci at all, if I boot with the Firmware Not Present jumper closed I can flash the card, but the original firmware is still present after a cold boot:

    Code:
    Image type:            FS2
    FW Version:            2.42.5000
    FW Version(Running):   5988.47812.10365
    FW Release Date:       5.9.2017
    Product Version:       02.42.50.00
    Rom Info:              type=PXE version=3.4.752
    Device ID:             4103
    Description:           Node             Port1            Port2            Sys image
    GUIDs:                 ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
    MACs:                                       248a07eefc80     248a07eefc81
    VSD:                
    PSID:                  MT_1200111023
    
    After a cold boot the card is gone again. Booting with the jumper closed gives the above output from flint.

    Anyone know how to flash something like this?

    Edit: Here's the product page for the card: IBM Knowledge Center

    Edit: This is either an 00RX859 or an 00RX856 -- it shipped with a high profile bracket so I think it's an 859.
     
    #31
    Last edited: Sep 13, 2018
  12. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    980
    Likes Received:
    716
    Have you tried zeroing out the entire flash chip while in recovery mode? It seems it's booting from a second recovery partition or something that still has the original firmware in it. Make sure in recovery mode when flashing you're programming back in the GUID's as well (they seem blank in that output). can't remember the flash erase command offhand but I know it's in the MFT manual

    out of curiosity before erasing anything, can you dump the entire flash?

    flint -d /dev/mst/mt4099_pci_cr0 rb 0 2097152 before.bin

    that dumps the entire flash size of the regular cx3, but I'd imagine the cx3 pro uses the same flash size. If you want to be sure, there's a command that shows flash IC info in mft including the size, you just need to put that in the command in bytes. Post the dump here if you can, I'm curious
     
    #32
    arglebargle likes this.
  13. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    315
    Likes Received:
    84
    So here's the flash IC info:

    Code:
    root@ted:~# flint -d /dev/mst/mt504_pciconf0 hw query
    HW Info:
      HwDevId               503
      HwRevId               0x0
    Flash Info:
      Type                  M25PXxx
      TotalSize             0x200000
      Banks                 0x1
      SectorSize            0x1000
      WriteBlockSize        0x10
      CmdSet                0x80
    
    It's exactly the same size as the flash on the CX-2 in one of my other machines.

    I'll pull the image now.
     
    #33
  14. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    980
    Likes Received:
    716
    OK yup, it's the exact same flash as the normal cx3 then, 2097152 bytes (0x200000 in hex)
     
    #34
    arglebargle likes this.
  15. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    315
    Likes Received:
    84
    Here's the full dump -- I've already flashed stock CX3 Pro En on to one of the slots at this point. Also included are a dump of the FW bin before that and the .ini.

    00RX859.tgz

    Also, the GUIDS are empty because there aren't any 10GbE CX-3 Pro VPI cards, just EN.
     
    #35
  16. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    315
    Likes Received:
    84
    Before I go ahead and zero the firmware completely what's my recovery command to flash the before dump back?

    bb ?
     
    #36
  17. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    980
    Likes Received:
    716
    Looks like there's definitely multiple images in that dump.

    To write back an image raw to the flash:

    wb <data-file> <addr> :

    flint -d /dev/mst/mt4099_pci_cr0 wb before.bin 0

    If I remember right the only flash erase command is sector by sector, so to zero out the flash you might need to use that wb command to write a 2097152 byte file full of FF's to it. I've attached one for your erasure pleasure
     

    Attached Files:

    #37
    arglebargle likes this.
  18. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    980
    Likes Received:
    716
    #38
  19. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    315
    Likes Received:
    84
    Unfortunately I don't, I'm 100% consumer hardware over here atm.

    So this is getting interesting:
    Code:
    flint -d /dev/mst/mt504_pciconf0 wb zero.bin 0
    
    Success, flash is gone. I shut it down and cold booted for good measure.
    Code:
    flint -d /dev/mst/mt504_pciconf0 -i fw-ConnectX3Pro-rel-2_42_5000-MCX312B-XCC_Ax-FlexBoot-3.4.752.bin -mac 248a07eefc80 --nofs b
    
    Success, sort of?
    Code:
    root@ted:~# flint -d /dev/mst/mt504_pciconf0 q
    Image type:            FS2
    FW Version:            2.42.5000
    FW Version(Running):   5988.47812.10365
    FW Release Date:       5.9.2017
    Product Version:       02.42.50.00
    Rom Info:              type=PXE version=3.4.752
    Device ID:             4103
    Description:           Node             Port1            Port2            Sys image
    GUIDs:                 0000000000000000 0000000000000000 0000000000000000 0000000000000000 
    MACs:                                       248a07eefc80     248a07eefc81
    VSD:                   
    PSID:                  MT_1200111023
    
    Ok, maybe that's just the failsafe FW.. Shutdown, pull the jumper, boot: card is gone again.

    Shutdown, replace jumper, boot:
    Code:
    root@ted:~# flint -d /dev/mst/mt504_pciconf0 q
    Image type:            FS2
    FW Version:            2.42.5000
    FW Version(Running):   5988.47812.10365
    FW Release Date:       5.9.2017
    Product Version:       02.42.50.00
    Rom Info:              type=PXE version=3.4.752
    Device ID:             4103
    Description:           Node             Port1            Port2            Sys image
    GUIDs:                 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    MACs:                                       248a07eefc80     248a07eefc81
    VSD:                   
    PSID:                  MT_1200111023
    
    I'm not really sure where to go from here.
     
    #39
  20. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    315
    Likes Received:
    84
    Yeah, here's the whole process one more time for luck:

    Code:
    root@ted:~/tmp/mlx/cx3pro# flint -d /dev/mst/mt504_pciconf0 wb zero.bin 0
    Writing Block:   - OK
    root@ted:~/tmp/mlx/cx3pro# flint -d /dev/mst/mt504_pciconf0 q
    -E- Failed to query Device: /dev/mst/mt504_pciconf0.No valid image found
    root@ted:~/tmp/mlx/cx3pro# flint -d /dev/mst/mt504_pciconf0 -i fw-ConnectX3Pro-rel-2_42_5000-MCX312B-XCC_Ax-FlexBoot-3.4.752.bin -mac 248a07eefc80 --nofs b
        You are about to change the Guids/Macs/Uids on the device:
                            New Values              Current Values
            Node  GUID:     0000000000000000              N/A
            Port1 GUID:     0000000000000000              N/A
            Port2 GUID:     0000000000000000              N/A
            Sys.Image GUID: 0000000000000000              N/A
            Port1 MAC:          248a07eefc80              N/A
            Port2 MAC:          248a07eefc81              N/A
     Do you want to continue ? (y/n) [n] : y
    -W- Can not extract VSD/PSID info from flash.
        To use a specific VSD, abort and re-burn specifying the
        needed info (using command line flags -vsd / -use_image_ps).
        You can also continue burn using blank VSD.
     Do you want to continue ? (y/n) [n] : n
    -E- Aborted by user
    root@ted:~/tmp/mlx/cx3pro# flint -d /dev/mst/mt504_pciconf0 -i fw-ConnectX3Pro-rel-2_42_5000-MCX312B-XCC_Ax-FlexBoot-3.4.752.bin -mac 248a07eefc80 --nofs --use_image_ps --dual_image b
        You are about to change the Guids/Macs/Uids on the device:
                            New Values              Current Values
            Node  GUID:     0000000000000000              N/A
            Port1 GUID:     0000000000000000              N/A
            Port2 GUID:     0000000000000000              N/A
            Sys.Image GUID: 0000000000000000              N/A
            Port1 MAC:          248a07eefc80              N/A
            Port2 MAC:          248a07eefc81              N/A
     Do you want to continue ? (y/n) [n] : y
        Current FW version on flash:  N/A
        New FW version:               2.42.5000
    Burn process will not be failsafe. No checks will be performed.
    ALL flash, including the Invariant Sector will be overwritten.
    If this process fails, computer may remain in an inoperable state.
     Do you want to continue ? (y/n) [n] : y
    Burning FS2 FW image without signatures - OK
    Restoring signature                     - OK
    root@ted:~/tmp/mlx/cx3pro# flint -d /dev/mst/mt504_pciconf0 q
    Image type:            FS2
    FW Version:            2.42.5000
    FW Version(Running):   5988.47812.10365
    FW Release Date:       5.9.2017
    Product Version:       02.42.50.00
    Rom Info:              type=PXE version=3.4.752
    Device ID:             4103
    Description:           Node             Port1            Port2            Sys image
    GUIDs:                 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    MACs:                                       248a07eefc80     248a07eefc81
    VSD:                   n/a
    PSID:                  MT_1200111023
    root@ted:~/tmp/mlx/cx3pro#
    
    Cold booting leaves the device undetected again, re-installing the jumper puts me right back where I am with FW 5988.* running on the card.
     
    #40

Share This Page