Flash OEM (IBM) Mellanox ConnectX-3 EN with stock firmware?

Discussion in 'Networking' started by Stereodude, Feb 21, 2016.

  1. Stereodude

    Stereodude Active Member

    Joined:
    Feb 21, 2016
    Messages:
    372
    Likes Received:
    54
    So I picked up an IBM flavored Mellanox ConnectX-3 EN (MCX312A-XCBT / Dual 10GbE SFP+) from eBay. It's FRU: 00D9692. The latest WinOF driver installed without issue (Windows 7 Pro).

    The card has a PSID of IBM1080111023 so the standard MT_1080110023 firmware won't load on it. Here's what I see:

    ------------------------------------------------------------------------
    C:\Program Files\Mellanox\WinMFT>flint -d mt4099_pci_cr0 -i fw-ConnectX3-rel-2_34_5000-MCX312A-XCB_A1-FlexBoot-3.4.521.bin burn

    Current FW version on flash: 2.30.3200
    New FW version: 2.34.5000

    -E- PSID mismatch. The PSID on flash (IBM1080111023) differs from the PSID in the given image (MT_1080110023).
    ------------------------------------------------------------------------

    Is there a way to flash the generic firmware? I saw this guide but I'm not sure if I can apply it to my case or not. The firmware I have is a .bin, not a .mlx, and I'm not sure about where to get an equivalent .ini file.

    Has anyone tried this or know how it can be done?
     
    #1
    Last edited: Feb 21, 2016
  2. Stereodude

    Stereodude Active Member

    Joined:
    Feb 21, 2016
    Messages:
    372
    Likes Received:
    54
    I guess I should have looked a little harder. I found this: http://www.mellanox.com/page/custom_firmware_table

    However, the ConnectX-3 EN download package is for rel-2.36.5000, and the firmware page shows rel-2.34.5000 for the MT_1080110023. I'm not sure whether the IBM1080111023 is a MT_1080110023 or a MT_1080120023...

    The sticker on the back of the card says Rev. A5 and was made in 2014-03-27 which would seem to suggest it might be equivalent to a MT_1080120023, but I'm not sure.

    Since I don't have my twinax cable yet or anything to plug this into, I guess I will sit tight for a while. Hopefully someone can shed some light on this.
     
    #2
    Last edited: Feb 21, 2016
  3. Stereodude

    Stereodude Active Member

    Joined:
    Feb 21, 2016
    Messages:
    372
    Likes Received:
    54
    So someone on another forum pointed out the mlxup utility for Mellanox cards. That brought the card up from 2.30.3200 to 2.33.5000. Here's what I saw (MAC addresses removed by me):

    Code:
    Querying Mellanox devices firmware ...
    
    Device #1:
    ----------
    
      Device Type:  ConnectX3
      Part Number:  00D9691_Ax
      Description:  Mellanox ConnectX-3 EN Dual-port SFP+ 10GbE Adapter
      PSID:  IBM1080111023
      PCI Device Name:  mt4099_pci_cr0
      Port1 MAC: 
      Port2 MAC: 
      Versions:  Current  Available  
      FW  2.30.3200  2.33.5000  
      PXE  3.4.0148  3.4.0460  
      UEFI  11.0.0022  14.6.0025  
    
      Status:  Update required
    
    ---------
    Found 1 device(s) requiring firmware update...
    
    Device #1: Updating FW ...  Done
    
    
    Restart needed for updates to take effect.
    I'd still like to know if I can turn it into a retail MT_1080120023 (assuming that's what it is).
     
    #3
  4. izx

    izx Active Member

    Joined:
    Jan 17, 2016
    Messages:
    113
    Likes Received:
    36
    If you got FRU 00D9692, its equivalent to the Mellanox MCX312A-XCBT

    [​IMG]

    If it's an A5, Mlnx PSID should be MT_1080120023 (for rev. A2-A6). Verify mlxburn -d <device> -vpd gives you the same revision number.
     
    #4
  5. Stereodude

    Stereodude Active Member

    Joined:
    Feb 21, 2016
    Messages:
    372
    Likes Received:
    54
    Here's what I get (less the serial number which I've edited out):
    Code:
    C:\Program Files\Mellanox\WinMFT>mlxburn -d mt4099_pciconf0 -vpd
    
      VPD-KEYWORD  DESCRIPTION  VALUE
      -----------  -----------  -----
    Read Only section:
    
      PN  Part Number  00D9691
      EC  Revision  N32186C
      SN  Serial Number 
      V0  Misc Info  Mellanox ConnectX3 10 GbE Adapter for IBM System x
      RV  Checksum Complement  0x18
      IDTAG  Board Id  Mellanox CX3 Ethernet Adapter
      MN  Manufacture ID  A5
     
    #5
  6. izx

    izx Active Member

    Joined:
    Jan 17, 2016
    Messages:
    113
    Likes Received:
    36
    Ok, it's the last line. Please check that the recovery jumper is present on your card. It should be labeled FLSH PRSNT or similar. Can be straight or angled. A few cards won't have pins but the holes in the PCB should be there.
     
    #6
  7. Stereodude

    Stereodude Active Member

    Joined:
    Feb 21, 2016
    Messages:
    372
    Likes Received:
    54
    The through holes are on the PCB, but there is no header for a jumper. I can solder one on pretty easily.
     
    #7
  8. Stereodude

    Stereodude Active Member

    Joined:
    Feb 21, 2016
    Messages:
    372
    Likes Received:
    54
    So if I solder a 2 pin header and close the jumper what's that get me, or what's the next step?
     
    #8
  9. izx

    izx Active Member

    Joined:
    Jan 17, 2016
    Messages:
    113
    Likes Received:
    36
    You don't need to solder header pins there, just use a little bare copper wire (solid or multistranded) to loop through the vias and give it a few twists to short them. Now plug the card back in -- it should boot without the firmware in a "DFU" mode. This is just to check you have a means to recover the card if somehow it gets bricked.

    (Thanks @Stereodude for this info:) See what it shows up as in MST and Device Manager -- should show up as a Memory Controller in Linux, or a PCI Memory Controller on Windows. Run mststatus to verify it sees the "livefished" card as a Mellanox PCI conf interface (mtxxx_pciconf0). On Linux, lspci -vv should also provide further information, in the form of Mellanox (Flash Recovery Mode) or similar.

    Here's the Mellanox docs on livefish: Dropbox - Mellanox_MFT_Livefish.pdf
     
    #9
  10. Stereodude

    Stereodude Active Member

    Joined:
    Feb 21, 2016
    Messages:
    372
    Likes Received:
    54
    Okay, so the card be recovered if bricked with the vias shorted and using mlxburn?

    But how do I get the IBM card to take the standard Mellanox firmware in the first place or is livefishing the card and using mlxburn how I get the standard firmware on the card?
     
    #10
  11. izx

    izx Active Member

    Joined:
    Jan 17, 2016
    Messages:
    113
    Likes Received:
    36
    Cross-flashing to Mellanox FW (ConnextX-2 or ConnextX-3 based devices only)
    Not tested with all devices - may brick yours permanently - YMMV!

    • First, backup the existing configuration if you ever need to recover it and to compare notes post-flash
    1. Save basic info such as GUIDs, MACs, etc.:
    Code:
    flint -d <device> query full > flint_query.txt
    2. Save low-level flash chip info:
    Code:
    flint -d <device> hw query > flint_hwinfo.txt
    3. Save existing FW:
    Code:
    flint -d <device> ri orig_firmware.bin
    4. Save existing FW configuration:
    Code:
    flint -d <device> dc orig_firmware.ini
    5. Save existing PXE ROM image (if any):
    Code:
    flint -d <device> rrom orig_rom.bin
    6. Save existing PCI VPD (vital product data):
    Code:
    mlxburn -d <device> -vpd > orig_vpd.txt
    • Now, download "custom" Mellanox firmware tarball for your adapter from here (e.g. for the MCX312A-XCBT) and extract. Note that it must contain a multi-adapter MLX firmware file, NOT a BIN file.
    • Finally, customize and burn:
    1. Identify the correct configuration file for your device/revision (here, it's MCX_312A-XCB_A2-A6.ini)
    2. "Burn" the new firmware to an image file (not the device) with the correct config:
    Code:
    mlxburn -fw fw-ConnectX3-rel.mlx -conf MCX312A-XCB_A2-A6.ini -wrimage mlnx_firmware.bin
    3. Verify the image is good with flint (no errors or warnings, should say "is bootable"):
    Code:
    flint -i mlnx_firmware.bin verify
    4. And check that the image's FW version and PSID are what you want:
    Code:
    flint -i mlnx_firmware.bin query full
    5. Let's roll!
    Code:
    flint -d <device> -i mlnx_firmware.bin -allow_psid_change burn
    
    Reboot, and verify with flint that the PSID has changed while the GUIDs/MACs are the same as before.
     
    #11
    Last edited: Feb 29, 2016
  12. izx

    izx Active Member

    Joined:
    Jan 17, 2016
    Messages:
    113
    Likes Received:
    36
    Yes. mlxburn uses flint to do the actual flashing, so either works.

    As I laid out in the steps above, flint has an override to allow the PSID change. mlxburn includes card-specific information, such as the PSID, from the configuration file into the generic blob to generate a specific fw image. flint has tons of other overrides -- Mellanox flashing tools have always been insanely flexible. Heck, it'll let you write arbitrary data to arbitrary flash addresses if you want!

    From what I saw in the MFT manual, the process appears to be more involved beginning with the ConnectX-4 series. That's why I said only CX2 & CX3 :)

    Also, this disclaimer about livefish mode straight from the MFT manual:
    The MTUSB-1 is a USB-to-I2C adapter, and I'm sure Mellanox has sufficient documentation somewhere to let you use any generic USB-to-I2C. Of course, the very next disclaimer says:
    :D
     
    #12
    Stereodude likes this.
  13. Stereodude

    Stereodude Active Member

    Joined:
    Feb 21, 2016
    Messages:
    372
    Likes Received:
    54
    Thanks for the very detailed reply. I will try this later today.
     
    #13
  14. Stereodude

    Stereodude Active Member

    Joined:
    Feb 21, 2016
    Messages:
    372
    Likes Received:
    54
    The whole sequence worked! PSID changed, GUIDs/MACs didn't. It now has the latest firmware. (I've removed the MACs)
    Code:
    Image type:          FS2
    FW Version:          2.36.5000
    FW Release Date:     26.1.2016
    MIC Version:         2.0.0
    Config Sectors:      2
    PRS Name:            cx3-1_MCX312A.prs
    Rom Info:            type=PXE version=3.4.460 devid=4099
                         type=UEFI version=14.6.25
    Device ID:           4099
    Description:         Node             Port1            Port2            Sys image
    GUIDs:               ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
    MACs:
    VSD:               
    PSID:                MT_1080120023
    Interestingly it still has some IBM traces in it though. The VPD didn't change for example.
    Code:
    VPD-KEYWORD  DESCRIPTION  VALUE
      -----------  -----------  -----
    Read Only section:
    
      PN  Part Number  00D9691
      EC  Revision  N32186C
      SN  Serial Number  
      V0  Misc Info  Mellanox ConnectX3 10 GbE Adapter for IBM System x
      RV  Checksum Complement  0x18
      IDTAG  Board Id  Mellanox CX3 Ethernet Adapter
      MN  Manufacture ID  A5
    You should probably put a note here for the vpd saving to use the pciconfX device, not pci_crX. I ended up with a orig_vpd.txt that just had an error message in it.
     
    #14
  15. izx

    izx Active Member

    Joined:
    Jan 17, 2016
    Messages:
    113
    Likes Received:
    36
    Very nice! Thanks for being the guinea pig :)

    I will. I'd thought about putting a note to not use the cr0 interface because it's s.l.o.w. compared to direct PCI, and this is another reason not to!
     
    #15
  16. izx

    izx Active Member

    Joined:
    Jan 17, 2016
    Messages:
    113
    Likes Received:
    36
    Yeah, the VPD shouldn't (it's supposed to be read only). I notices the VPD "row" headers were rewritten: row 2 now reads Revision, but that data remains in row 6 as before.

    Mlxburn will let you overwrite the VPD data, but those values aren't in the firmware or conf files. You'd need someone with a Mellanox "original" to tell you what they should be set to. It's just dressing anyway -- doesn't affect operation.

    The card does identify as Mellanox in Device Manager, right?
     
    #16
  17. Lance Joseph

    Lance Joseph Member

    Joined:
    Oct 5, 2014
    Messages:
    79
    Likes Received:
    31
    This was a very helpful post, @izx !! Thank you.
    I've successfully flashed one MCX313A and two MCX314A (ConnectX-3 QSFP+) cards.

    Just a heads up for anyone else that comes along.
    If you'd like to switch between Infiniband/Ethernet/VPI modes, just use the following command:
    mst.exe status
    This will show you the device name. In my case, the device name was "mt4099_pciconf0"
    To set a given port to operate in Infiniband mode, I used the following command:
    mlxconfig.exe -d mt4099_pciconf0 set LINK_TYPE_P2=1
    This would set port 2 to Infiniband mode.

    Cheers
     
    #17
    metag likes this.
  18. Kuz

    Kuz New Member

    Joined:
    Oct 7, 2016
    Messages:
    20
    Likes Received:
    15
    I am trying to do this as well with a MCX314A but it looks like Mellanox has removed the custom firmware page from their site. Anyone able to share another source for that mcx314a?

    Thanks!

    **FOUND it** for anyone else looking they still have it up on their korean site ;)

    http://kr.mellanox.com/page/custom_firmware_table
     
    #18
    Last edited: Aug 5, 2017
  19. kiteboarder

    kiteboarder Member

    Joined:
    May 10, 2016
    Messages:
    49
    Likes Received:
    16
    Here is what I did after I could not find the custom firmware files. This worked for a Dell branded card. Note the below instructions do not include the steps of using a shorting wire to short the holes on the card. See the info in the posts above for those details. Just remember to remove the short after you update the fw.

    Step 1 - download and install
    MLNX_firmware_tool_WinMFT_x64_4_7_0_42.exe

    Step 2 - download:
    fw-ConnectX3-rel-2_40_7000-MCX312A-XCB_A2-A6-FlexBoot-3.4.746.bin.zip
    Extract .bin file and rename in to fw.bin
    copy it to:
    C:\Program Files\Mellanox\WinMFT>

    Step 3 - open command prompt as administrator
    cd to directory:
    C:\Program Files\Mellanox\WinMFT>

    Run:
    0) mst status
    1) mlxfwmanager
    2) flint -d mt26448_pci_cr0 -i fw.bin burn -allow_psid_change

    where "mt26448_pci_cr0" is the name output from commands 0 or 1

    The -allow_psid_change switch is dangerous and I make no guarantees that this will not brick your card. YMMV!!! But it worked perfectly for my Dell branded card.

    Good luck.
     
    #19
    segfault likes this.
  20. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    851
    Likes Received:
    89
    Has anyone managed to burn a pro fw to a non-pro device yet ?
    i.e. replace PSID MT_1090120019 with FW for 354a-fcct / PSID MT_1090111019
     
    #20
    Rand__ likes this.

Share This Page