Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

Discussion in 'Networking' started by ullbeking, Jun 12, 2018.

  1. ullbeking

    ullbeking Member

    Joined:
    Jul 28, 2017
    Messages:
    183
    Likes Received:
    10
    Hello!

    I have the opportunity to purchase some used dual-port Mellanox MCX354A-QCBT ConnectX-3 cards with VPI network adapter. (10 GbE Ethernet and 40 Gbps InfiniBand.) These would be perfect for my cluster after correct configuration..

    They were originally pulled from EMC systems and currently have EMC firmware installed. I want to flash the original stock (Mellanox) firmware and return them to functionally equivalant stock Mellanox ConnectX-3 cards.

    I found a few threads about this ( https://forums.servethehome.com/index.php?threads/anyone-flashing-mellanox-connectx-3-in-qty.16298/ and https://forums.servethehome.com/ind...lanox-connectx-3-en-with-stock-firmware.8689/ ). These discussions indicate that it's a risky process and that the original firmware files are not easy to find. If it goes horribly wrong for a particular card is it difficult to restore using a hardware flasher?

    Has anybody tried this recently, and what have your experiences been? Are there any particular hints or tips that I should follow? I intend to connect them to a Voltaire 4036 or 4036E switch, depending on what's available and how cost-effective the options are. Thanks!!

    Kind regards,

    ullbeking
     
    #1
    Last edited: Jun 12, 2018
  2. Freebsd1976

    Freebsd1976 New Member

    Joined:
    Feb 23, 2018
    Messages:
    14
    Likes Received:
    1
    i crossflash qcbt to fcbt two month ago ,so far so good
     
    #2
  3. ullbeking

    ullbeking Member

    Joined:
    Jul 28, 2017
    Messages:
    183
    Likes Received:
    10
    Thank you for the positive report, @Freebsd1976!

    Please could you provide a brief outline of what you did, or what instructions you followed, to do this? What host machine and OS did you use? Where did you get the firmware from, and did you do it by software flashing or hardware flashing?
     
    #3
  4. Freebsd1976

    Freebsd1976 New Member

    Joined:
    Feb 23, 2018
    Messages:
    14
    Likes Received:
    1
    basically i follow the first link you post
    and i use windows flash and software flash
    only problem is you need find the firmware link , i found in the this forum but now i cannot remember which post it is
     
    #4
  5. Freebsd1976

    Freebsd1976 New Member

    Joined:
    Feb 23, 2018
    Messages:
    14
    Likes Received:
    1
    #5
  6. ullbeking

    ullbeking Member

    Joined:
    Jul 28, 2017
    Messages:
    183
    Likes Received:
    10
    #6
  7. ullbeking

    ullbeking Member

    Joined:
    Jul 28, 2017
    Messages:
    183
    Likes Received:
    10
    @Freebsd1976, what firmware was on your MCX354A-QCBT before you flashed it to -FCBT? My point is my -QCBT card has EMC firmware and I want to replace it by Mellanox firmware. I think this issue is different from changing -QCBT to -FCBT. Please could you let me know what the firmware was before and after? (OEM or Mellanox.) Thanks!!
     
    #7
  8. Freebsd1976

    Freebsd1976 New Member

    Joined:
    Feb 23, 2018
    Messages:
    14
    Likes Received:
    1
    Mellanox firmware, I think Flash Oem to mellanox should be same
     
    #8
  9. oddball

    oddball Member

    Joined:
    May 18, 2018
    Messages:
    53
    Likes Received:
    9
    I just flashed a card last week as well. If you dig on their website they have all the versions of firmware available for download.

    Get their firmware tools zip first. Then interrogate your card and see what you need. Go from there.

    The issue I'm facing is the card won't recognize an Arista or Cisco DAC. Might need to break down and buy a Mellanox AOC. I had a similar issue with Cisco cards, they only recognize Cisco cables. My Mellanox branded cards take any DAC I can find.
     
    #9
  10. oddball

    oddball Member

    Joined:
    May 18, 2018
    Messages:
    53
    Likes Received:
    9
    Oh, you can save off your original firmware and flash back if you make a mistake.
     
    #10
  11. ullbeking

    ullbeking Member

    Joined:
    Jul 28, 2017
    Messages:
    183
    Likes Received:
    10
    Yes, I tried digging around but I wasn't sure of the correct one to use. Obviously I want to flash to the latest "good" firmware that makes my card into a standard Mellanox card rather than an EMC OEM card. One potentially major issue on my mind is this: Might there be anything about the EMC OEM hardware or firmware that prevents cross-flashing?

    The Mellanox firmware tools? Will this work with the EMC firmware? (I haven't actually purchased the card yet but I'm currently submitting an order for express delivery because this is kind of important to me.)

    Of course. Are there standard flashing tools that I can use to do this? For example, would `flashrom` work?

    Please could you give me some examples of useful tools for reading original firmware and flashing new firmware? My understanding from my research so far is that the EMC (or Mellanox?) tools do not allow cross-flashing and that using more flexible or generic tools will be necessary.

    Do you know if it's possible to flash the firmware using a hardware flasher? This would obviously require extracting the actual firmware image from the .exe file on the Mellanox website.
     
    #11
  12. oddball

    oddball Member

    Joined:
    May 18, 2018
    Messages:
    53
    Likes Received:
    9
    You're overthinking this. Mellanox makes it REALLY easy.

    They have firmware tools like flint. Look at the first link. Those are all commands from Mellanox's tools. The firmware tools are a download on their site.

    You run one of their tools to find the current firmware and PSID. From that you then use the chooser on their website to find the latest that's compatible with your card. You run flint with the new bin file and about 30s later you have new firmware. You can run all of this if the card is physically present but the OS doesn't detect it. They're built to handle undetected cards.

    You don't need hardware flashing tools or anything. Their tools allow you to pull off all of the current firmware and save it. Their own tools have everything you'd need. They have really nice documentation that walks you through it as well.

    They even have a section of their website with vendor specific details. I re-flashed a Fujitsu card to be a standard Mellanox one.

    From what I understand the cards are 100% generic with vendor add-ons in flash only. If this is an EMC version of a Connectx-3 then you're fine.

    And just to clarify, you have an EMC version of a Mellanox Connectx-3 that you want to flash to a Mellanox generic Connectx-3 right? Your last post made me think you're trying to toss custom firmware on there. Flint will allow that as well, but that's way out of my league.
     
    #12
  13. oddball

    oddball Member

    Joined:
    May 18, 2018
    Messages:
    53
    Likes Received:
    9
    #13
  14. ullbeking

    ullbeking Member

    Joined:
    Jul 28, 2017
    Messages:
    183
    Likes Received:
    10
    Thank you @oddball !! This has been a huge help. I spent the evening experimenting with some of the flashing tools, compiling them, etc, and am getting a better feel now for how this is supposed to work. I’ll let you know how I get on once I receive my experimental HCAs.
     
    #14
  15. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    33
    Likes Received:
    4
    Hey guys. So before I actually burn my card I want to verify this should be correct. Here is prior info

    I bought some MCX354A-QCBT

    During the firmware creation process instead of my normal card I choose MCX354A-FCB_A2-A5.ini

    This is the firmware info

    So if thats good I should be all set!
     
    #15
  16. fohdeesha

    fohdeesha Well-Known Member

    Joined:
    Nov 20, 2016
    Messages:
    567
    Likes Received:
    399
    If you're just flashing a MCX354A-QCBT to a MCX354A-FCBT, you don't need to compile a firmware image - you can just flash the premade latest MCX354A-FCBT image from their site - this is nice because the latest firmware sources available for compiling are actually quite old.

    When you use the firmware sources with the stock unedited FCBT ini file to compile a new image, this is all you're generating anyway. Except the pre-generated images on mellanox's site have been compiled with much newer firmware sources

    from my notes:

    Code:
    ##Download latest Mellanox Firmware tools and install them  + dependencies:
    
    wget http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-deb.tgz
    tar -xvf mft-4.10.0-104-x86_64-deb.tgz
    cd mft-4.10.0-104-x86_64-deb/
    apt-get install gcc make dkms unzip
    apt install linux-headers-$(uname -r)
    ./install.sh
    
    #Start MST service:
    mst start
    mst status
    #copy the dev address with cr0 in it, like:
    /dev/mst/mt4099_pci_cr0
    #Use that address in the following commands
    
    #Download latest FCBT firmware and unzip:
    wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip
    unzip fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip
    
    #Backup the cards current board definition file just in case
    flint -d /dev/mst/mt4099_pci_cr0 dc orig_firmware.ini
    
    #Flash the FCBT image to the card
    flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn
    
    Cold boot the server when done
    
    
    Some useful config commands to config the card when it comes back up:

    Code:
    #start mst service again
    mst start
    
    #get detailed firmware info
    mlxfwmanager --query
    
    #get the device ID again:
    mst status
    
    #use it to see the cards current configuration:
    mlxconfig -d /dev/mst/mt4099_pci_cr0 query
    
    use it in config commands to configure the card
    #for instance, to turn both ports from VPI/Auto to Ethernet only:
    mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
    
    
    #turn off bootrom crap
    mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P1=false
    mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P2=false
    mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P1=0
    mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P2=0
    
    
    
    
    
    
     
    #16
  17. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    33
    Likes Received:
    4
    Ohh great, that was a lot easier. Looks like it worked well. 2 more cards to flash. Thank you!
     
    #17
    Tha_14 and fohdeesha like this.
  18. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    33
    Likes Received:
    4
    Is it possible to flash to ConnectX-3 Pro MCX354A-FCCT?
     
    #18
  19. arglebargle

    arglebargle Member

    Joined:
    Jul 15, 2018
    Messages:
    81
    Likes Received:
    36
    I'm interested in this as well -- Are the CX3 Pro cards different hardware or are the features just firmware locked?

    Edit: I have two QCBT cards here now, I'm waiting on full height brackets and then I'll try flashing one to FCCT to test.
     
    #19
    Last edited: Aug 1, 2018
    Hindsight likes this.
  20. arglebargle

    arglebargle Member

    Joined:
    Jul 15, 2018
    Messages:
    81
    Likes Received:
    36
    So no-go on the FCCT firmware, the firmware tools stop you from flashing the Pro firmware on a non Pro card.

    Does anyone know if the IC on the pro is actually different or are they just differentiating SKUs to segment the market? If the hardware is identical I'd be happy to crack the firmware tool and try flashing Pro firmware on a non pro.

    Also, I can't get my workstation to post with a CX-3 installed. This is the same machine I've had a CX-2 in for the last year, on the off chance this has happened to someone else is there any obvious reason a CX-3 would cause a Z97 board to hang on boot? I'm going to try again this afternoon after I swap cards around and update the firmware and disable PXE boot.
     
    #20
    tommybackeast and Tha_14 like this.
Similar Threads: Flashing stock
Forum Title Date
Networking Flashing Delta 7024 to Dell 8024 May 17, 2018
Networking Anyone flashing Mellanox ConnectX-3 in qty? Aug 27, 2017
Networking Flash OEM (IBM) Mellanox ConnectX-3 EN with stock firmware? Feb 21, 2016

Share This Page