Flash OEM (IBM) Mellanox ConnectX-3 EN with stock firmware?

Discussion in 'Networking' started by Stereodude, Feb 21, 2016.

  1. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,045
    Likes Received:
    757
    hmmm, the only thing I can think of at this point - are you sure it's a CX3 Pro? This is the exact same behavior I got when I flashed a non-pro with pro firmware - firmware query in recovery mode would show the flash is perfectly fine and had a firmware image in it, but booting without the jumper, the code just couldn't boot the card.

    or maybe if it is a cx3pro card, are you sure you're flashing the right firmware? Showing proper flash and firmware loaded but not initializing the card all points to incorrect FW to me but I could be wrong

    Is that IBM-00RX859.ini ini in the zip you posted dumped from BEFORE you flashed the card? Because that definitely calls it out as

    Code:
    Name = MCX312B-XCC_Ax
    Description = ConnectX-3 Pro EN network interface card; 10GigE; dual-port SFP+; PCIe3.0 x8 8GT/s; RoHS R6
    PRS_name    = cx3pro_MCX312B_2mb.prs
    which is definitely what you're trying to flash, so I dunno. It is also IDENTICAL to the MCX312B-XCC_Ax.ini that gets used to build the official image you're flashing, so I dunno why it wouldn't work. I guess the card could be bad, it does happen
     
    #41
    arglebargle likes this.
  2. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    379
    Likes Received:
    99
    Yeah, the .ini and .bin in the tarball are dumps taken before I touched the flash at all. I wasn't expecting to need to dump the entire flash content or I'd have done that before touching the card.

    Here's IBM's current Mellanox reference: RG_IBM.pdf -- I'm fairly certain this is a CX3 Pro.

    Welp, I figured for $55 I could roll the dice on this one. I wasn't expecting IBM's firmware to be dug in like a tick.
     
    #42
  3. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    379
    Likes Received:
    99
    The only thing I can think of now is finding someone with an identical CX3 Pro card and having them dump their flash, maybe there's something in one of the other segments that I'm missing.

    Code:
    root@ted:~# !483
    flint -d /dev/mst/mt504_pciconf0 verify
    
        FS2 failsafe image. Start address: 0x80000. Chunk size 0x80000:
    
        NOTE: The addresses below are contiguous logical addresses. Physical addresses on
              flash may be different, based on the image start address and chunk size
    
        /0x00000038-0x0000065b (0x000624)/ (BOOT2) - OK
        /0x0000065c-0x0000297f (0x002324)/ (BOOT2) - OK
        /0x00002980-0x00003923 (0x000fa4)/ (Configuration) - OK
        /0x00003924-0x0001d14b (0x019828)/ (ROM) - OK
        /0x0001d14c-0x0001d18f (0x000044)/ (GUID) - OK
        /0x0001d190-0x0001d317 (0x000188)/ (Image Info) - OK
        /0x0001d318-0x0002a6af (0x00d398)/ (DDR) - OK
        /0x0002a6b0-0x0002b713 (0x001064)/ (DDR) - OK
        /0x0002b714-0x0002bae3 (0x0003d0)/ (DDR) - OK
        /0x0002bae4-0x00069eb3 (0x03e3d0)/ (DDR) - OK
        /0x00069eb4-0x0006ed37 (0x004e84)/ (DDR) - OK
        /0x0006ed38-0x0007332b (0x0045f4)/ (DDR) - OK
        /0x0007332c-0x00073e23 (0x000af8)/ (DDR) - OK
        /0x00073e24-0x000a3fc7 (0x0301a4)/ (DDR) - OK
        /0x000a3fc8-0x000a7b73 (0x003bac)/ (DDR) - OK
        /0x000a7b74-0x000bd067 (0x0154f4)/ (DDR) - OK
        /0x000bd068-0x000bd16f (0x000108)/ (DDR) - OK
        /0x000bd170-0x000c21f3 (0x005084)/ (DDR) - OK
        /0x000c21f4-0x000c39ef (0x0017fc)/ (Configuration) - OK
        /0x000c39f0-0x000c3a63 (0x000074)/ (Jump addresses) - OK
        /0x000c3a64-0x000c405b (0x0005f8)/ (FW Configuration) - OK
        /0x00000000-0x000c405b (0x0c405c)/ (Full Image) - OK
    
    -I- FW image verification succeeded. Image is bootable.
    
     
    #43
  4. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    47
    Likes Received:
    4
    i crossflash three Mellanox MCX354A-qcbt (not Oem version)to fcbt using self complied firmware,two card is ok both port work in eth mode .but one card only has one port work (in Windows it recognized two port),the second port always show cable disconnect.
    I carefully look through the three crad ,they should seems.

    Is this problem due to cross flash or bad hardware?

    Now I can try following two way ,which way is better?
    1 using Mellanox originall fcbt firmware to try
    2 using Mellanox original qcbt firmware to try
    Btw I back up the original ini file but now I delete it
     
    #44
  5. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    379
    Likes Received:
    99
    Try the Mellanox QCBT image and see if both ports work, if both ports work then you know that the card functions.

    I'd then try the FCBT image and verify that both ports still work, then you can try and find the problem with your custom image.

    One thing, check the hardware revision of all three cards. IIRC the Rev A1 cards can't run at the higher clock rate that is required for VPI functionality -- I suspect that if 2 of 3 cards work for you and all are undamaged then the last may be a Rev A1.
     
    #45
  6. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    47
    Likes Received:
    4
    Checked the three card, all are rev 01 on the label ,product date different but all producted in 2014, I will try further
     
    #46
  7. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    47
    Likes Received:
    4
    EF53556A-8C2F-4CDF-BE3B-4D68A491E462.jpeg
    This is revision picture , all three card same
     
    #47
  8. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    379
    Likes Received:
    99
    Flash the QCBT firmware and verify that both ports work, then flash the FCBT firmware and test both ports again. If only one port works with QCBT the card is damaged. If only one port works with FCBT then I'm at a loss.
     
    #48
  9. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    379
    Likes Received:
    99
    A couple of members tried forced conversion of regular CX-3 cards to Pro and didn't have any success. The cards don't initialize on boot, you have to use the flash not present jumper to boot them in recovery mode to see them at all.
     
    #49
  10. tommybackeast

    tommybackeast Member

    Joined:
    Jun 10, 2018
    Messages:
    97
    Likes Received:
    23
    I have a Dell T620; with iDRAC 7 Enterprise ; with the Mellanox 10/40 card in it (using 10GB back to 10GB on Brocade 6450-24p)

    Edit: what do those RACADM commands do exactly?
     
    #50
  11. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    47
    Likes Received:
    4
    test on windows further today,using one mellanox dac cable connect two port , flash to qcbt and change to eth mode both port work, but what strange are on ib or vpi mode not work (opensm started but always said port down).
    flash to mellanox fcbt or compiled firwmware only one port work

    seems card is ok on qcbt firmware but don’t know why it not work on ib or vpi mode, maybe casused by one dac connect two port?
    i am total confusing , the dac i used is mc2207130-001 .
    now i have two questions:
    1 why it can work under qcbt firmware on eth but not on ib or vpi mode
    2 why this card only one port work on fcbt firmware ,the other two card have same revision both port work
     
    #51
    Last edited: Sep 15, 2018
  12. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    47
    Likes Received:
    4
    more test , use two card and same dac to test .
    the bad card only work on eth mode both port (qcbt firmware) on 10gb, on ib mode only one port work (qcbt firmware) on 40gb.
    the bad card(fcbt firmware) work only one port on eth or ib mode on 56gb.

    maybe bad lucky , this is a bad card? one port only work on eth mode?
     
    #52
  13. TeleFragger

    TeleFragger Member

    Joined:
    Oct 26, 2016
    Messages:
    85
    Likes Received:
    9
    I got an OLD hp branded connect-x card to flash. HP only went to 2.8 for firmware and I force flashed 2.9.1000 and got it working in esxi 6.7, freenas 11.1, and windows 10. this card is the old cx4 port (I think)… and would love to know what switch works with these... but going to try and do pier to pier for now...
     
    #53
  14. Stereodude

    Stereodude Active Member

    Joined:
    Feb 21, 2016
    Messages:
    398
    Likes Received:
    60
    Just as a heads up the Mellanox WinMFT tools (v 4.11.0) for Windows don't come with mic.exe anymore so you'll have to download an older version. I downloaded 4.3.0 for the .bin creation step (mlxburn -fw fw-ConnectX3-rel.mlx -conf MCX312A-XCB_A2-A6.ini -wrimage mlnx_firmware.bin) because that's what I had used back in 2016. However, the old 4.3.0 version doesn't work under Windows 10, for all the other steps so I re-installed 4.11.0. I ended up using 4.3.0 for only the .bin creation step.

    I used mlxup.exe after I crossflashed from the OEM version to stock in order to get the latest stock FW on the CX-3 EN since the tarballs are no longer seem to be available and I only had an older tarball.
     
    #54
  15. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,045
    Likes Received:
    757

    for regular cx3 cards (eg not niche mezzanine versions of the cards) you don't need to compile your own image from the old source tarballs - indeed you don't WANT to because they are quite old versions of the driver. You can directly flash the latest firmware files right off their site (exact firmware files and commands may differ from windows to linux and differing cx3 models): https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015
     
    #55
  16. Stereodude

    Stereodude Active Member

    Joined:
    Feb 21, 2016
    Messages:
    398
    Likes Received:
    60
    Good to know. I'll keep that in mind should I buy more of them that need crossflashing.
     
    #56
  17. Blue)(Fusion

    Blue)(Fusion Member

    Joined:
    Mar 1, 2017
    Messages:
    75
    Likes Received:
    8
    I just picked up 4 of these IBM1080110023 dual port CX3 cards. They are all running firmware 2.33.5000. I see IBM has the latest firmware at 2.40.5048 on their website. I downloaded the binary file, but it appears to be a linux program, not an image I can flash to the card using mstflint.

    I have only mstflint to upload the latest firmware. No mlxburn package available (is it even avail for linux?) What exactly do I do to get the latest firmware on my cards?
     
    #57
  18. edrozenberg

    edrozenberg New Member

    Joined:
    Nov 26, 2017
    Messages:
    4
    Likes Received:
    0
    If it's a something.bin file, that's not a Linux binary. That's the firmware image. There's lots of info in this thread and others on SVTH on using flint (part of MFT tools) or mlxup to update the firmware. If you're burning a newer version of the existing firmware, it's straightforward with either of these tools.

    http://www.mellanox.com/page/firmware_HCA_FW_update

    If you're trying to burn firmware to change from say IBM or Dell OEM to Mellanox, it requires some additional magic such as the '--allow-psid-change' mstflint flag (care required to make sure to use the correct firmware) to force burning of an image it would normally not allow you to flash.

     
    #58
  19. Blue)(Fusion

    Blue)(Fusion Member

    Joined:
    Mar 1, 2017
    Messages:
    75
    Likes Received:
    8
    I wish it were the binary firmware file, but according to the IBM instructions where I downloaded it, it is to be executed in some proprietary software to perform the update. mstflint does not recognize it. Peaking a little at the .bin file, it is an executable, not a firmware image.
     
    #59
  20. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    379
    Likes Received:
    99
    Mellanox OFED is available on their website and includes all of the standard tools for manipulating the cards. I've installed Ubuntu to a USB drive before to update firmware, you might try something like that if IBM provides a linux binary. Worst case scenario if you get a linux system up and running you can just use OFED and cross flash the cards to standard firmware and go from there.
     
    #60

Share This Page