Flash OEM (IBM) Mellanox ConnectX-3 EN with stock firmware?

Discussion in 'Networking' started by Stereodude, Feb 21, 2016.

  1. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    980
    Likes Received:
    716
    hmmm, the only thing I can think of at this point - are you sure it's a CX3 Pro? This is the exact same behavior I got when I flashed a non-pro with pro firmware - firmware query in recovery mode would show the flash is perfectly fine and had a firmware image in it, but booting without the jumper, the code just couldn't boot the card.

    or maybe if it is a cx3pro card, are you sure you're flashing the right firmware? Showing proper flash and firmware loaded but not initializing the card all points to incorrect FW to me but I could be wrong

    Is that IBM-00RX859.ini ini in the zip you posted dumped from BEFORE you flashed the card? Because that definitely calls it out as

    Code:
    Name = MCX312B-XCC_Ax
    Description = ConnectX-3 Pro EN network interface card; 10GigE; dual-port SFP+; PCIe3.0 x8 8GT/s; RoHS R6
    PRS_name    = cx3pro_MCX312B_2mb.prs
    which is definitely what you're trying to flash, so I dunno. It is also IDENTICAL to the MCX312B-XCC_Ax.ini that gets used to build the official image you're flashing, so I dunno why it wouldn't work. I guess the card could be bad, it does happen
     
    #41
    arglebargle likes this.
  2. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    315
    Likes Received:
    84
    Yeah, the .ini and .bin in the tarball are dumps taken before I touched the flash at all. I wasn't expecting to need to dump the entire flash content or I'd have done that before touching the card.

    Here's IBM's current Mellanox reference: RG_IBM.pdf -- I'm fairly certain this is a CX3 Pro.

    Welp, I figured for $55 I could roll the dice on this one. I wasn't expecting IBM's firmware to be dug in like a tick.
     
    #42
  3. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    315
    Likes Received:
    84
    The only thing I can think of now is finding someone with an identical CX3 Pro card and having them dump their flash, maybe there's something in one of the other segments that I'm missing.

    Code:
    root@ted:~# !483
    flint -d /dev/mst/mt504_pciconf0 verify
    
        FS2 failsafe image. Start address: 0x80000. Chunk size 0x80000:
    
        NOTE: The addresses below are contiguous logical addresses. Physical addresses on
              flash may be different, based on the image start address and chunk size
    
        /0x00000038-0x0000065b (0x000624)/ (BOOT2) - OK
        /0x0000065c-0x0000297f (0x002324)/ (BOOT2) - OK
        /0x00002980-0x00003923 (0x000fa4)/ (Configuration) - OK
        /0x00003924-0x0001d14b (0x019828)/ (ROM) - OK
        /0x0001d14c-0x0001d18f (0x000044)/ (GUID) - OK
        /0x0001d190-0x0001d317 (0x000188)/ (Image Info) - OK
        /0x0001d318-0x0002a6af (0x00d398)/ (DDR) - OK
        /0x0002a6b0-0x0002b713 (0x001064)/ (DDR) - OK
        /0x0002b714-0x0002bae3 (0x0003d0)/ (DDR) - OK
        /0x0002bae4-0x00069eb3 (0x03e3d0)/ (DDR) - OK
        /0x00069eb4-0x0006ed37 (0x004e84)/ (DDR) - OK
        /0x0006ed38-0x0007332b (0x0045f4)/ (DDR) - OK
        /0x0007332c-0x00073e23 (0x000af8)/ (DDR) - OK
        /0x00073e24-0x000a3fc7 (0x0301a4)/ (DDR) - OK
        /0x000a3fc8-0x000a7b73 (0x003bac)/ (DDR) - OK
        /0x000a7b74-0x000bd067 (0x0154f4)/ (DDR) - OK
        /0x000bd068-0x000bd16f (0x000108)/ (DDR) - OK
        /0x000bd170-0x000c21f3 (0x005084)/ (DDR) - OK
        /0x000c21f4-0x000c39ef (0x0017fc)/ (Configuration) - OK
        /0x000c39f0-0x000c3a63 (0x000074)/ (Jump addresses) - OK
        /0x000c3a64-0x000c405b (0x0005f8)/ (FW Configuration) - OK
        /0x00000000-0x000c405b (0x0c405c)/ (Full Image) - OK
    
    -I- FW image verification succeeded. Image is bootable.
    
     
    #43
  4. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    33
    Likes Received:
    4
    i crossflash three Mellanox MCX354A-qcbt (not Oem version)to fcbt using self complied firmware,two card is ok both port work in eth mode .but one card only has one port work (in Windows it recognized two port),the second port always show cable disconnect.
    I carefully look through the three crad ,they should seems.

    Is this problem due to cross flash or bad hardware?

    Now I can try following two way ,which way is better?
    1 using Mellanox originall fcbt firmware to try
    2 using Mellanox original qcbt firmware to try
    Btw I back up the original ini file but now I delete it
     
    #44
  5. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    315
    Likes Received:
    84
    Try the Mellanox QCBT image and see if both ports work, if both ports work then you know that the card functions.

    I'd then try the FCBT image and verify that both ports still work, then you can try and find the problem with your custom image.

    One thing, check the hardware revision of all three cards. IIRC the Rev A1 cards can't run at the higher clock rate that is required for VPI functionality -- I suspect that if 2 of 3 cards work for you and all are undamaged then the last may be a Rev A1.
     
    #45
  6. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    33
    Likes Received:
    4
    Checked the three card, all are rev 01 on the label ,product date different but all producted in 2014, I will try further
     
    #46
  7. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    33
    Likes Received:
    4
    EF53556A-8C2F-4CDF-BE3B-4D68A491E462.jpeg
    This is revision picture , all three card same
     
    #47
  8. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    315
    Likes Received:
    84
    Flash the QCBT firmware and verify that both ports work, then flash the FCBT firmware and test both ports again. If only one port works with QCBT the card is damaged. If only one port works with FCBT then I'm at a loss.
     
    #48
  9. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    315
    Likes Received:
    84
    A couple of members tried forced conversion of regular CX-3 cards to Pro and didn't have any success. The cards don't initialize on boot, you have to use the flash not present jumper to boot them in recovery mode to see them at all.
     
    #49
  10. tommybackeast

    tommybackeast Member

    Joined:
    Jun 10, 2018
    Messages:
    95
    Likes Received:
    23
    I have a Dell T620; with iDRAC 7 Enterprise ; with the Mellanox 10/40 card in it (using 10GB back to 10GB on Brocade 6450-24p)

    Edit: what do those RACADM commands do exactly?
     
    #50
  11. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    33
    Likes Received:
    4
    test on windows further today,using one mellanox dac cable connect two port , flash to qcbt and change to eth mode both port work, but what strange are on ib or vpi mode not work (opensm started but always said port down).
    flash to mellanox fcbt or compiled firwmware only one port work

    seems card is ok on qcbt firmware but don’t know why it not work on ib or vpi mode, maybe casused by one dac connect two port?
    i am total confusing , the dac i used is mc2207130-001 .
    now i have two questions:
    1 why it can work under qcbt firmware on eth but not on ib or vpi mode
    2 why this card only one port work on fcbt firmware ,the other two card have same revision both port work
     
    #51
    Last edited: Sep 15, 2018
  12. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    33
    Likes Received:
    4
    more test , use two card and same dac to test .
    the bad card only work on eth mode both port (qcbt firmware) on 10gb, on ib mode only one port work (qcbt firmware) on 40gb.
    the bad card(fcbt firmware) work only one port on eth or ib mode on 56gb.

    maybe bad lucky , this is a bad card? one port only work on eth mode?
     
    #52
  13. TeleFragger

    TeleFragger Member

    Joined:
    Oct 26, 2016
    Messages:
    38
    Likes Received:
    2
    I got an OLD hp branded connect-x card to flash. HP only went to 2.8 for firmware and I force flashed 2.9.1000 and got it working in esxi 6.7, freenas 11.1, and windows 10. this card is the old cx4 port (I think)… and would love to know what switch works with these... but going to try and do pier to pier for now...
     
    #53

Share This Page