Help with old hp connectx cx4 style cards

Discussion in 'Networking' started by TeleFragger, Jan 9, 2019.

  1. TeleFragger

    TeleFragger Member

    Joined:
    Oct 26, 2016
    Messages:
    85
    Likes Received:
    9
    Ok so I have the following:
    esxi 6.7 - 1
    esxi 6.7 - 2 (well not yet but going to build)
    freenas - so I can do vmotion
    gaming rig
    server 2016

    now I have 10gb ib cards (old hp connect cx4 style) in each machine and got them working in a pier to pier mode. The only problem I have run into on these cards is I can only get the top port to activate. I get link lights with the cable attached but if I attempt to use any of the bottom ports, I do not get any link lights. Interestingly enough every O/S sees the 2nd port. states its there, sees the MAC, and all info but I cant get a dang activity light.

    I have flashed these with HP firmware up to 2.8, and when that didnt fix it, force flashed to Mellanox 2.9.1000 and that worked. The software reads all of the cards, sees all the ports, sees info on the ports...

    Example on my windows server...I didn't force this to 2.9.1000
    Code:
    C:\Users\Jeff>ibstat
    CA 'ibv_device0'
            CA type:
            Number of ports: 2
            Firmware version: 2.8.0
            Hardware version: 0xa0
            Node GUID: 0x001635ffffbf0bb8
            System image GUID: 0x001635ffffbf0bbb
        Port 1:
            State: Down
            Physical state: Disabled
            Rate: 1
            Base lid: 0
            LMC: 0
            SM lid: 0
            Capability mask: 0x80500000
            Port GUID: 0x021635fffebf0bb9
            Link layer: Ethernet
            Transport: RoCE v1.0
        Port 2:
            State: Down
            Physical state: Disabled
            Rate: 1
            Base lid: 0
            LMC: 0
            SM lid: 0
            Capability mask: 0x80500000
            Port GUID: 0x021635fffebf0bba
            Link layer: Ethernet
            Transport: RoCE v1.0
    CA 'ibv_device1'
            CA type:
            Number of ports: 2
            Firmware version: 2.8.0
            Hardware version: 0xa0
            Node GUID: 0x001635ffffbf0ba0
            System image GUID: 0x001635ffffbf0ba3
        Port 1:
            State: Active
            Physical state: LinkUp
            Rate: 10
            Base lid: 0
            LMC: 0
            SM lid: 0
            Capability mask: 0x80500000
            Port GUID: 0x021635fffebf0ba1
            Link layer: Ethernet
            Transport: RoCE v1.0
        Port 2:
            State: Down
            Physical state: Disabled
            Rate: 1
            Base lid: 0
            LMC: 0
            SM lid: 0
            Capability mask: 0x80500000
            Port GUID: 0x021635fffebf0ba2
            Link layer: Ethernet
            Transport: RoCE v1.0
    
    
    
    C:\Program Files\Mellanox\WinMFT>mst status
    MST devices:
    ------------
      mt26418_pci_cr0
      mt26418_pciconf0
      mt26418_pci_cr1
      mt26418_pciconf1
    
    
    C:\Program Files\Mellanox\WinMFT>mlxfwmanager.exe
    Querying Mellanox devices firmware ...
    Device #1:
    ----------
      Device:        0000:04:00.0
      Part Number:
      Description:
      PSID:          HP_09D0000008
      Versions:      Current        Available
         FW          2.8.0000       N/A
      Status:        No matching image found
    Device #2:
    ----------
      Device:        0000:06:00.0
      Part Number:
      Description:
      PSID:          HP_09D0000008
      Versions:      Current        Available
         FW          2.8.0000       N/A
      Status:        No matching image found
    
    
    
     

    Attached Files:

    • 1gb.PNG
      1gb.PNG
      File size:
      106 KB
      Views:
      3
    • 10g.PNG
      10g.PNG
      File size:
      122.8 KB
      Views:
      3
    #1
  2. TeleFragger

    TeleFragger Member

    Joined:
    Oct 26, 2016
    Messages:
    85
    Likes Received:
    9
    would love how to get my hands on a fw that is 2.9.1200 or up as I heard they are faster due to some settings... and not sure how to build my own firmware...
    but like said.. 2.9.1000 went on..
     

    Attached Files:

    #2
  3. RageBone

    RageBone Member

    Joined:
    Jul 11, 2017
    Messages:
    139
    Likes Received:
    22
    Well, CX4 like, rather seems to be, CX2.
    And and it reads like those are dual port, and VPI, meaning infiniband and Ethernet.

    I suspect, those are mhqh29b s, right?
    There are tables out there, relating HP part numbers to Mellanox ones.

    Good news in terms of firmware,
    I have 2.10.720 and 2.9.1200 as the build it yourself kit saved, I think I got them from somewhere that was linked on here, but I can't remember.


    Now to your problem, those cards, mhqh29b can work in auto, ib or ETH mode.

    If both are set to auto, I think the default is IB but link detection can be problematic.

    I had Hughe problems with ipxe not getting a link since both were on auto and didn't manage to work together.
    Setting one to ib or ETH fixed it in my case.

    Modes can also be programmed into the FW, so the two Woking ports are maybe, luckily configured right.
     
    #3
  4. TeleFragger

    TeleFragger Member

    Joined:
    Oct 26, 2016
    Messages:
    85
    Likes Received:
    9
    I set them both to ETH in win10 when I was testing and could not get them to get port 2 to work. port 1 works no problem.

    and can you get me a link to the firmware? I'm on 2.8 on most.. did a non hp force flash to 2.9.1000 and would love 2.10.720... hell I hear even 2.9.1200 is faster than the 2.9.1000
     
    #4
  5. TeleFragger

    TeleFragger Member

    Joined:
    Oct 26, 2016
    Messages:
    85
    Likes Received:
    9
    also just bought a procurve off ebay. accepted my $55 shipped offer and will be waiting for it so 2nd port is no big deal now.
    as I can just do 1 card in each machine and 1 port each...
     
    #5
  6. TeleFragger

    TeleFragger Member

    Joined:
    Oct 26, 2016
    Messages:
    85
    Likes Received:
    9
    [​IMG]

    nope.. mine is
    MHGH29-XTC

    tried putting a ConnectX-2 fw on it and
    -E- Burning FS2 image failed: Device/Image mismatch: FW image file cannot be programmed to device ConnectX, it is intended for: ConnectX-2 only

    so this is a ConnectX-1 card.. with the cx4 style plug
    I have also seen them named SFF-8470-Infiniband-CX4-Connectors

    Google image..
    top is cable
    bottom is dual port card


    SFF-8470-Infiniband-CX4-Connectors.jpg
     
    #6
  7. nev_neo

    nev_neo Member

    Joined:
    Jul 31, 2013
    Messages:
    110
    Likes Received:
    17
    How did you get them to work with ESXi 6.7 ??
    I have a whole bunch of those and they only worked with esx 5.5 or something really old.
    I had to dump the whole lot of them cables and switch and move to the 10g sfp+ kits.
     
    #7
  8. TeleFragger

    TeleFragger Member

    Joined:
    Oct 26, 2016
    Messages:
    85
    Likes Received:
    9
    im a dummy on this stuff and been screwin with it..
    they work in

    windows 10
    esxi 6.7
    freenas 11.1

    all I did was put them in a win10 box and flash them.. hell I just got rid of the HPW fw and flashed them Mellanox 2.9.1000 just fine.. cant go any higher..

    find yours and help me get to 2.9.1200 or up as it supports dsrm or whatever as that is even faster but dude these things are FAST..

    here is a video and I explain it



    but in esxi.. it wont use them as InfiniBand - not supported but I DONT CARE!!! I Want ethernet!!!!
    and by default automagically does it.. hell I had my esxi already up, put in flashed card and boom it saw it

    see here







    **********************edited
    had to put this here... if not ok mods remove..
    a guy on hardforum was my ear at this.. he gave some thought but kept hearing me and gave me a reason to keep pursuing as I replied he replied and I kept googling, chugging along..
    starting to play with 10gb - Mellanox and need help
     
    #8
    nev_neo likes this.
  9. TeleFragger

    TeleFragger Member

    Joined:
    Oct 26, 2016
    Messages:
    85
    Likes Received:
    9
    Procurve is in.. WOOT..
    I can ping between 2 machines!!!!!!!! SHAWEET!!!! forget the bottom port now.. ive got 6 ports on switch and 5 machines to use!!!!
     
    #9

Share This Page