Help with old hp connectx cx4 style cards

[TeleFragger]

Ok so I have the following:
esxi 6.7 - 1
esxi 6.7 - 2 (well not yet but going to build)
freenas - so I can do vmotion
gaming rig
server 2016

now I have 10gb ib cards (old hp connect cx4 style) in each machine and got them working in a pier to pier mode. The only problem I have run into on these cards is I can only get the top port to activate. I get link lights with the cable attached but if I attempt to use any of the bottom ports, I do not get any link lights. Interestingly enough every O/S sees the 2nd port. states its there, sees the MAC, and all info but I cant get a dang activity light.

I have flashed these with HP firmware up to 2.8, and when that didnt fix it, force flashed to Mellanox 2.9.1000 and that worked. The software reads all of the cards, sees all the ports, sees info on the ports...

Example on my windows server...I didn't force this to 2.9.1000
​

C:\Users\Jeff>ibstat
CA 'ibv_device0'
        CA type:
        Number of ports: 2
        Firmware version: 2.8.0
        Hardware version: 0xa0
        Node GUID: 0x001635ffffbf0bb8
        System image GUID: 0x001635ffffbf0bbb
    Port 1:
        State: Down
        Physical state: Disabled
        Rate: 1
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x80500000
        Port GUID: 0x021635fffebf0bb9
        Link layer: Ethernet
        Transport: RoCE v1.0
    Port 2:
        State: Down
        Physical state: Disabled
        Rate: 1
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x80500000
        Port GUID: 0x021635fffebf0bba
        Link layer: Ethernet
        Transport: RoCE v1.0
CA 'ibv_device1'
        CA type:
        Number of ports: 2
        Firmware version: 2.8.0
        Hardware version: 0xa0
        Node GUID: 0x001635ffffbf0ba0
        System image GUID: 0x001635ffffbf0ba3
    Port 1:
        State: Active
        Physical state: LinkUp
        Rate: 10
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x80500000
        Port GUID: 0x021635fffebf0ba1
        Link layer: Ethernet
        Transport: RoCE v1.0
    Port 2:
        State: Down
        Physical state: Disabled
        Rate: 1
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x80500000
        Port GUID: 0x021635fffebf0ba2
        Link layer: Ethernet
        Transport: RoCE v1.0



C:\Program Files\Mellanox\WinMFT>mst status
MST devices:
------------
  mt26418_pci_cr0
  mt26418_pciconf0
  mt26418_pci_cr1
  mt26418_pciconf1


C:\Program Files\Mellanox\WinMFT>mlxfwmanager.exe
Querying Mellanox devices firmware ...
Device #1:
----------
  Device:        0000:04:00.0
  Part Number:
  Description:
  PSID:          HP_09D0000008
  Versions:      Current        Available
     FW          2.8.0000       N/A
  Status:        No matching image found
Device #2:
----------
  Device:        0000:06:00.0
  Part Number:
  Description:
  PSID:          HP_09D0000008
  Versions:      Current        Available
     FW          2.8.0000       N/A
  Status:        No matching image found

​

 

[TeleFragger]

would love how to get my hands on a fw that is 2.9.1200 or up as I heard they are faster due to some settings... and not sure how to build my own firmware...
but like said.. 2.9.1000 went on..

 

[RageBone]

Well, CX4 like, rather seems to be, CX2.
And and it reads like those are dual port, and VPI, meaning infiniband and Ethernet.

I suspect, those are mhqh29b s, right?
There are tables out there, relating HP part numbers to Mellanox ones.

Good news in terms of firmware,
I have 2.10.720 and 2.9.1200 as the build it yourself kit saved, I think I got them from somewhere that was linked on here, but I can't remember.


Now to your problem, those cards, mhqh29b can work in auto, ib or ETH mode.

If both are set to auto, I think the default is IB but link detection can be problematic.

I had Hughe problems with ipxe not getting a link since both were on auto and didn't manage to work together.
Setting one to ib or ETH fixed it in my case.

Modes can also be programmed into the FW, so the two Woking ports are maybe, luckily configured right.

 

[TeleFragger]

I set them both to ETH in win10 when I was testing and could not get them to get port 2 to work. port 1 works no problem.

and can you get me a link to the firmware? I'm on 2.8 on most.. did a non hp force flash to 2.9.1000 and would love 2.10.720... hell I hear even 2.9.1200 is faster than the 2.9.1000

 

[TeleFragger]

also just bought a procurve off ebay. accepted my $55 shipped offer and will be waiting for it so 2nd port is no big deal now.
as I can just do 1 card in each machine and 1 port each...

 

[TeleFragger]

Well, CX4 like, rather seems to be, CX2.
And and it reads like those are dual port, and VPI, meaning infiniband and Ethernet.

I suspect, those are mhqh29b s, right?
There are tables out there, relating HP part numbers to Mellanox ones.

Good news in terms of firmware,
I have 2.10.720 and 2.9.1200 as the build it yourself kit saved, I think I got them from somewhere that was linked on here, but I can't remember.

nope.. mine is
MHGH29-XTC

tried putting a ConnectX-2 fw on it and
-E- Burning FS2 image failed: Device/Image mismatch: FW image file cannot be programmed to device ConnectX, it is intended for: ConnectX-2 only

so this is a ConnectX-1 card.. with the cx4 style plug
I have also seen them named SFF-8470-Infiniband-CX4-Connectors

Google image..
top is cable
bottom is dual port card

 

[nev_neo]

How did you get them to work with ESXi 6.7 ??
I have a whole bunch of those and they only worked with esx 5.5 or something really old.
I had to dump the whole lot of them cables and switch and move to the 10g sfp+ kits.

 

[TeleFragger]

How did you get them to work with ESXi 6.7 ??
I have a whole bunch of those and they only worked with esx 5.5 or something really old.
I had to dump the whole lot of them cables and switch and move to the 10g sfp+ kits.

im a dummy on this stuff and been screwin with it..
they work in

windows 10
esxi 6.7
freenas 11.1

all I did was put them in a win10 box and flash them.. hell I just got rid of the HPW fw and flashed them Mellanox 2.9.1000 just fine.. cant go any higher..

find yours and help me get to 2.9.1200 or up as it supports dsrm or whatever as that is even faster but dude these things are FAST..

here is a video and I explain it

but in esxi.. it wont use them as InfiniBand - not supported but I DONT CARE!!! I Want ethernet!!!!
and by default automagically does it.. hell I had my esxi already up, put in flashed card and boom it saw it

see here

**********************edited
had to put this here... if not ok mods remove..
a guy on hardforum was my ear at this.. he gave some thought but kept hearing me and gave me a reason to keep pursuing as I replied he replied and I kept googling, chugging along..
starting to play with 10gb - Mellanox and need help

 

[TeleFragger]

Procurve is in.. WOOT..
I can ping between 2 machines!!!!!!!! SHAWEET!!!! forget the bottom port now.. ive got 6 ports on switch and 5 machines to use!!!!