Looking for HP z840 workstation users with Infiniband network cards

mokurum

New Member
Dec 20, 2021
9
1
3
Hi all, I have a couple z840 workstations that I'd like to connect with Mellanox ConnectX-3 VPI 40Gb network cards. These are PCIe gen3 cards that the mobo on the z840 supports (I have tested the PCI slots with two graphics cards and the machines boot fine).

When I install the network card though, I get a 6 beep bios error. Card is not being recognized. HP z840 quickspecs pg16 does not list any network card faster than 10Gb so the cards I am trying are not listed. I called the HP support and they flat out say: "if it's not on the quickspecs, it is not supported".

Despite this Autodesk has a guide on how to install a mellanox card on a HP z840 (which also did not work for me).

I feel like there has to be a BIOS hack or something that would allow me to use the network cards.

Any idea, hack around this would be very much appreciated.
 

LodeRunner

Active Member
Apr 27, 2019
383
158
43
I cannot find the thread for some reason, but wasn't there a potential issue with these cards on some motherboards similar to the SMBus issue with LSI HBA/RAID controllers? Taping pins 5 and 6 was the fix for those cards. Really can't recall if that trick is applicable to the CX3s.
 

mokurum

New Member
Dec 20, 2021
9
1
3
Man I would really really appreciate if you can elaborate on this - I feel like there is no reason for the cards to not work! I will google the "taping pins5 and 6" solution.
 

LodeRunner

Active Member
Apr 27, 2019
383
158
43
Something to do with the SMBus signaling and the 3.3v pins on some cards on certain boards. Like using Dell-branded LSI controllers in non-Dell systems would sometimes result in non-booting systems or the card not being detected; tape over the 3.3v pins and the problem goes away. I think the CX3s can suffer from the same issue.

SuperMicro for example had an no-boot issue with CX3s for which SM released a BIOS update to fix.
 

mokurum

New Member
Dec 20, 2021
9
1
3
This is the Mellanox product description it says here in section 2.2. Power
All adapter cards receive 12V and 3.3V power from the PCI Express edge connector. All other required power voltages are generated by on-board switch mode regulators. See “Specifications” on page 39

Edit: that's nice of SM - I am using the latest BIOS available from HP for z840 in 2018 (M60 v.02.47)
 

mokurum

New Member
Dec 20, 2021
9
1
3
From @BLinux's channel

CX3 works in my Z620. I don't have a RAID controller installed though.
Thanks for taking the time and sharing the video! I tried blocking the b5+b6 with tape and put the card into slot 2 and moved the gpu to slot 7 and still the same issue. This is the only combination I tried bc the tape gets "ripped" pulling the card out and leaves a bit of residue inside the slot which I successfully cleaned (phew).

some questions:
Did you have to do this hack to make your CX3 work on Z620?
Do you mind letting me know your physical setup (which card is installed in which slot?)
Any bios changes ?
 

klui

Active Member
Feb 3, 2019
545
240
43
Slot 5. I didn't need to do anything to it nor did I need to use some special BIOS.

6 beeps is associated with graphics card--see Z840 maintenance and service guide. What made you think it's because of the NIC? My guess is there's not enough power or graphics cards need to be reseated after cleaning slots.

Troubleshooting 101: remove all cards and disconnect all drives. Insert your CX3 and see if it behaves the same, then insert your other cards one at a time.
 

mokurum

New Member
Dec 20, 2021
9
1
3
Slot 5. I didn't need to do anything to it nor did I need to use some special BIOS.

6 beeps is associated with graphics card--see Z840 maintenance and service guide. What made you think it's because of the NIC? My guess is there's not enough power or graphics cards need to be reseated after cleaning slots.

Troubleshooting 101: remove all cards and disconnect all drives. Insert your CX3 and see if it behaves the same, then insert your other cards one at a time.
Yep aware of the 6 beeps' meaning from the maintenance guide.
What made me think it's the NIC is that I installed a second GPU and the machine boots fine. Only when I insert the NIC it starts beeping again.
And I did remove every single card from the system (even the HP Z M.2 PCI hard drive - booting from USB) and tried installing the network on every possible slot.
 

mokurum

New Member
Dec 20, 2021
9
1
3
Insert into a non-Z840 and see if it has the latest firmware and update if old.
Thanks for all the suggestions everyone. Solved.
@klui thanks for pointing out the obvious thing to do.

Inserted the cards on a non-Z840 machine. Non-Z840 pc booted fine.
Flushed the firmware on the card and burned the latest from Firmware for ConnectX®-3 IB/VPI (mellanox.com)
Put it back to z840 - Boots fine and I can see the card listed with lspci
 
  • Like
Reactions: klui