HP/PMC SA in whitebox

chinesestunna

Active Member
Jan 23, 2015
562
140
43
53
I wanted to start a thread as a compilation of Servers/motherboards and cards tested
HP SA cards are very fast and stable but often have issues dealing with non-HP boards.
While this will change in the future with PMC licensing the SA brand this is the present we live in.

For those that have used HP cards in white boxes if you would list the configuration and any tweaks required to make it work.

I have a P222 running in an Asus Z9PE-D16/2L and it required turning off Option rom prompting in the bios. Otherwise it would hang on drive initialization.


Of note... for those that have not tried a SA card in a whitebox but would like to...
I have a few spare p420's that you can play with.
Patriot,

I'd be open to experimenting with one of these cards, I have a SuperMicro X8DTE setup I can try and make work
 

5teve

Active Member
Jan 23, 2015
106
35
28
45
Perth, Australia
Well after rounding up (nearly) all the hardware i need I can finally contribute toward this thread :)

System is as follows:-

Asrock C226ws running firmware 2.30
Intel Xeon E3 1280v3
32gb DDR3 1333 Ram
Gigabyte R7 25x Graphics Card
Corsair c600 - 600w psu
Fractal Design R2 XL define case
2x Sandisk Extreme Pro 256gb SSD in raid 0
12x Hitachi Ultrastar 2tb 7k3000 drives

Attempting to install the following:-

1x HP P822 with 2gb FBWC and supercap firmware 3.22
1x HP P420 with 2gb FBWC and supercap firmware 3.22
1x HP P420 with 2gb FBWC and supercap firmware 3.22

While i'm at it i'm also checking (on the above system and in a N40L microserver):-

HP NC328T 2port gigabit NIC
HP NC328T 4port gigabit NIC

Things will be a bit random and over a few posts, so apologies in advance.

HP822 (fw 3.22) - connected to 8x drives on internal ports only
installed into PCIE 1 on the motherboard (I read the manual wrong and this is a 4x slot of 16x size and is PCIE2.0 - I'll post PCI 3.0 results in a while) - Turned PC on - entered bios and went to boot > CSM (compatibility support module) > changed options to legacy - rebooted - Card posted its initialisation - finished initialisation - then blank screen no system bios screen.

Entered Bios and changed CSM back to uefi only restarted and system booted into windows 8.1 (new fresh install - no additional drivers added yet) , with no HP initialisation pre system bios screen. Windows installed drivers straight away with no internet connection.

Installed HP SMART Storage Administrator (server 2012 version) and it saw the card no problem. Created a Raid 6 array - no issues there.

Updated the Bios to 6.34 (how easy have HP made this to do!) rebooted > changed system bios CSM to legacy again. Rebooted - Card Initialised - then proceeded to boot into windows.. no blank screen this time.

Rebooted to try and get into ORCA but no prompt to enter, tried the usual F8 etc but nothing worked. Returned system bios back to uefi to reduce boot times.

Conclusion - With newer bios the HP P822 can boot on the c226ws board with no bios tweaks - but ORCA cant be entered (that i have found). Windows 8.1 has default drivers built in and Array creation was no trouble in HP SMART Storage Administrator - Bios updates from older 3.22 works perfectly

HP420 2gb FBWC (fw 3.22) - connected to 4x drives on internal ports only
installed into PCIE 1 on the motherboard (I read the manual wrong and this is a 4x slot of 16x size and is PCIE2.0 - I'll post PCI 3.0 results in a while) - Turned PC on - entered bios and went to boot > CSM (compatibility support module) > changed options to legacy - rebooted - Card posted its initialisation - finished initialisation - then blank screen no system bios screen.

Entered Bios and changed CSM back to uefi only restarted and system booted into windows 8.1 (new fresh install - no additional drivers added yet) , with no HP initialisation pre system bios screen. Windows installed drivers straight away with no internet connection.

Installed HP SMART Storage Administrator (server 2012 version) and it saw the card no problem. Created a Raid 6 array - no issues there.

Updated the Bios to 6.34 (how easy have HP made this to do!) rebooted > changed system bios CSM to legacy again. Rebooted - Card Initialised - then proceeded to boot into windows.. no blank screen this time.

Rebooted to try and get into ORCA but no prompt to enter, tried the usual F8 etc but nothing worked. Returned system bios back to uefi to reduce boot times.

Conclusion - With newer bios the HP P420 can boot on the c226ws board with no bios tweaks - but ORCA cant be entered (that i have found). Windows 8.1 has default drivers built in and Array creation was no trouble in HP SMART Storage Administrator - Bios updates from older 3.22 works perfectly

HP420 1gb FBWC (fw 3.42)

Same as the 2gb version - firmware update again was no issue to 6.34

With both cards no temps are reported in HP SMART Storage Administrator using HP Drivers

Performance is next...
 
Last edited:

5teve

Active Member
Jan 23, 2015
106
35
28
45
Perth, Australia
Tests carried out on drive with 1tb written..

P822 2gb FBWC in correct PCIE slot (8x PCIE 3.0 - on bus with Graphics card) Cache set to 10% read 90% write


Atto Disk Benchmark - 8 Drives - 2tb Hitachi Ultrastar 7k3000 in raid 6 array - 256kb strip size

default setting except total length set to 2gb (max allowed)




P420 2gb FBWC in correct PCIE slot (8x PCIE 3.0 - on bus with Graphics card) Cache set to 10% read 90% write

Atto Disk Benchmark - 8 Drives - 2tb Hitachi Ultrastar 7k3000 in raid 6 array - 256kb strip size

default setting except total length set to 2gb (max allowed)




P420 1gb FBWC in correct PCIE slot (8x PCIE 3.0 - on bus with Graphics card) Cache set to 10% read 90% write

Atto Disk Benchmark - 8 Drives - 2tb Hitachi Ultrastar 7k3000 in raid 6 array - 256kb strip size

default setting except total length set to 2gb (max allowed)



looks like the cache certainly helps!

Any suggestions for other benchmarks? Tweaks? anything anyone wants to see... ?

12 disk test on the p822 will be posted once my cables arrive. I'll try the network cards out shortly too..

Steve
 

5teve

Active Member
Jan 23, 2015
106
35
28
45
Perth, Australia
Off topic ... but it is HP...

Network Cards...

the HP NC328T 2port gigabit NIC installs (default windows drivers) and is seen - I dont have any way of testing currently - but once i have new lines in i'll test in the microserver.
The HP NC329T 4port gigabit NIC wont fit in the bottom slot due to the front panel connector and the width of the gigabit connector block on the card. Its due to go in a ML350 g6 shortly tho

Steve
 
Last edited:

5teve

Active Member
Jan 23, 2015
106
35
28
45
Perth, Australia
Further update.. after getting controller lockups I have had to add a fan to the HP P822 - This is basically an 80x25mm fan currently cable tied to a pci blanking plate blowing directly on the card heatsink, jerry rigged like this to prove heat was the issue.. I'll be looking for a 60 or 70mm x10 or 15mm fan for the final solution..

If you dont have sufficient airflow in your case you will need to add an active cooling solution.

Steve
 

JohnRB

New Member
Feb 11, 2017
7
1
3
40
New York, NYC
www.cakesbycoley.com
I currently have a few P840's and use in Asus Z10 and Supermicro X11 boards. I love HP SA cards for many reasons but I always build servers from the ground up. The only thing that stinks with the P840 is I cant connect it to a battery because that series connects the Cache card and the Battery into the "Proliant" motherboard and not to each other. For me this is not an issue because my servers are on big APCs. I have been trying to figure some kind of workaround for a while, but since it is not "Needed" I have kind of given up the pursuit. The previous generation SA cache card has the exact specs as mine and looks identical but I doubt it would work in the P840 if it did problem solved because those directly connect cache to battery.
So obviously it's not the same having a SA card in a Proliant server vs other board/build, but I am still very happy with the setup and performance. HP has also gotten a bit better maintaining there linux repo.
 

Patriot

Moderator
Apr 18, 2011
1,311
695
113
How risky do you want to get? you could cross flash a p830 to p840 and use the p830 cache... or just try the cache first.
 

JohnRB

New Member
Feb 11, 2017
7
1
3
40
New York, NYC
www.cakesbycoley.com
Yea I have been on the hunt for the cache card, its worth a shot. Don't know about trying to flash the older her fw. Doubt you can even do it (or is your statement telling me I can?). Building new server so may not even use the p840 anymore, but if I get the workaround to take I may.
 

BLinux

cat lover server enthusiast
Jul 7, 2016
2,528
975
113
artofserver.com
So, i'm pretty new to the HP SSA stuff of recent years. I use to work in a data center full of Compaq Proliant servers (and the fact they were still "Compaq" should date how long ago that was), but recently trying to get some HP H240 HBAs to work in Supermicro servers, so here's what I have so far:

HP H240 in Supermicro X9DRi-LN4F+ (works)
HP H240 in Supermicro X8DTH-iF (works)
HP H240 connected to BPN-SAS-216A (works)
HP H240 connected to BPN-SAS-836TQ (works)

The above works with or without option ROM disabled on the PCI-E slot where the cards are installed (I have 3 of them). On the other end of things, also seems to work with the BPN-SAS-216A backplane. The drives connected are HP branded HGST HUSMM 800GB/400GB SAS3 SSDs and they all seem to connect with a few quirks (more later). However, I've also tested these cards in a Supermicro 836 w/ X8DTH-iF using non-HP 3.5" HDDs and that seems to work fine too.

Now, a few questions and quirks I've noticed:

1) With option ROM enabled, I see the HP H240 initializing during POST. Each H240 scans for devices, one after another, and then continues to boot. However, it never shows any message about a key sequence to press to get into the BIOS ROM setup. I've been configuring the card in a Linux OS using the ssacli tool. I would like to access the BIOS ROM too. So, question: is there a key sequence to press that will get me into the BIOS ROM on the H240 card? if so, what is it?

2) If I boot the H240 card with the SSDs or HDDs unplugged from the above backplanes, and then plug them in after the system has booted, the drives will not get recognized. However, if I boot the H240 with the SSDs or HDDs connected, they are recognized and are available in the OS. At which point, I can hot unplug them and plug them back in several minutes later, and the drive disconnects and re-connects as expected. I've never seen this type of behavior with LSI based controllers. Is this typical? Is there a setting to change this behavior? If so, what is it and how do I do it?

3) I have 1 H240 card that shows during POST, but disables itself once booted into Linux. At which point, it says:

"CACHE STATUS PROBLEM DETECTED: The controller is disabled because the cache
module is not attached. Please re-attach the
cache module to re-enable the controller."

I don't really know what this means, as my understanding is that the H240 doesn't have cache module option? Am I wrong about this? Or, as I was reading the ssacli user guide, I wonder if there was a previous configuration (the card is in RAID mode, not HBA mode and I can't change it due to above error) had configured an SSD as cache drive and now that drive is disconnected? Either way, is there a way for me to get the card to work, perhaps clearing previous RAID configuration? force override?

4) One more quirk is that all H240 cards show as being in slot=255. I don't know why, and obviously with multiple cards this gets tricky trying to run ssacli commands like:

> controller slot=255 show detail

It will error out. Instead, I had to address the individual cards by SN:

> controller sn="<serial number>" show detail
 
Last edited:

Patriot

Moderator
Apr 18, 2011
1,311
695
113
Don't know about the slot issue, as far as the roms loading... they are working on a uefi application but currently this is the main way to configure drives, other than an ilo interface for simple setup. You need this. Offline HPSSA . There is a small embedded cache, not sure why it is giving an error.
 

nthu9280

Well-Known Member
Feb 3, 2016
1,588
441
83
San Antonio, TX
Don't know about the slot issue, as far as the roms loading... they are working on a uefi application but currently this is the main way to configure drives, other than an ilo interface for simple setup. You need this. Offline HPSSA . There is a small embedded cache, not sure why it is giving an error.
There is a more recent version from the link you provided. Is there a specific to use the older version? For example need to use Sas2Flash P14 or before to bypass the MfgPage mismatch error on LSI cards.

Drivers & Software - HPE Support Center.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Mega-dump about setting up an H240 in HBA mode further down:

3) I have 1 H240 card that shows during POST, but disables itself once booted into Linux. At which point, it says:

"CACHE STATUS PROBLEM DETECTED: The controller is disabled because the cache
module is not attached. Please re-attach the
cache module to re-enable the controller."

I don't really know what this means, as my understanding is that the H240 doesn't have cache module option? Am I wrong about this? Or, as I was reading the ssacli user guide, I wonder if there was a previous configuration (the card is in RAID mode, not HBA mode and I can't change it due to above error) had configured an SSD as cache drive and now that drive is disconnected? Either way, is there a way for me to get the card to work, perhaps clearing previous RAID configuration? force override?
This turned out to be a faulty card right? I think I remember reading that in another thread earlier tonight. Mine complains about the lack of cache and a super-cap but then proceeds to work fine.

4) One more quirk is that all H240 cards show as being in slot=255. I don't know why, and obviously with multiple cards this gets tricky trying to run ssacli commands like:

> controller slot=255 show detail

It will error out. Instead, I had to address the individual cards by SN:

> controller sn="<serial number>" show detail
When I dropped my H240 into my DL380p gen8 in pcie slot 2 it showed as "Slot=2", I think the slot numbering gets weird on non-HP servers because it's looking for HP's actual marked slot numbers in some ROM table included in their BIOS. I'm betting 255 is just what the driver reports if it can't look up the slot number in that missing lookup table in ROM. When I popped the card into one of my T730's it reported "Slot=255" like yours.

Also, at least on firmware 5.04(A) and onward there's no issue addressing the adapter by "slot=255" or whatever it decides to report back in a whitebox.

I had some other firmware issues with my H240 that I've mostly solved but might still poke at again. I'll describe them in case anyone else manages to screw the same things up (bear with me, while the DL380p _is_ an HP box it's a completely unsupported use-case to use gen9 hardware in a gen8 machine so it might as well be whitebox for the purpose of demonstration.):

Test machine: Ubuntu 18.04 Bionic, HWE kernel 4.18.n

My H240 arrived with firmware 5.04 and seemed to function fine in my gen8 DL380p on first boot: the option rom popped up during boot and the hpsa driver loaded properly in Linux. I ran `ssacli ctrl all show` and `show detail` and everything looked good other than the card being stuck in RAID mode. I tried setting hbamode the same way I did previously on the integrated p420i with `ssacli ctrl slot=2 modify hbamode=on forced` but that failed with "Error: hbamode is not supported on this controller."

What I learned later is that I should have run the following:
Code:
(where <target> == "ctrl slot=N") and N is the adapter's slot location

<target> modify clearconfigdata
<target> modify clearnvram
<target> modify bootvolume=clearprimary
<target> modify bootvolume=clearsecondary
<target> modify raidmode=off
I then (stupidly, in retrospect) updated the firmware to the current release (6.88) thinking there may be issues with the older firmware preventing me from setting HBA mode. The flash worked fine but after rebooting the adapter hung for a solid 2-3 minutes during POST, the option rom didn't display and when it finally booted back into Linux the driver reported
Code:
*[  132.760072] hpsa 0000:07:00.0: board not ready, timed out."
The board refused to initialize and wasn't available in ssacli. I tried rebooting a couple of times (with a cold boot as well) to see if it just needed a power cycle to sort itself out but had no success.

The firmware tools *could* see the board, however, so it wasn't bricked but with post times up to around 4 minutes I pulled it out of the server and dropped it into one of my T730 thin clients to attempt recovery. In the thin client the board posted immediately and the drivers came up as expected. So I reflashed 6.88 just in case, popped it back into the server and had the same experience all over again for another 20 minutes.

I put the adapter back into my recovery machine and this time it hung during POST and the driver refusing to initialize. Out of frustration I shut down and pulled it thinking I'd try flashing it back to 5.04 in my desktop; then I thought better and popped it back into the thin client one last time and it booted perfectly (HP, whyyy.)

So at this point it's basically a roll of the dice whether the card is going to initialize properly during boot with firmware 6.88. I took the opportunity to downgrade back to 5.04(A) and everything is working perfectly again, the card posts and initializes perfectly in all of my machines.

At this point after dumping as many commands as I could from ssacli with "ssacli help <command>" I tried to get the card out of raid mode again using `ssacli ctrl slot=N modify raidmode=off` and was finally successful at setting hba mode. (Note: You don't set "hbamode=on" on these boards, you set "raidmode=off".) I went ahead and ran `modify clearconfigdata` and `modify clearnvram` and unset the stale primary boot volume from the last owner then shut everything down, reinstalled the board in my gen8 machine and moved all of my bays from the onboard controller to the H240.

At this point everything is working flawlessly, I'm slamming 6 SSDs with badblocks right now to see if there are any stability issues but so far (an hour and a half later) everything is still humming along perfectly.

A lot of this was done on an HP server but it's such an unsupported configuration that it might as well be whitebox as far as the hardware is concerned.

After I finish running some performance comparisons against the onboard p420i I think I'm going to go back and see if I can get firmware 6.88 working. At the very least I should be able to bisect the updates and figure out exactly where everything goes to shit.
 
Last edited:
  • Like
Reactions: metag

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
@arglebargle thanks for sharing. by the way, did you run into issue #2 I mentioned above?
I actually never booted the machine without drives in the bays, I need to flip-flop my SAS cables on the drive bay later so I'll leave a few unplugged during boot and check.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
@BLinux

lsscsi at boot with two drives unplugged:
Code:
[2:0:0:0]    disk    HP iLO   Internal SD-CARD 2.10  /dev/sdg
[3:0:0:0]    storage HP       H240             5.04  -      
[3:0:1:0]    disk    ATA      Samsung SSD 850  4B6Q  /dev/sda
[3:0:2:0]    disk    ATA      ST5000LM000-2AN1 0001  /dev/sdb
[3:0:3:0]    disk    ATA      ST5000LM000-2AN1 0001  /dev/sdc
[3:0:4:0]    disk    IBM-SSG  ASQX400          2B29  /dev/sdd
[3:0:5:0]    disk    ATA      Samsung SSD 850  4B6Q  /dev/sde
[3:0:6:0]    disk    SmrtStor TXA2D20400GA6XYR KZ40  /dev/sdf
[3:0:7:0]    enclosu HP       H240             5.04  -

lsscsi after inserting the two drives:
Code:
[2:0:0:0]    disk    HP iLO   Internal SD-CARD 2.10  /dev/sdg
[3:0:0:0]    storage HP       H240             5.04  -      
[3:0:1:0]    disk    ATA      Samsung SSD 850  4B6Q  /dev/sda
[3:0:2:0]    disk    ATA      ST5000LM000-2AN1 0001  /dev/sdb
[3:0:3:0]    disk    ATA      ST5000LM000-2AN1 0001  /dev/sdc
[3:0:4:0]    disk    IBM-SSG  ASQX400          2B29  /dev/sdd
[3:0:5:0]    disk    ATA      Samsung SSD 850  4B6Q  /dev/sde
[3:0:6:0]    disk    SmrtStor TXA2D20400GA6XYR KZ40  /dev/sdf
[3:0:7:0]    enclosu HP       H240             5.04  -      
[3:0:8:0]    disk    ATA      ST5000LM000-2AN1 0001  /dev/sdh
[3:0:9:0]    disk    IBM-SSG  ASQX400          2B29  /dev/sdi

I gave the HBA a kick with `ssacli ctrl all rescan` after that for good measure, I'm not sure I actually needed to. It seems like everything is behaving as expected.

This is on Ubuntu 18.04, I'm running the HWE kernel but everything was working properly with the standard kernel a couple of days ago w.r.t. hotplugging while using the onboard p420i.
 

BLinux

cat lover server enthusiast
Jul 7, 2016
2,528
975
113
artofserver.com
@arglebargle was that done on a HP machine or non-HP machine? I had problem #2 on non-HP machine; I believe it should be without issue on HP machine. (i'm asking because that looks like a HP machine? I see the iLO in the listing)
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
@arglebargle was that done on a HP machine or non-HP machine? I had problem #2 on non-HP machine; I believe it should be without issue on HP machine. (i'm asking because that looks like a HP machine? I see the iLO in the listing)
Ah, yeah, it is an HP machine, my bad there. I can check it out on a bench machine after I get some space cleaned up this week. Hot swap is part of the SAS spec right? It's not a backplane/cage specific thing? I don't have anything with hotswap bays besides the HP gen8 box but I do have SFF-8087 to SFF-8482 breakout cables that I could hotplug with on an open bench.
 
Last edited:

NowhereRx7

New Member
May 4, 2020
1
0
1
SUCCESS:
I have enabled the ROM and completed POST with a P400, P410, and P420, all running newest firmware, on a Supermicro H8DME8-2. One of the systems is even booting from the P410.
There is no prompt for the config environment during boot - at least not that I have noticed - and HP Offline ACU has to be used, but works just fine.

I wondered if this had anything to do with the boards having IPMI cards on them, so I removed the card and tested; still booted. I am still wondering if the fact that the board has IPMI support has something to do with the ROM working on them.

I also performed an upgrade (two actually) from P400 to P410 and did get the prompts to import the foreign config. It's been a while since I did that, so I don't remember many details. I have another system that I plan on upgrading the P410 to P420 and will use the Supermicro board to boot the ROM and import the config; maybe I'll remember to post some more details then.

I hope this info helps someone out, even if it is an older Socket F board.

UNSUCCESSFUL:
I however did not have success with a very similar, also Socket F motherboard, the SuperMicro H8DIi+. This board does not have IPMI and does not support IPMI add-on cards, unfortunately for me; the board must be purchased with the IPMI option. I only mention this, because as I stated above, I'm really wondering how much this has to do with the success of the other board.