HP DL380 Gen 9 drive sled lights issues

[ziggygt]

I bought a DL380 G9 24+2 SFF disk shelf. HP ProLiant DL380 G9 24xSFF Server CTO 2 x Heatsink Smart Array 2 x 800W PSU | eBay. I purchased some third-party drive sleds and some Hitachi and Seagate drives. I am driving the rack with an HP Expander and a non-HP HBA. The lights in the HP sleds with HP drives seem to work correctly. The "third party" Sleds with Seagate drives have a strange behavior. The indicator lights spin constantly. In an HP sled (800GB)with a 1.2GB hitachi drive have similar behaviour. The HP drives in HP sleds work properly.

Key questions:

1. Does the activities of the lights affect drive performance.
2. Is the drive firmware truly the key to the proper use of the lights.
3. Best way to read SMART data on these drives?

With Windows, I can add all the drives, and they operate properly in Windows, but they do not appear in the crystal diskinfo to review the SMART data. My goal is to use these drives with TRUENAS Scale. I want to review the SMART data to determine the age of the drives before putting them into service.

 

[Jelle458]

1. Light on the drive sleds does not affect performance
2. The HP drive firmware does act differently. If you have iLO advanced it can send you an email when a drive is about to fail, it won't do that on 3rd party drives. I have seen many drive sleds which are not original to not work with the lights. The best possible way to get around this is buying actual HP drives that are extremely cheap (72GB I believe exist in smart carrier, which is what HP calls this drive sled), then salvage the drive sled because it's original.
3. You need an HBA card not P840. I forget with Gen9 but you might be able to update the firmware on your P840 and get the option to change it into HBA mode. After HBA mode (or HBA "IT-mode") is used I am using systemrescuecd as a linux live with a bunch of tools. In the terminal you can do sg_scan -i to see drives, then smartctl -a /dev/sg* to see SMART info. -x gives you even more info. Can also be used to format drives with sg_format.

Did you run the newest SPP (Service Pack for ProLiant) on it? Gen9.1 is the newest. After newest SPP has been run on it, you should double check everything is updated, some times stuff like controllers need to be stepped up in firmware, they can't always just take the newest one.
After SPP there is a much newer BIOS and iLO update available, to get the latest security updates and such.

I work with HP everyday, however Gen9 is so old that I don't really see them anymore. Nowadays I mostly work with Gen10 and Gen11.

SPP Gen9.1 is hosted here: Accuris Technologies Public CDN
You can't get it directly from HP. Only starting from Gen10 HP is hosting them.

This site is also hosting iLO 4 firmware:

Accuris Technologies Public CDN

New BIOS can be found, I think I have version 3.40 somewhere.

 

[ziggygt]

1. Light on the drive sleds does not affect performance
2. The HP drive firmware does act differently. If you have iLO advanced it can send you an email when a drive is about to fail, it won't do that on 3rd party drives. I have seen many drive sleds which are not original to not work with the lights. The best possible way to get around this is buying actual HP drives that are extremely cheap (72GB I believe exist in smart carrier, which is what HP calls this drive sled), then salvage the drive sled because it's original.
3. You need an HBA card not P840. I forget with Gen9 but you might be able to update the firmware on your P840 and get the option to change it into HBA mode. After HBA mode (or HBA "IT-mode") is used I am using systemrescuecd as a linux live with a bunch of tools. In the terminal you can do sg_scan -i to see drives, then smartctl -a /dev/sg* to see SMART info. -x gives you even more info. Can also be used to format drives with sg_format.

Did you run the newest SPP (Service Pack for ProLiant) on it? Gen9.1 is the newest. After newest SPP has been run on it, you should double check everything is updated, some times stuff like controllers need to be stepped up in firmware, they can't always just take the newest one.
After SPP there is a much newer BIOS and iLO update available, to get the latest security updates and such.

I work with HP everyday, however Gen9 is so old that I don't really see them anymore. Nowadays I mostly work with Gen10 and Gen11.

SPP Gen9.1 is hosted here: Accuris Technologies Public CDN
You can't get it directly from HP. Only starting from Gen10 HP is hosting them.

This site is also hosting iLO 4 firmware:

Accuris Technologies Public CDN

New BIOS can be found, I think I have version 3.40 somewhere.
[

1. Light on the drive sleds does not affect performance
2. The HP drive firmware does act differently. If you have iLO advanced it can send you an email when a drive is about to fail, it won't do that on 3rd party drives. I have seen many drive sleds which are not original to not work with the lights. The best possible way to get around this is buying actual HP drives that are extremely cheap (72GB I believe exist in smart carrier, which is what HP calls this drive sled), then salvage the drive sled because it's original.
3. You need an HBA card not P840. I forget with Gen9 but you might be able to update the firmware on your P840 and get the option to change it into HBA mode. After HBA mode (or HBA "IT-mode") is used I am using systemrescuecd as a linux live with a bunch of tools. In the terminal you can do sg_scan -i to see drives, then smartctl -a /dev/sg* to see SMART info. -x gives you even more info. Can also be used to format drives with sg_format.

Did you run the newest SPP (Service Pack for ProLiant) on it? Gen9.1 is the newest. After newest SPP has been run on it, you should double check everything is updated, some times stuff like controllers need to be stepped up in firmware, they can't always just take the newest one.
After SPP there is a much newer BIOS and iLO update available, to get the latest security updates and such.

I work with HP everyday, however Gen9 is so old that I don't really see them anymore. Nowadays I mostly work with Gen10 and Gen11.

SPP Gen9.1 is hosted here: Accuris Technologies Public CDN
You can't get it directly from HP. Only starting from Gen10 HP is hosting them.

This site is also hosting iLO 4 firmware:

Accuris Technologies Public CDN

New BIOS can be found, I think I have version 3.40 somewhere.

Thanks so much for the advice and the resource links. HP is very brutal to "right to service". So little is public. I bought a GEN9 24xSFF chassis for $100 and an expander for $20. I wanted to populate it with 1.2 GB drives. It seems the Seagate and Hitachi drives don't play well with the lights. A genuine HP drive in the 3rd party sled works well. But the lights just spin in the Hitachi and Seagate drives even in a HP sled. I have not tried the the updates yet. I am having trouble getting the Gen 9 chassis to boot from a UEFI USB. I am trying to load Truenas. Truenas does not like the DIF feature so all drive need reformatting. It sure seems like Supermicro stuff is a lot more straightforward than HP. Wished I never started. But have to finish now.
Update: I shuffled the slots where I placed the drives. The spinning light stopped spinning in some of the drives. even the Hitachi and Seagate drives. I only have 3 that are spinning now. it is very confusing.

 

[Jelle458]

Especially Gen9 is locked behind a paywall with everything. With Gen10 HP opened up some more, and you can download most resources for Gen10 with a free HP account. The Gen9 is cool though, I liked them and have built thousands on them back when they were popular. I also had customers who were okay with non-HP drives, which is leading to what you are experiencing right now. The controller does not provide health info on non-HP drives and it breaks some stuff that makes it annoying to run, whereas SuperMicro (And Dell) really don't care about what drives you run.

I have had the most luck with cheaper HP 3Par drives (520 format), where after formatting to 512 format they still have some HP firmware, and the server likes that way more. Working in the business means I have access to these drives at stupid low prices, so I don't actually know the prices on the used market.

During POST your server might also throw some warnings with non-HP drives. On the screen where you press F9 to enter BIOS, it can say something like "Genuine HP drives not detected, server will not control the light on the drive". As soon as they are non-HP drives, the light on the sled becomes funky, but it can work sometimes, depends on various firmware versions how sensitive it is. I didn't do it enough to know what firmware version plays best sadly, I have just observed different behaviour with different versions.

When people use non-HP drives usually they ignore the lights, but it means it can be very hard to find a single failed drive out of 24 drives.

 

[ziggygt]

The HP 3Par is a good lead. Prices are pretty good, but boy, that yellow is ugly. It is quite a different platform. Shifting stuff around calmed some of the lights. I have no idea why. 2 drives are still chasing around. The Seagate and Hitachi drives I bought were formatted as 520 with DFI, which Truenas does not like. They all had to be reformatted. I'll stuff the drives where I can't see them.

Curious about the difference in the HP platforms, I asked Copilot for an analysis

 

[Jelle458]

Sounds like you are almost at your goal.

The 3Par is a SAN, and it uses 520 format, so you will have to reformat them to 512 also. Some drives like K2P91B (HP part number) is flooded on the market, and my workplace has them at almost 0 cost. I hope you can find some cheap ones, they work nicely.

 

[ziggygt]

Found one of the spinning light drives is failing SMART. Following up with the other one now. The busy work of recovering from the basement flood is slowing progress.

Thanks for the tip on the HP drives. Wow, those are nice, large drives. I don't see eBay listings for any. I am only seeing the 900GB versions, and they are not worth the trouble. I was looking for 1.2GB as the minimum size, as all the 4TB are too expensive, and I have a lot of bays to fill. I have a Gen 9 24 bay server, in a standard box and a custom Frankenstein 2U JBOD box with 2 x Gen 8 eight bay drive shelves and 1 Gen 8 ten bay shelf using an HP expander.(if I had noticed the 3PAR stuff, I would have probably gone that way). I have 36 drives now.

I also have a GEN 10 LFF server that I would like to sell to finance the acquisition of the drives. I quickly learned it was not possible to mount and power a Gen 8 drive shelf from a Gen 10 server. There is a lot of redundant engineering in the HP products.

 

[ziggygt]

Sounds like you are almost at your goal.

The 3Par is a SAN, and it uses 520 format, so you will have to reformat them to 512 also. Some drives like K2P91B (HP part number) is flooded on the market, and my workplace has them at almost 0 cost. I hope you can find some cheap ones, they work nicely.

I bought a number of Seagate drives and some 3PAR drives at a very low price. Thanks for the tip. Reformatting takes a while. It's crazy how different the packaging of the 3PAR is; perhaps it is to improve cooling. I am planning to 3D print some slides for the rear of the drives and modifying the blank slot fillers to mount the drives. I don't have to buy sleds, and the light doesn't spin. I built an Excel sheet with the serial numbers and a two-digit code I will affix to the drive and the sled, so I can find the bad drives, just have to look it up in my chart.

 

[Jelle458]

The 3Par drives have a different caddy because they go into the 3Par system. The chassis itself is made by Xyratex, they also make the chassis for the competitor IBM V7000. I believe they also make the chassis for some Microsoft storage systems (yes they exist) and others too. IBM moved away from Xyratex some time ago, but the tray from the 3Par system does fit into older V7000 and other systems made by Xyratex.

For reference the tray for the drives you are looking for, HP is calling "Smart carrier". I don't remember if I wrote that earlier, or if you know, but just an FYI. They are sold decently cheap on Ebay. It's the route I would go, instead of 3D printing something.
The light on the Smart carrier tray is controller by a small patch that connects to the backplane. If you do not have the tray, you have no LEDs on the front at all. They can be nice to have because they will light up when using the drive, it's good to quickly identify which drive you are currently working with.

 

[series80]

I have an HP 380DL Gen9 with a generic LSI 9300-16i controller connected to the 380DL drive backplane with two 10Gtek HD miniSAS (SFF-8643) to miniSAS (SFF-8087) 0.8-Meter cables. These cables cost 20 EUR for the pair (on Amazon.de).

I had the same problem with drive LEDs staying dark during accesses, so I checked the cables and they did indeed have the sideband signaling wires, so I upgraded the 9300 firmware to the current version 16.00.10.00 (downloaded from the Broadcom site) and now the LEDs work (they show the circling pattern during accesses).

I have not done any further testing, but I am now very happy that my drives are putting a light show for me

Vassilis

 

[ziggygt]

Are you getting the activities indicator to work on non-hp drives? I have the HP expander between the drives and the LSI controller and I do not have HP cable that came with that. I did not inspect the cables but I do not think they have the extra wires.

 

[series80]

Are you getting the activities indicator to work on non-hp drives? I have the HP expander between the drives and the LSI controller and I do not have HP cable that came with that. I did not inspect the cables but I do not think they have the extra wires.

Yes all my drives are non HP (WD Red SA500 SSD, Seagate IronWolf PRO and Seagate Enterprise NAS HDD) .

Cheers
Vassilis

 

[ngiann]

I have an HP 380DL Gen9 with a generic LSI 9300-16i controller connected to the 380DL drive backplane with two 10Gtek HD miniSAS (SFF-8643) to miniSAS (SFF-8087) 0.8-Meter cables. These cables cost 20 EUR for the pair (on Amazon.de).

I had the same problem with drive LEDs staying dark during accesses, so I checked the cables and they did indeed have the sideband signaling wires, so I upgraded the 9300 firmware to the current version 16.00.10.00 (downloaded from the Broadcom site) and now the LEDs work (they show the circling pattern during accesses).

I have not done any further testing, but I am now very happy that my drives are putting a light show for me

Vassilis

I have a DL380 G9 too, 12LFF backplane. I had the impression that LSI 9300-16i cannot control the LED status on caddies (only HP controllers P840 etc). So, these are good news. Is this because of new firmware on LSI? Please reconfirm this so i proceed to buy one. Also, what about LSI 9300-16i effect on servers fans, are they more noisy?
Cheers
Nikos

 

[series80]

The caddies have all kinds of lights that work with the HP controller. With the LSI 9300-16i I have and with cables that have the sideband signaling cables (I believe most do, but you can check with a continuity meter to confirm) the disc **access** lights work. As I mentioned before, you need to have a recent version of the **controller** firmware for this to work.

My advice it to buy from sellers that accept returns, so that if there are any incompatibilities, you can return the items.

Remember you only get the disk access lights. So procedures for getting the drive to light some other LED (e.g. to identify the specific disk) do NOT work. However if you only want to confirm that a disk device (e.g. /dev/sde) corresponds to a specific drive, you can do a
dd if=/dev/sde of=/dev/null
and see which drive has its access lights blinking continuously.

Apart from that, since all my drives are grouped into one pool, all the disk activity lights more or less blink together. This is rather boring and doesn't convey any useful info (other than that the machine is accessing the drives which you know anyway, since it works).

The caddies I use are original HP, but the drives themselves are Seagate and WD.

BTW, even the LED on the small caddie in the back is also working. I have a WD SSD drive there for the TrueNAS runtime.

Cheers
Vassilis

 

[ngiann]

The caddies have all kinds of lights that work with the HP controller. With the LSI 9300-16i I have and with cables that have the sideband signaling cables (I believe most do, but you can check with a continuity meter to confirm) the disc **access** lights work. As I mentioned before, you need to have a recent version of the **controller** firmware for this to work.

My advice it to buy from sellers that accept returns, so that if there are any incompatibilities, you can return the items.

Remember you only get the disk access lights. So procedures for getting the drive to light some other LED (e.g. to identify the specific disk) do NOT work. However if you only want to confirm that a disk device (e.g. /dev/sde) corresponds to a specific drive, you can do a
dd if=/dev/sde of=/dev/null
and see which drive has its access lights blinking continuously.

Apart from that, since all my drives are grouped into one pool, all the disk activity lights more or less blink together. This is rather boring and doesn't convey any useful info (other than that the machine is accessing the drives which you know anyway, since it works).

The caddies I use are original HP, but the drives themselves are Seagate and WD.

BTW, even the LED on the small caddie in the back is also working. I have a WD SSD drive there for the TrueNAS runtime.

Cheers
Vassilis

Thanks, which controller fw version and where from? Also what about heat and fan noise?
Nikos

 

[series80]

Normal PCIe connectors on the main boards have a maximum performance of 25W. But the LSI 9300-16i controller requires 26.9W.
Some newer motherboards may have a higher performance. Don't know what's the status with the DL380.

I have found that cables such as these
für HP ProLiant DL380 G8 G9 GPU 10pin to 6+8pin Power Adapter PCIE kabel 50cm | eBay.de
work fine, but you can make your own as, despite the size of the connectors, you only need two cables (power and ground)

Regarding the temperature, I have placed a thermometer probe on the 9300 heatsink and it reports a steady 40 degrees C.
I am running the DL380 with 8 disks and the other slots have their covers installed. I believe it is important to have all the openings covered to ensure a controlled air flow over the CPUs and the cards.

The ambient temp. in the room is around 20 degrees and the fans are fairly quiet. They are not noticeable to me at least.

However, some times they rev up for short periods of time (I guess when the CPUs are throttling up).

I am not sure what will happen in the summer, I guess I would need to move the server to an air conditioned room.

The firmware is on a zip file with the name 9300_16i_Package_P16_IT_FW_BIOS_for_MSDOS_Windows.zip
I got it from broadcom.com. You can update the bios from Linux (i.e. you don't need to use the UEFI stuff).
You use the sas3flash command which is already installed in the TrueNAS SCALE distribution.
E.g. to see the version of your firmware you type
sudo sas3flash -listall
The zip file includes a detailed manual for using sas3flash.

BTW, you have to take all drives attached to the 9300 offline before doing the firmware upgrade.

Cheers

Vassilis