HP DL180 G6 and SAS HBA Hell (Help?)

Discussion in 'RAID Controllers and Host Bus Adapters' started by mattlach, Oct 14, 2017.

  1. mattlach

    mattlach Member

    Joined:
    Aug 1, 2014
    Messages:
    123
    Likes Received:
    11
    Hey all,

    So this is a duplication of a thread I have going in another forum, but I figured I'd try here as well because - you know - more eyeballs. When you are dealing with relatively rare problems, anything might help.

    Anyway, let's go.


    Background:


    About 3.5 of years ago I bought a used HP DL180 G6 server on eBay, because it looked like a great deal. Dual Xeon 1366 sockets and a built in backplane with SAS expander sounded great to me. I didn't opt to buy one of the official HP RAID cards because I preferred an HBA, as I wanted to run ZFS with a JBOD configuration.

    Instead I bought an IBM M1015, flashed it to an LSI IT mode and it worked great... EXCEPT

    Apparently HP servers look to make sure that you use officially supported HP PCIe cards, and if you don't go into a full fan speed freakout mode because they can't tell the temp of the PCIe card and err on the side of safety and blast the shit out of it with full fan speed. When you have 8x little 18krpm fans blowing through a 2U chassis this is LOUD to the point of being practically unusable for home use.

    At over 100db and literally sounding like a jumbojet taxiing down the runway preparing for takeoff, even when it was hidden in the basement, I could hear it two floors up in my bedroom with all doors shut in between.

    A noise reference:



    At the time I searched for a noise solution. HP at some point released a firmware update for the 4U DL380 G6 that eliminated this fan problem, but the 2U DL180 G6 never got one.

    I eventually gave up, harvested the CPU's and RAM from the server, and got a Supermicro board and did a custom server build in a Norco case and have been happy with my silence ever since...


    ...until a couple of weeks ago.

    Slightly inebriated, I was browsing eBay and came across a matched pair of L5630 Xeon's for ~$5, and was like, why the hell not. I have 12x 8GB of registered DDR3 just hanging around, why not play with the old Dreamliner again?

    I took to the internet, and someone suggested that HP had a compatible HBA called the H220 that wouldn't trigger fan freakout mode, and this was enough to get me thrilled.

    Reason being, I am going through a storage update on my main server (related threads here, here and here) and since my current backup solution, Crashplan is exiting the consumer backup market, I was thinking, I could use this old server to do some backup of my own, with my old drives.



    The Problem:


    So I went ahead and ordered one of those HP H220 HBA's, and it looks like the DL180G6 recognizes it as a friendly device, because thus far we haven't gone into fan freakout mode, which is a good thing!

    I do have some problems though.

    I installed the HBA, and hooked it up to the backplane in the DL180G6, and powered up my dreamliner.

    The BMC rewrote the firmware (like it always does when you insert a new PCIe card for some reason) and everything looked good, until the H220 option ROM loaded, at which point it froze eternally on the following:

    Code:
    Initializing.../
    With the slash spinning in circles for all eternity. (I let it sit overnight, just to make sure it wasn't just doing something, and it still stayed in that state.)

    Alright. Some troubleshooting. Disconnected the backplane, and connected some random drives I had kicking around using one of those SAS to SATA breakout cables, and this worked just fine, with the devices detected properly.

    [​IMG]

    So, it seems like the H220 has some trouble initializing when connected to the backplane, but otherwise works fine. I know that the backplane works, as I have had it working with 12 disks connected to a IBM M1015 SAS controller with everything working. The server was just deafening, because it ddidn't recognize that SAS controller as a known card, so I decided not to use it that way.

    So, it looks like I am dealing with some sort of incompatibility between the HP H220 controller and the backplane in the HP DL180G6.

    Here is what I have tried thus far:



    1.) Disable Boot ROM:

    Gone into the SAS configuration page and disabled the boot ROM. This allows me to boot with the backplane connected. I was hoping it would just be a bootrom init problem, and then it would work in OS, but that is not the case. Ubuntu 16.04LTS freezes during boot at the following point:

    Code:
    mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00503c950), phys(8)
    mpt2sas_cm0: expander_add: handle(0x0009), parent(0x0001), sas_addr(0x50001c1071540000), phys(19)
    So, it looks like it is having trouble with the SAS expander in the backplane.



    2.) Unplug Backplane I2C Connector


    Doing some more googling about I found someone suggesting that they got a troublesome H220 to boot with their HP blackplane by pulling out the i2c cable that goes between the backplane and the motherboard in their server.

    I tried this, and I was able to get the server to boot!

    Only problem is, now it doesn't see any of the drives connected tot he backplane, so I guess that didn't really solve anything.



    Things I can think of trying next (but I could use some help)


    - I have read that the backplane has updateable firmware, but I don't quite know how to do this, and I don't know if I can do so without a working connection to the HBA... Any suggestions?

    - Also, I want to try to update the H220 firmware, but I don't know where HP hides their firmware and firmware update tools for these cards. I've googled a bit without much success. I found the official HP support page, but when I filter on H220 it just seems to contain tons of hard drive firmware?? Any suggestions?

    - I could try to flash the controller with standard LSO 2308 firmware, but I'm guessing this will remove whatever the BMC in the server uses to recognize it as a properly supported PCIe card, and send it back into fan freakout mode, which was the problem I was trying to avoid in the first place.

    I have done some other googling, and there are a few instances of people with similar problems, but none of them have any solutions. If anyone knows anything I might try, I'd appreciate it.


    Of course, now that I read HP's docs, they list support for Gen 8 servers only, so maybe this is a lost cause, and I should just buy a cheap case, another used Supermicro server board, and just recycle this damned HP server and never let it cause me any aggravation ever again...


    I'd appreciate any suggestions!
     
    #1
  2. tomaash

    tomaash Member

    Joined:
    Oct 11, 2016
    Messages:
    41
    Likes Received:
    20
    #2
  3. mattlach

    mattlach Member

    Joined:
    Aug 1, 2014
    Messages:
    123
    Likes Received:
    11
    #3
  4. mattlach

    mattlach Member

    Joined:
    Aug 1, 2014
    Messages:
    123
    Likes Received:
    11
    #4
  5. tomaash

    tomaash Member

    Joined:
    Oct 11, 2016
    Messages:
    41
    Likes Received:
    20
    Try starting here. From what I can see there seem to be just rpm packages or Windows binaries.
    Download links don't work for me somehow, so I just used the e-mail icon to get direct links.

    Edit: Use filters on bottom left of the page.
     
    #5
  6. mattlach

    mattlach Member

    Joined:
    Aug 1, 2014
    Messages:
    123
    Likes Received:
    11

    Just gave it a try.

    I was able to decompress the rpm using rpm2cpio as described here.

    The archive contains the following files:

    Code:
    $ ls -l
    total 9264
    -r--r--r-- 1 matt matt   34647 Oct 14 16:35 CP032390.xml
    -r--r--r-- 1 matt matt     279 Oct 14 16:35 firmware.cfg
    -r--r--r-- 1 matt matt  799080 Oct 14 16:35 H210i_it.fw
    -r--r--r-- 1 matt matt  891376 Oct 14 16:35 H220i_it.fw
    -r--r--r-- 1 matt matt  891376 Oct 14 16:35 H220_it.fw
    -r--r--r-- 1 matt matt  891376 Oct 14 16:35 H221_it.fw
    -r--r--r-- 1 matt matt  891376 Oct 14 16:35 H222_it.fw
    lrwxrwxrwx 1 matt matt       5 Oct 14 16:35 hpsetup -> setup
    -rwxr-xr-x 1 matt matt 4216010 Oct 14 16:35 lsiflash
    -r--r--r-- 1 matt matt  202240 Oct 14 16:35 mptsas2.rom
    -rwxr-xr-x 1 matt matt  638868 Oct 14 16:35 sas2flash
    -r-xr-xr-x 1 matt matt    6904 Oct 14 16:35 setup
    
    Executing these binaries didn't go too well:

    Code:
    $ sudo ./setup
    Supported hardware is not present, not enabled, or in a state that an installation could not be attempted.
    Code:
    $ sudo ./lsiflash
    Usage:
       /b debug_level     Ramps up the debugging
       /c                Flash controller
       /d xmlname        List controllers
       /l                Log utility output
       /t model          Specifies an applicable controller model
    $ sudo ./lsiflash /d
    Segmentation fault (core dumped)
    Code:
    $ sudo ./sas2flash 
    LSI Corporation SAS2 Flash Utility
    Version 13.10.03.00 (2012.07.09) 
    Copyright (c) 2008-2012 LSI Corporation. All rights reserved 
    
        No LSI SAS adapters found! Limited Command Set Available!
        Finished Processing Commands Successfully.
        Exiting SAS2Flash.
    So, it can't seem to see that a compatible card is installed, but I know it is, because:
    Code:
    $ dmesg |grep -i lsi
    [    9.015578] mpt2sas_cm0: LSISAS2308: FWVersion(13.10.113.00), ChipRevision(0x05), BiosVersion(07.25.03.00)
    and

    Code:
    $ dmesg |grep -i mpt2sas
    [    8.205639] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (98996244 kB)
    [    8.249833] mpt2sas_cm0: MSI-X vectors supported: 16, no of cores: 16, max_msix_vectors: -1
    [    8.263679] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 38
    [    8.517897] mpt2sas0-msix1: PCI-MSI-X enabled: IRQ 39
    [    8.533143] mpt2sas0-msix2: PCI-MSI-X enabled: IRQ 40
    [    8.548222] mpt2sas0-msix3: PCI-MSI-X enabled: IRQ 41
    [    8.662721] mpt2sas0-msix4: PCI-MSI-X enabled: IRQ 42
    [    8.676817] mpt2sas0-msix5: PCI-MSI-X enabled: IRQ 43
    [    8.690844] mpt2sas0-msix6: PCI-MSI-X enabled: IRQ 44
    [    8.705001] mpt2sas0-msix7: PCI-MSI-X enabled: IRQ 45
    [    8.719055] mpt2sas_cm0: iomem(0x00000000fbdf0000), mapped(0xffffc9000ce80000), size(65536)
    [    8.733714] mpt2sas_cm0: ioport(0x000000000000d000), size(256)
    [    8.805823] mpt2sas_cm0: sending message unit reset !!
    [    8.822410] mpt2sas_cm0: message unit reset: SUCCESS
    [    8.900084] mpt2sas_cm0: Allocated physical memory: size(16421 kB)
    [    8.915624] mpt2sas_cm0: Current Controller Queue Depth(8056),Max Controller Queue Depth(8192)
    [    8.949477] mpt2sas_cm0: Scatter Gather Elements per IO(128)
    [    9.015578] mpt2sas_cm0: LSISAS2308: FWVersion(13.10.113.00), ChipRevision(0x05), BiosVersion(07.25.03.00)
    [    9.032135] mpt2sas_cm0: Protocol=(
    [    9.151834] mpt2sas_cm0: sending port enable !!
    [   25.947774] mpt2sas_cm0: _config_request: timeout
    [   26.083616] mpt2sas_cm0: fault_state(0x7400)!
    [   26.094540] mpt2sas_cm0: sending diag reset !!
    [   30.796977] mpt2sas_cm0: diag reset: SUCCESS
    [   30.808282] mpt2sas_cm0: failure at /build/linux-Pcn0xK/linux-4.4.0/drivers/scsi/mpt3sas/mpt3sas_scsih.c:4946/_scsih_sas_host_add()!
    [   30.811557] mpt2sas_cm0: port enable: FAILED with (ioc_status=0x00000004)
    Maybe I received a bad card?
     
    #6
  7. nthu9280

    nthu9280 Well-Known Member

    Joined:
    Feb 3, 2016
    Messages:
    1,092
    Likes Received:
    250
    Appears that the OS is able to correctly identify the card So I doubt the card is bad. Not sure if that specific HP server throws a hissy-fit if you flash LSI native IT FW. Depending on the version of the 2308 you have, either 9205-8i for Co stepping or 9207-8i for D1 stepping available for that card.
     
    #7
  8. mattlach

    mattlach Member

    Joined:
    Aug 1, 2014
    Messages:
    123
    Likes Received:
    11

    Thank you for that.

    How do you find the stepping? I don't see any markings on the card that give it away.

    [​IMG]
     
    #8
  9. mattlach

    mattlach Member

    Joined:
    Aug 1, 2014
    Messages:
    123
    Likes Received:
    11
    Doh, I must have been sleep deprived yesterday or something. I didn't think to look at the bottom of the card :p (Its so annoying that you have to take the entire riser/drive cage out in order to remove a PCIe card in this server...)

    [​IMG]

    Looks like I have the 9205-8i version.

    I guess I'll try to flash it with the LSI firmware and see if that works.

    Will be interesting if the HP server looks for firmware cues that the right card is installed, or if it recognizes the actual hardware...

    Seems like a decent HBA either way, so if it doesn't quite work out in this server, I'm sure I'll find another use for it.

    Now to find that thread I remember reading (either on here, the hardforums or the FreeNAS forums) where someone explained how to flash this card with LSI firmware. It involved something silly, like using the flashing executable from the old P14 release (before the binary checked to see if it was a HP card) but substituting the firmware binaries from a more modern release of the firmware.

    I'll have to look up somewhere which phase LSI drivers are in recent releases of Ubuntu. I've read that not matching LSI driver phase with firmware phase is asking for trouble...
     
    #9
  10. vanfawx

    vanfawx Active Member

    Joined:
    Jan 4, 2015
    Messages:
    252
    Likes Received:
    40
    @mattlach I actually had the exact same issue with my H220 connected to a SuperMicro 4U JBOD. I flashed it with the 20.00.07.00 version for the 9205-8e and it's been working perfectly since.
     
    #10
    mattlach likes this.
  11. mattlach

    mattlach Member

    Joined:
    Aug 1, 2014
    Messages:
    123
    Likes Received:
    11
    I just finished doing this and was coming over here to write up how I completed it! It wasnt QUITE that straight forward for me though.

    In the end what did it was this.

    Oddly enough, even though the HP SAS expander / backplane combo in the G6 is older than the H220 SAS controller, older firmwares refuse to work with this sas expander / backplane.

    The solution is needing the latest firmware, but the latest HP firmware is too old.

    So, I had to flash LSI's IT firmware, but this was not straight forward.


    First problem:

    The Sas2flsh application recognizes the card as an HP branded card, and refuses to flash LSI firmware to it.

    The sneaky workaround. Download the older LSI P14 firmware update, and use the sas2flsh executable from that package. It predates the HP branded controller, and thus doesnt recognize it as such.

    Now download the latest firmware update and use the P20 firmare bin file from that package. Use the old sas2flash executable to flash the new firmware.


    Second Problem:


    While the sticker on the card identifies it as a SAS 9205-8i, the 9205 firmware on LSI's site refuses to flash, stating the stepping of the card is a later revision than supported by the firmware.

    I did some googling and read that others have had success with the 9207 firmware, so I downloaded the 9207-8i firmware and it worked like a charm.

    The great news is the server didn't go into panic fan mode once flashed!


    For posterity, here are the appropriate command lines, after creating a bootable freedos USB stick and booting to it:

    Erase firmware first (optional)

    I did this to get rid of the boot ROM, as I don't plan on booting from this controller and it just slows down the boot time. Warning If you do this, you can brick the controller if you don't flash a new firmware before rebooting.

    sas2flsh -o -e 6

    Now flash the new firmware

    sas2flsh -o -f 9207-8.bin -c 0

    Took a little while for it to finish, but once done, worked like a charm.
     
    #11
    Last edited: Oct 22, 2017
    jamesy_1988 likes this.
  12. Stefan75

    Stefan75 Member

    Joined:
    Jan 22, 2018
    Messages:
    69
    Likes Received:
    7
    Good to know.. I'm thinking about using a DL380 G6 with a Fujitsu D2607 (SAS2008).
     
    #12
  13. darkcyde

    darkcyde New Member

    Joined:
    Nov 6, 2018
    Messages:
    1
    Likes Received:
    0
    I know this is an old thread, but I just wanted to say thanks for coming back and actually finishing the thread off with the solution. So many people don't do this, and the net is full of useless threads because of this!

    I was running a Dl180 G6 with the suppleid P410 PCI card, but I didn't like the fact that I didn't have spin up/down and SMART capability, and when my Parity drive kept failing for some reason I decided to buy a H220 after reading this thread.

    I fully prepared myself for some flashing fun as per your instructions, and I'm glad I did....I seated the new card, plugged in one SAS drive and it detected and accepted it and booted straight into unRaid where it showed me the drive....brilliant, I thought!

    So I powered down, plugged in the SATA cache drive and the other 8 SAS drives.....and it sat initialising....and initialising.....and initialising!

    The Firmware was on P15, so I plugged in my pre prepared flash drive with DOS and the sas2flash programs and binaries for the 9207 chipset, used the EXE from the P14 update, and the binary from the P20, used your instructions, and it worked first time!!

    Rebooted with everything plugged in, and instant joy when all the drives were visible....so now they are all preclearing ready for the great data migration tomorrow from my backup drives!!

    Thanks again....I could never have worked this out on my own!!
     
    #13
Similar Threads: DL180 Hell
Forum Title Date
RAID Controllers and Host Bus Adapters Proliant DL360 G6 + DL180 backplane (+ PCI-E HBA) = Fan noise? Jul 19, 2016
RAID Controllers and Host Bus Adapters HP DL180 G6 USB not working when added a Dell H310 May 9, 2016
RAID Controllers and Host Bus Adapters Recomend a RAID-controller for HP DL180 G5 Dec 19, 2013
RAID Controllers and Host Bus Adapters Solution to Supermicro/LSI/Intel RAID hell May 3, 2018
RAID Controllers and Host Bus Adapters How to check HBA queue depth on Windows from Powershell ? Feb 21, 2017

Share This Page