JBOD adventures with Dell PERC H710P Mini (LSI 2208)

Malvineous

New Member
Sep 28, 2018
10
6
3
Hi all,

I've recently gotten hold of a Dell PowerEdge R720 with a PERC H710P Mini controller. Wanting to try out ZFS I had hoped to make the individual drives available to the host (as JBOD/passthrough drives) but sadly the Dell firmware does not support JBOD for this controller. Other cards based on the same LSI 2208 "Thunderbolt" controller do, so after reading a number of posts about people who had talked about flashing alternate firmware on the card (but none that I could see who had tried), I thought I would give it a go.

Unfortunately, none of the official LSI firmwares seem to work. They flash successfully but the card boots up with a firmware failure and no drives are seen. I haven't tried firmwares from other manufacturers, but the purpose of this post is to explain how to recover from a failed flash, in case anyone else wants to give it a go with other firmwares.

So, to recover the H710 you will need:

1. A bootable USB stick with the LSI MegaRAID recovery image on it.

The instructions for making the bootable USB under Linux didn't work for me, so I had to manually create a bootable USB with SYSLinux and copy across all the configs and programs. Make sure you get this to work before starting! Since rebooting the R720 is quite slow, you can test it out locally with QEMU first:

Code:
qemu-system-x86_64 -hda /dev/sdx
If it boots (and you can run the MegaRAID commands and just get errors about no controllers found) then you're good to go.

They have a bootable ISO as well which would've been a lot easier, but this didn't work for me because they don't load any CD drivers, so there's no drive letter with the firmware on it after you've booted so that was completely useless.

2. A copy of the latest Dell firmware for the PERC.

I downloaded mine from the Dell support site. Put in your service tag and change the category to "SAS RAID". In my case, the download was called "Dell PERC H710P Mini Monolithic Adapter firmware release 21.3.5-0002". The .exe download can be extracted under Linux with "unzip", and contains a file called FW8148PM.rom. This is the file you will need. Make sure it's accessible from the USB stick.

3. Patience. If the wrong firmware is flashed, the system will take about 30 mins to boot, and will appear stuck on 'initializing firmware'. Just give it time and it will boot eventually. With the wrong firmware loaded, none of the usual tools will be able to see the card. Only the LSI recovery mode tool will recognise and be able to reflash it.

In order to do the recovery for the H710P, you'll need to boot the USB, go into the MegaRAID menu and then boot the recovery image.

dell-h710-2-lsi_recovery.png

In my case the USB drive was drive C, adjust as needed. Once booted, you can:

Code:
set path=a:\;a:\fdos;a:\fdos\bin   # Needed to avoid "DOS4GW not found" error
c:
cd\boot\lsi\fw\recover
# Try the LSI ROM (this flashes but doesn't work)
megacli -m0flash 0 ..\9286\mr2208fw.rom

# Go back to the Dell ROM
megacli -m0flash 0 \fw8148pm.rom
Note that although the command here is "megacli", it's actually megarec.exe that has been renamed for some reason.

The LSI ROM does flash:

dell-h710-1-2208_flash.png

But during boot it still says there was a FW failure. Linux dmesg says this:

Code:
megaraid_sas 0000:02:00.0: Waiting for FW to come to ready state
megaraid_sas 0000:02:00.0: FW in FAULT state!!
megaraid_sas 0000:02:00.0: FW restarted successfully from megasas_init_fw!
megaraid_sas 0000:02:00.0: Waiting for FW to come to ready state
megaraid_sas 0000:02:00.0: FW in FAULT state!!
megaraid_sas 0000:02:00.0: Failed from megasas_init_fw 5583
Flashing the Dell image works fine though and gets everything back to normal:

dell-h710-3-dell_recovery.png

After restoring the Dell FW, I didn't even lose my RAID config. But looks like I'll have to be content with two RAID0 virtual disks, each with only one physical disk, instead of true JBOD mode.

If anyone else decides to have a go with firmwares from other manufacturers, please let me know! (Also if you're reasonably certain of one that might work, let me know why and where it is and I might give it a go. I'd be more eager except it's such a time consuming process, having to wait 30 mins for the server to boot!)
 
  • Like
Reactions: fohdeesha

nthu9280

Well-Known Member
Feb 3, 2016
1,627
468
83
San Antonio, TX
Check the posts by @vanfawx here in the forums. He posted in detail on how to flash lsi 9270-8i (2208 chip) RAID cards to 2308 HBA like 9207-8i as well as back to MegaRAID FW. Since yours is a Dell branded card, not sure if that has any implications. I believe he also has a blog post on the interwebs.
 

Malvineous

New Member
Sep 28, 2018
10
6
3
Hmm, very interesting. It seems the Dell branded 2208 cards are not compatible with the LSI ones. One of the many differences seems to be that the SAS BIOS is part of the main system BIOS in a Dell, while with the other cards it's part of the option ROM on the card itself. I am assuming this because even after you reflash (or indeed, completely wipe) the "BIOS" component on the card, you still get the original POST message about pressing Ctrl+R to configure the card.

However if the LSI 2208 and 2308 firmwares are compatible, I wonder whether there's a Dell-branded 2308 card that I could grab the firmware from? Any tips on which Dell cards use the 2308 chipset?
 

nthu9280

Well-Known Member
Feb 3, 2016
1,627
468
83
San Antonio, TX
I believe 9217-8i is Dell's nomenclature. You should be able to grab the latest from LSI/Avago/Broadcom support section. If you use megarec and clear the FW and BIOS, It should just behave like a plain HBA.

smart @$$ in me wanted to post this link at first:
LMGTFY
 

Malvineous

New Member
Sep 28, 2018
10
6
3
Haha I did do some Googling before replying but all I could find are the same links you posted.

I'm reasonably sure those ones won't work, because the 9217-8i and similar are all LSI model numbers, so they would be firmwares for LSI cards that have been bundled with Dell hardware. The actual Dell devices have names like H710P, H310, etc. and there is very little mention of what controller chips they're running. The H710P firmware I flashed to recover the card for example, had no mention of LSI, 2208, Thunderbolt, or any of the other details about the card's chipset.

I can confirm that clearing the FW and BIOS does not work, because when booting Linux in this state the MegaRAID driver comes up saying the firmware is in fault and then gives up.

The only chance of this working is finding out which of Dell's PERC cards (not the LSI ones) are using the 2308 chipset. I will keep looking, but the PERC chipsets don't seem to be that widely documented.
 

Malvineous

New Member
Sep 28, 2018
10
6
3
Just had another go at this. Dell don't seem to have released any Dell-branded adapters based on the SAS2308 chipset so unfortunately there isn't anything there to flash. I tried flashing the H700 (SAS2108) firmware and the H730 (SAS3108) but even the recovery tool refused to flash them saying they were incompatible. I tried hex editing the H700 firmware file to make the chipset and PCI IDs match my H710P, but sadly that didn't make any difference.

I tried booting with a cleared flash and as expected this just results in a bunch of kernel warnings about hung tasks, as the driver keeps "waiting for FW to initialize".

Annoyingly, the H700 and H730 firmwares have references to the SAS2208 chipset, so it looks like they might actually be compatible, if anyone can figure out how to get a flash utility to accept the image.

Weirdly after all this, Dell's perccli tool that originally flashed the bad firmware no longer recognises the card, saying there are no controllers installed on the system!

EDIT: Also, in case anyone finds it useful, the Linux PCI ID database is handy to figure out what cards share the same chipsets. Here are links for the SAS2208 and SAS2308 device list.
 

Datashadow

New Member
May 3, 2019
1
1
1
Boston, MA
Hello Malvineous,

I've been having my own adventures with the Dell R710 (not P) and R310 controllers (both Mini Mono(lithic) in a Dell R720xd.

If you've not done so yet I'd recommend checking out the YouTube videos by ArtOfServer:
Main Channel: https://www.youtube.com/channel/UCKHE9DEep52XlmwLbZUKvyw

Flashing R710MM 5CT6D vs MCR5X with IT Mode firmware:
Finally got H710 mini MCR5X flashed with IT mode firmware:

The latter video has the key information that could have caused your problems.

Take care.
 
  • Like
Reactions: BLinux

Twix

New Member
Jan 13, 2020
1
0
1
Hey. Has anyone else done this? Can you share a little instruction on successful flashing to LSI 2208?
 

Ellen Hardin

New Member
Feb 19, 2020
7
0
1
I have successfully flashed a H710 Mini using directions here: Introduction - Fohdeesha Docs
My only problem is after flashing only 4 of my 8 drives have passed through. I am unable to enter the configuration utility. I have the older MCR5X version of the card. I suspect that there is something not quite right in the boot image ROM's. Any ideas on how to proceed to see all 8 drives?
My base system is a Dell R520.
 

Spartacus

Well-Known Member
May 27, 2019
786
327
63
Austin, TX
I have successfully flashed a H710 Mini using directions here: Introduction - Fohdeesha Docs
My only problem is after flashing only 4 of my 8 drives have passed through. I am unable to enter the configuration utility. I have the older MCR5X version of the card. I suspect that there is something not quite right in the boot image ROM's. Any ideas on how to proceed to see all 8 drives?
My base system is a Dell R520.
Are you sure it was the flash and not a cable or something else, have you tried switching the cables to see if the other set of drives shows up or otherwise?

Also I’m pretty sure you could add the boot rom if you wanted/needed.
 

Ellen Hardin

New Member
Feb 19, 2020
7
0
1
The H710 adapter showed all drives before flashing, so I am pretty sure cables and back plane are OK. I have loaded the boot roms provided by Fohdeesha, but am unable to enter it with Cntl C. The only option I have within the system bios is to active or deactivate the ROM. I have located the images from the LSI website. If I can decypher the bat files from Fohdeesha, I may try flashing those. I appreciate any other suggestions.
 

nthu9280

Well-Known Member
Feb 3, 2016
1,627
468
83
San Antonio, TX
IIRC, fujitsu oem HBAs had similar issue with cross flash. It is resolved by flashing correct SBR file. Empty SBR only enables 4 drives. Alternatively,
You may want to try LSIutil.exe and enable if the ports are disabled. The menu options are god awful though.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,522
2,697
113
31
fohdeesha.com
I have successfully flashed a H710 Mini using directions here: Introduction - Fohdeesha Docs
My only problem is after flashing only 4 of my 8 drives have passed through. I am unable to enter the configuration utility. I have the older MCR5X version of the card. I suspect that there is something not quite right in the boot image ROM's. Any ideas on how to proceed to see all 8 drives?
My base system is a Dell R520.
Not sure why you posted in an unrelated 2 year old thread instead of our main crossflashing/guide thread, but anyway :p

Sounds like your flash went wrong or you skipped a step. I just grabbed the same card to test (MCR5X), and it can see all 8 drive slots without issue. I can also enter the boot configuration menu during boot without a problem as well after flashing the bootrom like the guide instructs:



Note: if you're trying to enter it using the idrac remote web console, you can't just hit ctrl+c, your web browser will interrupt the ctrl key. You need to click "keyboard" up top to pull up the virtual keyboard, and use that to send ctrl+c when you see the LSI /avago boot crap during boot.

I would check the cabling in your server to ensure everything is good, then follow the guide from scratch again, ensuring you follow both the freedos and linux steps and programming a sas address back, for your revision of card, because it can definitely see all 8 slots. Also note the bootrom files in the guide are unmodified straight from LSI, so if you go and download the boot files from LSI like you state you'll end up with the exact same files

when done flashing, unplug the battery from the raid controller, and completely remove power from the server for about a minute, then plug it back in and boot it back up to the linux iso, and send me the output of the command "info" so I can see if everything flashed correctly
 
Last edited:

Ellen Hardin

New Member
Feb 19, 2020
7
0
1
Sorry I have posted in the wrong area. Please be patient with up newbies.

I have re-flashed the card with careful attention to your directions. There is no change and only 4 drives are visible. Here is the info results your requested:

upload_2020-2-22_14-45-20.png

lsblk still shows only 4 drives + usb key:
upload_2020-2-22_14-49-32.png

Thanks again for your help. Any Ideas?
 

Attachments

Ellen Hardin

New Member
Feb 19, 2020
7
0
1
yes, I will be booting with UEFI. I will be running openmediavault on this machine. If I need to, I can move the ssd to the sata port that normally attaches to the DVD. I did test without any rom and with only the BIOS boot rom and then adding the UEFI rom. Nothing made any difference.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,522
2,697
113
31
fohdeesha.com
your card looks flashed correctly, and there hasn't been a single report anywhere else of missing drives, so it's probably an issue with the OS you're using to look for drives or something hardware related. To rule out the OS and check if the card itself can see drives:

Enter the boot configuration utility. you may need to flash the BIOS bootloader for this, you're running the UEFI bootloader and it looks like your server is booting in BIOS mode. remember to use the virtual keyboard to send ctrl+c or else the ctrl key won't get sent

In the blue menu screen, hit enter on the card name (should be SAS9205-8E). In the menu after that, arrow down to "SAS Topology" and hit enter. On the screen after that, hit enter on "Direct Attach Devices" and it should drop down a list of all the drives the card is connected to. you should see all 8 of your drives hopefully, here's mine showing my two drives, please post a screenshot of this screen:




If they're not even showing up there, then we can move on to potential HW or config issues. Are these spinning drives? I suppose it could also be the default drive spin-up time values on the IT firmware not being as lax as the stock FW, those can be adjusted if the previous steps do not work
 

Ellen Hardin

New Member
Feb 19, 2020
7
0
1
Thank you for your patience and all your help.

I have used several different options to look for drives. I have used the linux portion of your method, Ubuntu live boot and also Centos on a key drive. None of them can see more than 4 drives. I initially tried to just map each drive as a Raid0, but this would not work with ZFS which is what I would like to use for OpenMediaVault.

This is a screen shot of the original Dell Bios utility:
upload_2020-2-24_11-55-51.png
This shows all drives detected. This leads me to believe that the adapter, cables and backplane are not the issue.

I was having trouble entering the bios utility after flashing the adapter. I had been booting your deesh-Linux-v1.0.iso using UEFI. When I just booted in bios mode, and then reflased I could consistently enter the bios utility.

I have a small SanDisk SSD that I plan to use as my boot drive. I have 7 Hitachi 2TB drives (SATA but server class) which are spinning disks. All have checked out in other machines.

After entering the bios utility
upload_2020-2-24_11-56-36.png
I note that this shows slots 0,2,3 and 4, but no slot 1. This makes me worry a little that there is a problem with card, cable or back plane, but I can’t explain why all the disks would be seen on the original firmware for the card if that is the case.

I also tied increasing the amount of time allowed for spinpup, but this did not make any difference. I carefully restored everything to default after my failed experiments.

I appreciate any other ideas you might have. Thank you again for your help.