SuperMicro 3108 'Bricked'

[Dipsier]

Hello all,

I stumbled upon this forum because of an issue I am having with a RAID controller. It helped me already a great deal - amazing what you guys put together and it helped me already out a great deal.

We have a SuperMicro 2028-HC1R ( 2028TP-HC1R | 2U | SuperServers | Products | Super Micro Computer, Inc. ). We upgraded two of the nodes to Server 2019 yesterday. All went perfectly well. Too well. Near the end, Windows managed to get hung up completely (as I have never seen before) during a Cluster Rename operation. The problem was, the at this moment, I was also upgrading the firmware of the controller (I did not expect a rename to have such a high impact).
Anyway, in the end we ended up seeing this message upon boot:

RAID Adapter
Firmware Failed Validation!!!
Adapter needs to be reflashed.
Press Any Key to continue...

What I tried so far is the following:

- storcli.efi --> is not working at all (basically when I execute any command, the EFI Shell gets more or less stuck)
- sas3flash.efi --> does not see any controller
- sas3flash.exe --> does not see any controller

The only success I had was with megarec.exe. I was able to clean the firmware and also start a flash, however, it fails at one of the sages (I believe it was 'CTRL' stage) complaining something about flash offset or something along these lines.
I tried both the official supermicro ROM and also a ROM from an LSI 9361, both resulting in the same.

I am a bit at a loss now on how I could proceed. Do you guys have any inputs/ideas?

The problem with the server/node is of course it's not that easy to get to the controller physically as it's part of the backplane.

 

[pricklypunter]

Tried using MegaCLI under DOS to flash the SM Rom? I believe that is what they normally use

 

[Dipsier]

I just tried and have the same beahviour as StorCLI in EFI - executing a command just leads it to hang forever

 

[Dipsier]

Actually the command above worked - it's just very slow (I waited around 3 minutes).
I now executed the command megacli -adpfwflash -f smc3108.rom -a0 and am waiting since around 5 minutes with no updates - let's see, I'll be patient
The firmware was erased previously with megarec.

Edit: It is taking a very long time (already waiting 1.5h) - surely this can't be normal. I have the fear that this won't work either.

 

[Dipsier]

Hi - thanks a lot.
May I ask where this is from? It looks like the file is also known as 'megacli.exe' but it is definitely different than the one I already have.

I will visit the datacenter again today and give it a try. MegaCLI really did not work, I tried with various methods (with cleanflash/without, with reboot in between/without,...). Apart from adpcount nothing really worked.

Edit: Oh I see - it's even part of the zip file promoted in the forum. I didn't see it. This worked, I got a bit closer bit still not quite there yet!

Edit2: I tried using MegaCLI now - good news is, it is now responsive and I can execute commands. I flashed the same FW 'over it' by ignoring the version (as it complained this version is already present) but it results in the same issue. Any ideas?

Edit3: I tried an older image from Supermicro with the same result

 

[JFO]

Hi!
You can try flash full 16MB firmware file from LSI PreBoot Installer. In this archive find megacli.exe and fw_16mb.rom in folder Recover.
Recover script from LSI in this PreBoot image with error.
So you can manually create bootable freedos usb disk and copy this file to it or do this with help this archive, instruction how create drive find in it.

For restore firmware on LSI MR SAS 9631-8i i flash fw_16mb.rom from folder boot\LSI\FW\Recover.

I am use command:
megacli.exe -readspd 0 <image>
megacli.exe -readsbr 0 <image>
megacli.exe -cleanflash 0
megacli.exe -m0flash 0 fw_16mb.rom
0 - controller number.

After flashing - reboot. Enter Controller bios utility, reset to defaults again reboot, boot to os or uefi and than uptade to latest firmware with storcli.

 

[Dipsier]

Hi JFO

Many thanks for your explanations.
When you say megacli, I guess you mean megarec3? As I don't think megacli has cleanflash/m0flash commands?

It is a good point regarding LSI Recovery image - I will copy the mentioned firmware and there is also an 3108 firmware on there, I will try that as well.

Currently I do run out of options - I was hoping I could use a SATADOM as a replacement (the node mainly acts as a hypervisor), but it looks like I can't even boot properly from the Windows Boot Stick anymore (I land in a WATCHDOG bluescreen) most likely due to the controller not responding properly.
I will report back soon.

 

[JFO]

Use megacli from folder Recover

 

[Dipsier]

I tried - these are no supported commands.

 

[Dipsier]

I got it working! Thanks everyone for all the help and pointing me in the right directions.
The solution was:

1. Create a bootable DOS stick using the LSI package from here: LSI MegaCLi (preboot CLi),StoreCLi, MegaSCU, MegaREC, SAS2flash and MegaOEM commands | ServeTheHome Forums
2. (Optional) You can also use the ISO linked here (Dropbox link), but make sure to add all the tools available from the ZIP in (1) : Is there a way to restore an LSI 2208 after firmware update failure? | ServeTheHome Forums
3. Get the LSI Pre-Boot USB and copy the MR3108FW.ROM from boot/lsi/fw to the same location as megarec3.
4. Create a new folder on the USB stick and copy the latest Supermicro firmware including storcli.efi on a separate folder on the USB stick.
4. Boot the USB stick and flash the recovery 3108 ROM:
megarec3 -cleanflash 0
megarec3 -m0flash 0 MR3108.ROM

--> REBOOT and boot into EFI Shell

Note that at this point, your controller should already be working again.

3. Boot the EFI shell and open the EFI folder containing the current firmware. Follow the instructions of the provided PDF, but usually it should be:
storcli.efi /c0 download file=smc3108.rom

--> REBOOT


A few important points:
- With a broken or 'empty' firmware, none of the CLI tools (megacli / storcli) will work. They will error or hang/freeze
- If you are at this point, you need to use megarec3 (the 3 is important here, don't use the old version) to first clean and then perform an m0flash. Note that also according to JFO above, this tool might, very confusingly, also be called megacli.
- You cannot flash directly the Supermicro ROM with megarec3. It will succeed, but the controller will complain about incorrect FW. You have to first flash the recovery ROM which I only found on LSI Pre Boot CD using megarec3.
- Even though you can flash the SM ROM directly, the controller will complain. You can use megacli and storcli at this point but you will not succeed in flashing any other firmware, this is why this is a dead-end.
- After flashing the recovery ROM using megarec3, the controller is already in a usable state - at this point, you can use it or flash an updated rom using the normal method


Thanks everyone!

 

[JFO]

Glad that you success reanimate you controller.
Think that Megacli.exe from directory "boot\LSI\FW\Recover" is renamed Megarec3.
LSI PreBoot Installer for USB also have recovery ROM this is file "FW_16MB.rom" from directory "boot\LSI\FW\Recover".

 

[bandit8623]

just wanted to share there is newer firmware listed on broadcom site. search under 9361-8i. jan 21 firmware. 24.21.0-0148

Support Documents and Downloads

Search technical documentation and downloads including firmware and drivers.

www.broadcom.com