Dell R715 iDRAC6 flash repair

H8ROADS

New Member
Oct 27, 2015
17
5
3
Denver, CO
I have 3 Dell R715s and have been going through hell getting them updated the last couple days. Turns out the Lifecycle Controller was too old to go through auto updates, so I had to find an old CentOS OM disk to get them updated to the point it will work through Lifecycle Controller. Everything went well on #1 and #2 and they are happy and up to date. #3 is having a bad day.

Midway through the flash of the Lifcycle/iDRAC controllers through the OM disk, the fans spun up and it was clear something went sideways. After waiting a few hours and no progress, I finally rebooted the box to try again. That has left the iDRAC in an unusable state and the BIOS can't connect. I ran through the OM disk again but it won't detect it so I can't reflash it that way. System Services is also disabled so I can't get into Lifecycle Controller either.

Please someone tell me I haven't just nuked my iDRAC/LC and there's a way I can reflash them somehow.

Option 2 would be if someone could tell me how I could pull it out if it is nuked so I don't have to deal with the myriad of warnings and delayed boot that it causes.

Any help or guidance is much appreciated!
 

H8ROADS

New Member
Oct 27, 2015
17
5
3
Denver, CO
Hi, thanks for the replies but none of that works or is relevant because the idrac is unaccessible. You can't use racadmin or flash via windows. I'm trying to do the TFTP method but that also is not working, so I'm fairly near declaring the mobo dead.
 

Terry Kennedy

Well-Known Member
Jun 25, 2015
1,118
569
113
New York City
www.glaver.org
Here's a link to literally an exact description of what happened and what I'm seeing. Hoping my only option is replacing the mobo per this thread. Dell R710 LifeCycle and iDRAC6 bad firmware flash
On the R710, there are two iDRAC daughtercards. One is the iDRAC Express (always installed) and the other is the Enterprise upgrade (different slot, gives you dedicated Ethernet and SD slot). The Enterprise card is optional. Either Express or Enterprise is $20-ish on eBay. On the R710, the Express card is required or the system will complain. On other G11 servers like the R310, the Express card is optional. On systems which always have an Express card, there is usually a retention spring across the top of the card - on systems where it is optional, it is usually held in with a blue plastic pin.

Your first step is to find out which card is bad. Power down (completely - power cords disconnected) and remove the Enterprise card (if installed). See if the system finds the Express card. If not, you need the Express card. If the Express card works, you need the Enterprise card.

The thing with the fans ramping up to full speed is expected - that is a fail-safe in case the management firmware isn't working properly, or during a reboot of the firmware.

Make sure your BMC firmware is up to date.

There may be some way to override various checks before updating the iDRAC. On the Linux version of the SUU DVD, you can get to the shell and look for the update utility in the "Repository" directory (note that you'll probably see filenames like blahbl~1.bin due to Windows long filenames, so you might want to mount the DVD under Windows first to find the correct short filename). Assuming the script is named "foo.bin", doing "./foo.bin --extract /tmp" should extract all of the update files to the /tmp directory. There will be a firmware image file, an update script, an update utility, and a bunch of mostly irrelevant files. Look at the update script (it is a text file) and find out what arguments the update utility expects. Sometimes running the update utility with a "-?" argument will give you a list of options. There's usually a "--force" or similar option that overrides most of the pre-flash checks.

This is all based on the R710, where I have lots of experience. It may or may not be 100% the same on the R715.
 
Last edited:

H8ROADS

New Member
Oct 27, 2015
17
5
3
Denver, CO
On the R710, there are two iDRAC daughtercards. One is the iDRAC Express (always installed) and the other is the Enterprise upgrade (different slot, gives you dedicated Ethernet and SD slot). The Enterprise card is optional. Either Express or Enterprise is $20-ish on eBay. On the R710, the Express card is required or the system will complain. On other G11 servers like the R310, the Express card is optional. On systems which always have an Express card, there is usually a retention spring across the top of the card - on systems where it is optional, it is usually held in with a blue plastic pin.

Your first step is to find out which card is bad. Power down (completely - power cords disconnected) and remove the Enterprise card (if installed). See if the system finds the Express card. If not, you need the Express card. If the Express card works, you need the Enterprise card.

The thing with the fans ramping up to full speed is expected - that is a fail-safe in case the management firmware isn't working properly, or during a reboot of the firmware.

Make sure your BMC firmware is up to date.

There may be some way to override various checks before updating the iDRAC. On the Linux version of the SUU DVD, you can get to the shell and look for the update utility in the "Repository" directory (note that you'll probably see filenames like blahbl~1.bin due to Windows long filenames, so you might want to mount the DVD under Windows first to find the correct short filename). Assuming the script is named "foo.bin", doing "./foo.bin --extract /tmp" should extract all of the update files to the /tmp directory. There will be a firmware image file, an update script, an update utility, and a bunch of mostly irrelevant files. Look at the update script (it is a text file) and find out what arguments the update utility expects. Sometimes running the update utility with a "-?" argument will give you a list of options. There's usually a "--force" or similar option that overrides most of the pre-flash checks.

This is all based on the R710, where I have lots of experience. It may or may not be 100% the same on the R715.
Thanks for the insight. After much frustration and research yesterday I came to the same conclusion re: daughterboard. I went ahead and ordered a new one to swap out later this week and give that a try. Unfortunately I have to pause due to some business travel, but I'm hoping the replaceable daughtercard with the ethernet will fix it. I do have the enterprise and I also made the assumption the R715 is the same as the R710 (of which I also have 3, but they aren't handy - in a corp DC)

Thanks for the tip on the SUU - I did try to do some force installations on a OMSA disc and it didn't work, but I haven't tried the original SUU where this all started. That's a good idea to try and repush with the -force option.