Dell C6100 - anyone brick a board yet?

[Patrick]

Had a reader contact me today. He apparently bricked two Dell C6100 motherboards by configuring IPMI via BIOS rather than through the OS. Symptoms are basically that the nodes continuously reboot every 45 seconds. One can get into the BIOS but then after 45 seconds they reboot. They report BMC not responding and will not respond to IPMItool.

Anyone else experience this?

 

[dba]

Yikes! I did preliminary configuration of IPMI via BIOS for all eight of my C6100 boards, setting static IP/Gateway/DNS. No problems.

Had a reader contact me today. He apparently bricked two Dell C6100 motherboards by configuring IPMI via BIOS rather than through the OS. Symptoms are basically that the nodes continuously reboot every 45 seconds. One can get into the BIOS but then after 45 seconds they reboot. They report BMC not responding and will not respond to IPMItool.

Anyone else experience this?

 

[Rob T]

Thanks, Patrick, for opening a thread for me. Not sure what happened - all I did was try to set a static IP for IPMI via the BIOS. What happend was sort of similar to what is described here, only I never attempted a FW update:

C6100 BMC Firmware Update - PowerEdge General HW Forum - Servers - Dell Community

If anyone else has seen anything similar and was able to fix it, I'd appreciate any help. I'm guessing that the BMC on these has somehow become bricked which is in turn causing the continuous reboot issue.

 

[nitrobass24]

Bummer dude! Some googling I found this.

Re: Dell C6100 BMC / BIOS HELP - PowerEdge General HW Forum - Servers - Dell Community

 

[Dragon]

Thanks, Patrick, for opening a thread for me. Not sure what happened - all I did was try to set a static IP for IPMI via the BIOS. What happend was sort of similar to what is described here, only I never attempted a FW update:

C6100 BMC Firmware Update - PowerEdge General HW Forum - Servers - Dell Community

If anyone else has seen anything similar and was able to fix it, I'd appreciate any help. I'm guessing that the BMC on these has somehow become bricked which is in turn causing the continuous reboot issue.

Damn, so that's why some people are selling C6100s with only 2 nodes

Who did you buy yours from?

 

[PigLover]

You didn't upgrade the firmware so its doubtful that you bricked it the same way they described in that blog post.

One thing worth a try before you give up: Most likely the BMC config data got corrupted. Try pulling the sled (so it is not powered) and then remove the BIOS battery for a few seconds. Put it all back and see if it will reload defaults.

 

[Patrick]

You didn't upgrade the firmware so its doubtful that you bricked it the same way they described in that blog post.

One thing worth a try before you give up: Most likely the BMC config data got corrupted. Try pulling the sled (so it is not powered) and then remove the BIOS battery for a few seconds. Put it all back and see if it will reload defaults.

That was my thought also. My bigger question is whether there is a BMC clear jumper on these. Installing one (maybe 2) of the 2.5" C6100's today. Also dba's idea to use the HP sleds seems to have worked on that! $20+ saved per sled!

 

[McKajVah]

Does the BMC heart beat LED blink?

What does the SEL (system event log) in Bios show? (45sec should be enough to check this).

There is also a "Bios clear CMOS" jumper and a "System reset" jumper you can try. (page 140 in the C6100 owner's manual) ftp://ftp.dell.com/Manuals/all-prod...ucts/poweredge-c6100_Owner's Manual_en-us.pdf

 

[Toddh]

Having problems with mine as well. I configured mine from the Bios and did not seem to have a problem until I went in and changed the IP, now getting BMC not found errors.

My board does not reboot at 45 sec. but it does show BMC now found.

My systems came with current firmware installed so I didn't have to update. I am looking for a way to re-flash the BMC from dos thinking this may help.



.

 

[Toddh]

A quick update.

Resetting the bios to defaults by choosing Load Optimal Defaults does not help.

Hard resetting the mb bios does not help. I pulled the node and remove the battery. This results in a VERY long boot time. Maybe an extended test mode during boot, even the AST Video post takes about 3 min. Still same error. It did complain about bios battery low so replaced the battery just to be sure.

However I believe I found a fix for the 45 sec reboots. My server started rebooting at short intervals, about 45 sec sounds right. I In the process of diagnosing I had inserted a USB drive. Going into Advanced Settings USB I turned of ECHI Handoff and this seems to have cleared up the 45 sec reboot.

Reflashed the mb bios from dos with 2.69 but does not help.

I am looking for a way to reflash the BMC.



.

 

[Patrick]

Does the BMC heart beat LED blink?

What does the SEL (system event log) in Bios show? (45sec should be enough to check this).

There is also a "Bios clear CMOS" jumper and a "System reset" jumper you can try. (page 140 in the C6100 owner's manual) ftp://ftp.dell.com/Manuals/all-products/esuprt_ser_stor_net/esuprt_cloud_products/poweredge-c6100_Owner%27s%20Manual_en-us.pdf

I saw that series of jumpers on the motherboard this weekend. Wonder if they would help?

 

[Toddh]

One thing to note, removing the battery and resetting the bios cleared settings in the bios for Name, Service Tag, and ePPID. I will have to locate how to reenter that info into the bios.

 

[shaneb383]

I've bricked two of them.... I've also bricked a BMC, but that is recoverable as long as the BIOS is ok.

The two i bricked:
1.) I was trying to update the Bios. I have several of these things with various Bios levels(1.05, 1.5, 1.62). When trying to update with the Dell Bios, i got an error that the xs23-ty3 was not compatible with the C6100 Cloud server Bios update. So i took a backup of one of my xs23-ty3 servers with BIOS 1.62, and then restored that BIOS to one of my servers at version 1.5. This bricked the BIOS, even though the BMC still works fine. I have a backup of the "old" BIOS, but i dont have anyway to restore it. I tried renaming to AMIBOOT.ROM and putting on a USB, but no luck..... which brings me to how i bricked the second server

2. Since i have no video on server 1, I have no idea if booting from a USB stick with AMIBOOT.ROM is even doing anything. I decided to try it on a working server, and i used a backup of THAT SERVERS BIOS FILE renamed to AMIBOOT.ROM. (This is the 1.62 ver). I booted the server with the USB, and i guess it did something because now that server is also bricked. It's "worse" than Server 1, even the BMC does not respond anymore.

I have spent several hours trying to recover both servers with no luck. I tried a serial connection, floppy image with AMIBOOT.ROM, USB drive with AMIBOOT.ROM, the various CMOS RST and SYS RST jumpers, nada. I even put in a secondary video card assuming the onboard video is dead with the Bios.

Looks like i can order a "blank" node for about $145... Hope this thread helps someone!

 

[shaneb383]

I reflashed the BMC by downlading - PEC6100BMC129.EXE . This containts 1.29 of the BMC firmware, and dos/windows based utilities for flashing it. I was able to recover a bricked BMC with this using the SOCFlash utility.

 

[Toddh]

I am thinking, although I do not know for sure, that the Nodes are tied to the Chassis and Service Tag. When I tried to move 2 nodes from one chassis to another it takes a long time to boot and the Name/Service Tag info shows up N/A in the bios and BMC shows unavailable.

I wonder if you have to identify a node with the chassis/Service Tag before the BMC will function. Anyone know how to set the Name/Service Tag/ePPID in the bios?

Thanks for the info on the BMC flash.



.

 

[dba]

I will know in about a week, but I may have two "extra" C6100 boards to sell - and for far less than $145!

I've bricked two of them.... I've also bricked a BMC, but that is recoverable as long as the BIOS is ok.

The two i bricked:
1.) I was trying to update the Bios. I have several of these things with various Bios levels(1.05, 1.5, 1.62). When trying to update with the Dell Bios, i got an error that the xs23-ty3 was not compatible with the C6100 Cloud server Bios update. So i took a backup of one of my xs23-ty3 servers with BIOS 1.62, and then restored that BIOS to one of my servers at version 1.5. This bricked the BIOS, even though the BMC still works fine. I have a backup of the "old" BIOS, but i dont have anyway to restore it. I tried renaming to AMIBOOT.ROM and putting on a USB, but no luck..... which brings me to how i bricked the second server

2. Since i have no video on server 1, I have no idea if booting from a USB stick with AMIBOOT.ROM is even doing anything. I decided to try it on a working server, and i used a backup of THAT SERVERS BIOS FILE renamed to AMIBOOT.ROM. (This is the 1.62 ver). I booted the server with the USB, and i guess it did something because now that server is also bricked. It's "worse" than Server 1, even the BMC does not respond anymore.

I have spent several hours trying to recover both servers with no luck. I tried a serial connection, floppy image with AMIBOOT.ROM, USB drive with AMIBOOT.ROM, the various CMOS RST and SYS RST jumpers, nada. I even put in a secondary video card assuming the onboard video is dead with the Bios.

Looks like i can order a "blank" node for about $145... Hope this thread helps someone!

 

[Toddh]

Made some progress. Have all three C6100 nodes that were giving me issues fixed.

Symptoms included BMC not found errors. Not being able to access the IPMI Settings from bios, long post times, Name and Services Tag missing from bios.

1. Changing the ECHI Handoff under Advanced Settings > USB corrects the rebooting at 45 sec.
2. Re flashing the BMC with the SOCFlash Util that comes with the Floppy firmware fixes the BMC not found issues. Used a USB to boot into dos and run the SOCFlash utility flash8.bat- Thanks Shaneb

The Name and Service Tag not showing in the bios is a result of the BMC firmware corruption, and not moving nodes to another chassis or resetting the bios to Optimized Defaults. Once the BMC is corrected the Name and ST showed up again and the nodes are booting fine.

So the moral of the story is NEVER Set the IPMI Lan IP in the bios. Get the IP from the bios or other means and configure it from the IPMI remotely.

 

[markwo]

I "bricked" two of my nodes tonight, but thanks to the posts in this thread knew I could quickly recover them.

I didn't configure the IP address in BIOS on either node, but I did configure the IP address via IPMI while the BIOS menu was running on each system (right after I viewed the IP address in the BIOS menu).

I also had one node that had an AMI Megarac branded BMC web interface, I thought that updating the firmware would update it to the Dell branding. When it didn't, I was pretty confused. While working with other settings, I power cycled the unit a bunch of times and changed its IP address and then it turned to the Dell branding seemingly by itself. My only theory is the software was cached in my browser and it reloaded when it hit the new IP address.

Anyway, I was very thankful for the info in this thread, especially the USB ECHI setting.

 

[Patrick]

I "bricked" two of my nodes tonight, but thanks to the posts in this thread knew I could quickly recover them.

I didn't configure the IP address in BIOS on either node, but I did configure the IP address via IPMI while the BIOS menu was running on each system (right after I viewed the IP address in the BIOS menu).

I also had one node that had an AMI Megarac branded BMC web interface, I thought that updating the firmware would update it to the Dell branding. When it didn't, I was pretty confused. While working with other settings, I power cycled the unit a bunch of times and changed its IP address and then it turned to the Dell branding seemingly by itself. My only theory is the software was cached in my browser and it reloaded when it hit the new IP address.

Anyway, I was very thankful for the info in this thread, especially the USB ECHI setting.

First off, great to hear everything works for you. Second Dell branded IPMI? Really? All of mine have been AMI branded. Have a screenshot?

 

[Chuck]

Man, was I glad to have found this post too. Thanks for sharing, it helped me in recovering two bricked boards. As soon as we quit getting snow I can move my rack from the garage down to my basement and begin setting up my VM lab.