Dell 3-Node AMD DCS6005

[spyrule]

As part of a conversation, it was noted that if you flash your BMC with the above method, you'll end up with a bunch of BMC's with the same MAC address. Here is the fix/resolution to that problem:

-----------------------
Yes, I discovered that after flashing, it would set the IPMI MAC address to the same MAC address on all servers.

If you browse to :

Dropbox - Files for bootable USB

You'll see a new folder I created called:

"Files for bootable USB"

grab that entire folder.
toss it on a bootable USB, and it should have ALL the tools you could need.
in the kcs folder there is a tool called IDNIC.exe (from intel) that will identify each network cards MAC address.

write them both down.

Now look at one of the affected sleds and its motherboard. You'll see 3 stickers placed in 3 different spots (under CPU0 heatsink, near the power, and between the two CPU heatsinks (or at least thats where they were for me)).
You'll notice that two of the MAC addresses match up to the ones on the screen, the one NOT listed, is your IPMI's ACTUAL MAC address.

I honestly cannot remember which tool worked, so try both.

first try it with the k.exe file and flash the new mac address (follow the syntax that is specifies), reboot and test, if same MAC, then try the KCSFLASH.exe file instead.

The one final step once you've rebooted, is to shutdown, and reset the BMC on the motherboard (do NOT need to reset the BIOS though).

-------------

 

[spyrule]

One other side note: Don't actually bother with the Noctua 80mm fans, they don't push enough air to keep the case cool enough. I ended up with a slow upward creep in temps until eventually sled 2 was getting critical temp levels for cpu0. I ended up switching to these fans:
Masscool FD08025B1M3-4 .
MassCool Cooling Fan for Computer Case (FD08025B1M3-4): Amazon.ca: Computers & Tablets

They arn't "quiet" fans by any means, but they bring the level down to a MUCH more reasonable level. They are NOT pwm though, so will always spin max, unless you run an independent fan controller.

They also keep my CPU's within a 45-50' temp range, so I'm happy with that.

 

[Lexi]

Has anyone been able to fix the problem where IPMI works, but the web interface isn't running? If so, how? I've been trying to get it working for a couple of hours now, but I am still apprehensive about flashing anything...

 

[spyrule]

Honestly, your best bet is to reflash with the above instructions. Just make sure to flash both BIOS and IPMI with the matched versions. This will reset any custom configurations you may have had, but it will guarantee that it will work (well mostly anyway). Last thing to do after flashing is setting your original MAC address for the IPMI interface.

 

[Lexi]

Thanks... Really didn't want to have to do that, but I'll give it a shot in a couple of days.

 

[MRose34]

June 2015 Update: Just to keep these things going, some of us are just now playing around with these nodes as they are still around. I have successfully installed ESX 6.0 on all three blades to internal USB without issues. Thanks for everyone's hard work on this but just wanted people to know as ESX 6.0 is now out, it works just fine on these units.

 

[MRose34]

Edit: turns out I could not get these onboard nics to play nice with my L3 HP switch no matter what I did. The HP switch would only work at 100MBs Half Duplex...so I just put in some real cheap Trendnet L2 switches between the servers and the L2 switch and now everything is happy at 1GB. Works for me


What in the world were the BIOS settings you used to get these nics to work at 1GB? I could use some help here as many of us are still having these issues as we are just discovering these units

Casbar/Javi/All:

After much pain, I FIXED THE ISSUE WITH THE GB NICs. Unfortunately, these DCS6005 units do come with custom BIOS configurations. For reasons I do not understand, there are certain settings in the BIOS that MUST NOT be touched including very specific settings in the Chipset configuration. In fact, setting values to "Auto" rather than selecting specific values will result in disabling components (such as NICs), as I discovered the hard way. After going through each setting one at a time (including the GPP3a Core settings - anyone know what this is or does???), I was able to get my GB NICs to work again. As a result, I validated that this IS NOT an Intel driver issue but, instead, a BIOS configuration issue.

Ultimately, I think my NIC issue was the result of a specific configuration that Dell built for a customer where the BMC & IPMI were using the NICs and were not allowing the NICs to be used for any other purpose (i.e. locking them in place). Since the IPMI in these units absolutely sucks (it isn't worth the time & effort to set up this crap), I typically start configuring a new server by clearing out any custom BIOS entries and starting with a baseline configuration. While most competent asset management firms will do this before selling a used or refurbished server off of eBay, unfortunately the eBay seller that I purchased my server from (loc: New Hudson, MI) either forgot to or purposefully did not clear the custom BIOS configuration for this server. The result: 2 full days of pain.

When I have more time this weekend, I will post pics and describe the BIOS configuration settings that I changed in order to get the server to work properly. I will say that, once you get through the pain, running Hyper-V Server 2012 R2 on this server is a beautiful thing. It just takes a lot of time to get there.

 

[MRose34]

Thanks for posting this, one of mine just crapped out and I am going to follow your lead on this.

Ok, this is annoying... but awesome at the same time.

Fixed my BMC error saying "BMC NOT RESPONDING"

So, while I was looking around for some command line options for KCSFLASH for pulling a copy of the existing BMC on a good/working sled, I came across a thread that had
mentioned that the user had gone to ASPEED's website (the BMC creator). I then downloaded the SOCFLASH tool from here : Driver Downloads：ASPEED 信驊科技

The trick is to download the LATEST version of the SOC tool. Install it on a bootable USB key. Also, if you havn't already, you MUST also flash your
BIOS to the same/matching version (the two files came together previously).

Also, while at it, copy the BMCBACK.ROM file from the link posted near the start of this thread. Once, I flashed that rom (which is a known good BMC copy), powered the sled off, rebooted, suddenly my BMC started to work perfectly again! Woohoo!

I'll be linking a copy of the BIOS/BMC and tools from my dropbox shortly, I just have to wait for a large file to finish syncing first.

 

[asshopo]

I 've seen a couple reports of the blade 2 running hot, but I actually had an issue after swapping out the "jet 9k" fans with some desktop 3K fans that the SR5650_TEMP on blade 1 would stay around 90C without any load on the server, and with a little load, it'd hit the 98C mark and shutdown. I ended up buying a couple 30x30x10mm fans and attached those to the heatsinks of the SR5650s, and now I'm staying down below 60C, which is a great improvement. I used the power header next to the memory, and so far it looks like it's going to work out well.

View attachment 65

Is there room for the 30x30x10mm fan + a riser card? I have a SuperMicro hd controller plugged into the riser, which is causing the sr5650 chip to heat up.

 

[MRose34]

Is there room for the 30x30x10mm fan + a riser card? I have a SuperMicro hd controller plugged into the riser, which is causing the sr5650 chip to heat up.

I am going to say no. There was literately zero room for anything once I put in an add in HBA card. It sat right above that heatsink and caused the unit to over heat since I have replaced the fans with lower speed fans. I tried to place a 10MM fan in the chassis between the card and the heatsink and it would not fit. If you are running stock fans you will probably be fine but not if you replace those fans.

 

[asshopo]

I am going to say no. There was literately zero room for anything once I put in an add in HBA card. It sat right above that heatsink and caused the unit to over heat since I have replaced the fans with lower speed fans. I tried to place a 10MM fan in the chassis between the card and the heatsink and it would not fit. If you are running stock fans you will probably be fine but not if you replace those fans.

I'm running stock fans and it overheats and shuts down already. I'm one of the few that don't care about noise but the fans run at 3500 rpm and wont go up at all. Really would be nice if the BMC allowed for fan speed settings . On a sled with no riser card, it runs at 90c idle and on the one with the riser card it runs 94c idle.

edit: I care if its max rpm, but wouldn't mind if it was a little louder for the sake of cooling.

 

[spyrule]

Is there room for the 30x30x10mm fan + a riser card? I have a SuperMicro hd controller plugged into the riser, which is causing the sr5650 chip to heat up.

It depends on the card tbh. If the HDD controller's ports are on the back (like many SM hdd controllers are), then definitely no. If it's on the top, and there is a little room (ie no heatsink at the back edge of the card), you can offset mount the 30x30 and make it fit. I did exactly this on my quad NIC card and it works perfectly fine.

 

[spyrule]

I'm running stock fans and it overheats and shuts down already. I'm one of the few that don't care about noise but the fans run at 3500 rpm and wont go up at all. Really would be nice if the BMC allowed for fan speed settings . On a sled with no riser card, it runs at 90c idle and on the one with the riser card it runs 94c idle.

edit: I care if its max rpm, but wouldn't mind if it was a little louder for the sake of cooling.

The BMC Fan controller IS pwm based, so as long as your fans are PWM, then they should spin up/down as needed. If not, then it sounds like they arn't the stock fans.

 

[MRose34]

I'm running stock fans and it overheats and shuts down already. I'm one of the few that don't care about noise but the fans run at 3500 rpm and wont go up at all. Really would be nice if the BMC allowed for fan speed settings . On a sled with no riser card, it runs at 90c idle and on the one with the riser card it runs 94c idle.

edit: I care if its max rpm, but wouldn't mind if it was a little louder for the sake of cooling.

Something must be wrong then. My stock fans idled at 5200 rpms and they did spin up and down according to heat. My replacement fans spin at 3600 and they also spin up and down. I have also flashed the BMC controller following someone's post and files on this forum. My temp on that chip is 79-80 on idle with the replacement fans. This is dumb question but did you check to make sure fans are installed correctly and connected correctly?

 

[spyrule]

Id reflash your bmc after checking the fans and connectors. The instructions are earlier in this thread from me.

 

[asshopo]

The BMC Fan controller IS pwm based, so as long as your fans are PWM, then they should spin up/down as needed. If not, then it sounds like they arn't the stock fans.

They are really loud when the BMC boots after plugging it in. After the BMC boots, they settle down and never seem to go up, tho I haven't been in front of it while putting all 3 nodes under load to test it. So they do change RPM, just not for that 1 chip it seems.

 

[asshopo]

The BMC Fan controller IS pwm based, so as long as your fans are PWM, then they should spin up/down as needed. If not, then it sounds like they arn't the stock fans.

They do, but as I just posted above this reply, the temp of that particular chip doesn't seem to trigger the fans speeding up. I need to put all 3 nodes under load and see if the fan's all speed up for that. Might be an overall temp of the case or CPUs directly vs the 1 chip that's overheating.

 

[asshopo]

Id reflash your bmc after checking the fans and connectors. The instructions are earlier in this thread from me.

Knowing that they sound like jet engines on BMC boot and that they settle down, is this still necessary? I'm not normally worried about flashing bios/firmwares but I know that this particular set of hardware can be cranky.

 

[MRose34]

Knowing that they sound like jet engines on BMC boot and that they settle down, is this still necessary? I'm not normally worried about flashing bios/firmwares but I know that this particular set of hardware can be cranky.

Knowing that your fans only idle at 3600 or something low like that seems to make me thing something is wrong especially since my stock fans idle at 5400. Also, you mention that chipset temp is at 90 degrees where mine is 79 degrees with replaced lower airflow fans. I would certainly work on flashing the BMC. I have flashed mine a couple of times but these BMC controllers are crap. I moved my server from one location to another and one of the BMCs stopped working. I am going to flash again but something doesn't seem to add up with your temp and fan speed. Knowing these things are just north of junk, makes me feel like I will be flashing the BMC every so often. . Other than the BMC being buggy, everything else is good so the server is still worth the money.

 

[spyrule]

The bmc responds to cpu temps, not chipset temps. With your temps so high on that chipset, i'd buy some good silver thermal transfer grease, pull the heat sink (you have to remove the mobo to do this mind you), and remove the old thermal compound (mine was dry and cracked), and re-apply the new thermal grease and see if that makes a difference. I'd do the chipset and both cpu's while you are at it. It dropped my temps by almost 10 degrees.