Dell 3-Node AMD DCS6005

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
Only way to learn about this stuff is by asking. In this case, I'd never thought that NUMA could cause any problems nor would I ever turn it off so that's why it sounded odd. I rebooted one of my C6100 test nodes to see what it does. This is dual socket with separate 3 sticks of memory per controller. This is also under CentOS, which is basically what ESX is running underneath, a Redhat variant. The OS itself is NUMA-aware. What you see is that the BIOS setting encapsulates each socket into a NUMA node so that it can properly map the memory and distances. Without this, it flattens it as one level deep so the OS cannot sort out what hardware you have.

So overall, I would recommend turning on NUMA if you've got it.

----

numactl --hardware:

NUMA Off in BIOS:
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 24566 MB
node 0 free: 23361 MB
node distances:
node 0
0: 10

NUMA on in BIOS:
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 8 9 10 11
node 0 size: 12278 MB
node 0 free: 11406 MB
node 1 cpus: 4 5 6 7 12 13 14 15
node 1 size: 12288 MB
node 1 free: 11599 MB
node distances:
node 0 1
0: 10 20
1: 20 10



Link to PDF versions
NUMA Disabled: http://postimg.org/image/838cbezvt/
NUMA Enabled http://postimg.org/image/5n6ixkht5/


I'm really just jumping into server virtualization and modern server know-how altogether. I was thinking that the NUMA option enabled would lock down an evenly distributed set of resources for each CPU node whereas the ESXI engine automatically distributes resources when and where needed and this ability is obstructed if NUMA is enabled. I certainly could have misinterpreted the information.
 

gmac715

Member
Feb 16, 2014
37
0
6
Only way to learn about this stuff is by asking. In this case, I'd never thought that NUMA could cause any problems nor would I ever turn it off so that's why it sounded odd. I rebooted one of my C6100 test nodes to see what it does. This is dual socket with separate 3 sticks of memory per controller. This is also under CentOS, which is basically what ESX is running underneath, a Redhat variant. The OS itself is NUMA-aware. What you see is that the BIOS setting encapsulates each socket into a NUMA node so that it can properly map the memory and distances. Without this, it flattens it as one level deep so the OS cannot sort out what hardware you have.

So overall, I would recommend turning on NUMA if you've got it.

----

numactl --hardware:

NUMA Off in BIOS:
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 24566 MB
node 0 free: 23361 MB
node distances:
node 0
0: 10

NUMA on in BIOS:
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 8 9 10 11
node 0 size: 12278 MB
node 0 free: 11406 MB
node 1 cpus: 4 5 6 7 12 13 14 15
node 1 size: 12288 MB
node 1 free: 11599 MB
node distances:
node 0 1
0: 10 20
1: 20 10



Link to PDF versions
NUMA Disabled: http://postimg.org/image/838cbezvt/
NUMA Enabled http://postimg.org/image/5n6ixkht5/
Thanks for providing the statistics you captured from your proof of concept. They are indeed helpful. I will enable the NUMA in the BIOS. What version of ESXi are you running might I ask?
 

M.Holder

New Member
Feb 5, 2014
9
0
1
Germany
More just curios.
Thinking of running the three nodes in a different chassis to make it silent.
Maybe I will put them upright in a 4u chassis with new airducts and 2u heatsinks.
I think I dont need the power of the Intels, the amds should be enough for my purposes..
 

kdh

New Member
Feb 6, 2014
4
0
0
I'm curious, did you attach the riser to the blade chassis? I eyeballed it on my C6005 and the chassis seemed to have the bracket for securing the riser right where the riser should be...

I might have a use to add another NIC to each blade, but not a pressing need at this point.
The screw holes in the board line up with the chassis so you could screw the board down if you wished. I did not, mostly because I did not have immediate access to my parts kits with the extra screws. It is a snug fit to begin with and with the RAID controller screwed in place, it's pretty solid.
 

Ken

New Member
Feb 10, 2014
49
1
0
I can post later this afternoon, working on a different server right now - Dell DCS FS12-NV7, my first Dell 'cloud' referb server... It came w/ dual Opteron 2419 EE, 64 Gig RAM, 12 HD trays (3.5") and ONE Dell PERC 6/i RAID controller, with only 8 of the 12 bays wired to the controller... Another cloud server cost savings measure? ;^)
 

lemieuski

New Member
Feb 15, 2014
7
0
1
I have been tracking down info on this server for the past few weeks and usually find myself reading servethehome.com forums more than anything else. Thanks everyone for the info, I got Windows 2012r2 and ESXi 5.5 installed on these blades because of the info on this form :)

Right now I'm having an issue though, no matter what I cannot get my NICS to link up. I've tried 3 different switches (Cisco SG300, Netgear, Linksys), NUMA on, NUMA off, and nada. Can someone post the bios settings for me ?
 

javi404

New Member
Jan 24, 2014
26
0
1
More just curios.
Thinking of running the three nodes in a different chassis to make it silent.
Maybe I will put them upright in a 4u chassis with new airducts and 2u heatsinks.
I think I dont need the power of the Intels, the amds should be enough for my purposes..
I am not sure this is going to work the way you are thinking. Its a standard chassis but all the wiring and harnesses are custom for these units. Lookup the Tyan S8208 server board. You will see it has a non standard power connector.
 

javi404

New Member
Jan 24, 2014
26
0
1
I have been tracking down info on this server for the past few weeks and usually find myself reading servethehome.com forums more than anything else. Thanks everyone for the info, I got Windows 2012r2 and ESXi 5.5 installed on these blades because of the info on this form :)

Right now I'm having an issue though, no matter what I cannot get my NICS to link up. I've tried 3 different switches (Cisco SG300, Netgear, Linksys), NUMA on, NUMA off, and nada. Can someone post the bios settings for me ?
Very interesting. I was told by the vendor about the NUMA setting some how being directly affecting the NICS although I am not sure how accurate that is.
They told me if the NICS don't show up in the OS, to turn on NUMA. Not sure I would want NUMA off anyway but I didn't have that specific problem.
My problem was like yours, that the NICS don't like my crappy trendnet switch. I also saw a post on the dell website of someone who dell refused to support, they couldn't get the lights to link on a dell switch.

Here is the very interesting part. My crappy trendnet switch (TEG-240WS (Version C2.0R)) with 2 shared SFP Ports.
I have to see if I can dig up the pics but I have a feeling it has to do with the chipset in the switch because I can only get the NICS to link with ports 21,22,23,24.
I have a feeling the shared ports (23,24) and the other (21,22) are sharing a different internal hardware than the rest of the switch. No other ports on this switch will link with the NICS.
Funny enough, crappy D-Link 8 port gigabit switch works just fine with the NICS.
Manually setting port speeds etc doesn't help. Its some kind of low level hardware issue still to be discovered. In the mean time it seems that only certain chipsets like the NICS on the DCS6005.

I am going to look inside both switches in about 15 min and see what I find. I want to solve this mystery because I am in the market for a nicer gige switch but don't want to get something this server doesn't like.
 

blitzsd

New Member
Feb 25, 2014
2
0
0
I have one of these DCS6105/DCS6005 servers and they have been great. I now have a need to upgrade from Windows 2008 R2 to Windows 2012 R2 and have searched the WEB for Dot Hill RaidCore VST 5100 drivers. I have tried everything that Tyan, Dot Hill, Dell and SuperMicro have for embedded Raid Controllers. The current 2008 R2 version is DriverVer = 09/16/2010,5.1.5.00020

ftp://ftp.tyan.com/drivers/Website%20Drivers/AMD%20-%20ATI/Storage/SATA%20RAID/Dot%20Hill/Windows/64-bit/SP5100_SATARAID_DotHill_v5.1.5_Windows_64.zip

ftp://ftp.tyan.com/drivers/Website%20Drivers/AMD%20-%20ATI/Storage/Management%20Utility/Dot%20Hill/Windows/SP5100_RAID-Management-Utility_DotHill_v5.1.5_Win_32-64.zip .

Has anybody figured out how to make the driver work for Server 2012 R2?

Thank you,

blitzsd
 

Ken

New Member
Feb 10, 2014
49
1
0
I have one of these DCS6105/DCS6005 servers and they have been great. I now have a need to upgrade from Windows 2008 R2 to Windows 2012 R2 and have searched the WEB for Dot Hill RaidCore VST 5100 drivers. I have tried everything that Tyan, Dot Hill, Dell and SuperMicro have for embedded Raid Controllers.

-- snip --

Has anybody figured out how to make the driver work for Server 2012 R2?

Thank you,

blitzsd
Are you sure the drivers are needed? I am running Windows Server 2012 R2 on my C6005 without issue, but I've not set a RAID array on my box.

The technology is widespread enough by now that the driver should/could be 'baked into' Windows Server 2012...

Have you tried starting the upgrade and seeing if the arrays are picked up by the installer?

Dot Hill's RAIDcore tech was very new when Server 2008 was first released, now it is a mature technology.
 
Last edited:

blitzsd

New Member
Feb 25, 2014
2
0
0
Ken,

I have setup a raid1 mirror of two 750GB drives in the Dot Hill utility pre-o/s install. With the BIOS set for RAID, Windows 2012 R2 won't even let you lay an O/S down without the controller and driver software.

Scott
 

Ken

New Member
Feb 10, 2014
49
1
0
Well I just checked, and to answer my own question you DO need a driver if you want to use Dot Hill RAID Core arrays under Windows Server 2012 R2.

This may be an opportunity to install an add-in RAID controller if you don't want to run with Software RAID...

Have you reached out to Broadcom, they apparently own Dot Hill now...
 

billwong

New Member
Feb 23, 2014
5
0
0
Dear All,

anyone of you can provide me with C6005 BMC/BOIS firmware? as i got issue with the current BMC which send out huge packets out e.g UPD brocast out, i want to re-flash the BMC firmware, but i cannot find any where i can download with. if anyone of you can provide a link to download, this will be perfect and great thank you!