Dell 3-Node AMD DCS6005

[Thanos]

I actually went through much of this myself back in February (see my posts) and I sympathize with your ordeal. I ended up getting a bare-bones server from the seller on EBay just to use as a parts server. I replaced the problem node with one of the nodes on the bare-bones server back in March and it has worked well for me since. It is very useful having this home lab setup with these servers. I will say I have had to replace 2 hard drives already and I just lost a Baracuda ES2 750GB hard drive today. The first 2 losses were on the same drive bay but the loss today is on a different drive bay. I guess I do expect to loose a few drives occasionally when trying to run a home data center (smile). Having the data center though has greatly helped me to learn a lot about networking, Linux, and VMware. So I guess it has been worth the setbacks that I had when I first started back in February.

Yeah, I read about that The guys that sell this were kind enough to replace the failed node (since it was in the very first days of getting the server). The replacement runs happily So I'm a happy customer of theirs! Kudos to them for the support! I don't -yet- have any failed disks but I do expect some to fail I have the 1TB disks. But I don't run the box all the time, only when I want to 'play' with the VMware stuff
Still no luck with VSAN though... I will probably get the disks in an external server as NFS/iSCSI and get over the lost disk space in the box...

 

[Thanos]

Anyone experience blades just halting? I am fairly certain it is not OS related as all 3 blades are running esxi 5.5 and only blades 1 and 2 are affected, 3 stays up. I enabled remote dump collection to see if it was ESXi purple screening, but it is not. The blade just stops and it only seems to happen overnight.....

Do you mean that blades 1 and 2 halt with no error output logged in ESXi? what is the IPMI status on power? (if any...) It may shutdown on idle or high temp??
I have disabled all power mgmt in BIOS and ESXi (from balanced -> high performance). Maybe the BIOS gives you a clue? (without proper docs it's a guessing game for all)
I have not left the server on for more than a working day. I turn off everything before going to sleep or when I don't need it anymore. I guess I'm just lucky on this one?

 

[Thanos]

I ended up swapping the fans with dell optiplex sanyo denki san ace 80 pulls. The are currently running at 3k RPM with all three systems running. Blade 2 is the hottest running 50C under moderate load in an air conditioned but still warm room. System is much quieter.

I was also able to get VMware Distributed Power Management to power back on a server through the IPMI interface, so I'll let DPM power one off when it doesn't need it.

Now this is something I would like to see a link Did you buy them from ebay/amazon etc? If you have the link, could you please share? 3K rpm is very low! I recall starting around 4500 then jumping above 5K rpm (ok, in a normal room with no a/c). Blade 2 run hot for me as well, I guess because of having a blade on top and another at it's side, doesn't help. But then, I would assume it's poor design on airflow? Again, I'm talking about no a/c in the room.
But I have to admit... the noisiest part is the Dell 6224 switch! It's like a leaf blower!! (1RU and so much noise...)

 

[Thanos]

That is very similar to the issue I was having on one of the physical nodes when trying to install vSphere 5.5 back in February. I was indeed able to finally get it to work with a link suggested in one of the earlier posts: Dell CS24 ESXi 5.5 Install Stuck "Relocating modules and starting up the kernel..." | RobWillis.info

I had to interrupt the installation process to specify a config setting to continue and get through the esxi 5.5 installation. I am running vSprere 5.5 and vCenter on all 3 nodes of one of these servers.

the "ignoreHeadless=TRUE" at the ESXi boot process wouldn't fix a broken BIOS (I guess). You should either find a way to re-flash the BIOS or ask for a replacement of the board (if it was broken from day 1).

 

[Thanos]

The HD bays are each wired in sets of four to the nodes, and the cables are super easy to reroute if you want to put all 12 to the same node. Each board has 6 SATA ports, so with a card and a riser it'll be easy to centralize the storage. I was debating doing this myself before I decided to embark on this silly build.

That being said, I ordered a new Seasonic 650W with a single 12V rail (don't want any circular current...) and wired up a generic adapter cable that can go on any PSU

I have a few more pictures including the chipset waterblocks installed over here

SIR, YOU ARE AWESOME!! I've never used water-cooling and I'm reluctant to experiment with this But you have a nice project there! You can create a very powerful/custom server and without the fan noise that goes with the chipset. It's one of my concerns with the blades, the high temps on the SR5650 and even the cpus... Water cooling must be the best thing for them (without the need of an A/C room). Well done!

 

[totalanni]

How do i use the ikvm with these?

 

[kdieiev2013]

Hi all, could somebody please post procedure for powering up TYAN S8208 motherboard outside the server case of DELLDCS6005, as I observe from the thread:
pin 1-8 -- 12V
pin 18 -- 5V
pin 11-17,19 --GND

is I need pin 20 to be used for any purposes or board start after power is connected?
is pin 18 (5V) is mandatory to start, could I use only 12V and GND wires to start?
what is the other important pins on motherboard for using outside of the case(like PWR ON, RESET, etc)?
is power drain about 150W@12V per motherboard in load?

thanks a lot,
I am considering to buy a couple of this boards without cases, equipped with 2387 Opteron, 32 GB RAM

 

[Jeremy]

How do i use the ikvm with these?

I was able to get the JViewer to work this morning by saving the jviewer.jnlp file, hacking it to correct broken XML tags, and then opening it up by double-clicking it in Nautilus. It worked under IcedTea, but I only just navigated the boot menu and booted the box, logged in, and shut down the node. I haven't tried redirecting ISO image yet.

 

[Lichtwald]

How do i use the ikvm with these?

So on the most recent box I bought (from the same seller) the Web IPMI/iKVM interface doesn't seem to be running. A port scan shows nothing at expected ports, and navigating there in a browser just errors out. I presume this is related to what you were facing totalanni.

Any input from anyone?

 

[Jeremy]

So on the most recent box I bought (from the same seller) the Web IPMI/iKVM interface doesn't seem to be running. A port scan shows nothing at expected ports, and navigating there in a browser just errors out. I presume this is related to what you were facing totalanni.

Any input from anyone?

Have you tried both ethernet ports on one host? Do you have it plugged into a LAN with a DHCP server that will assign it an IP? Did you use the BIOS setup via VGA+keyboard console to make sure the BMC is sharing the primary NIC and using DHCP?

 

[justin]

Hello everyone. OK so I have a problem with this server reliably connecting to my switches, I have a cisco sg200-26 gig port, and it is not linking up when I plug the nodes in. if I change the port speed from 1000m to 100m its magically comes online? Anyone come across this before?


Thank you.

 

[Lichtwald]

Have you tried both ethernet ports on one host? Do you have it plugged into a LAN with a DHCP server that will assign it an IP? Did you use the BIOS setup via VGA+keyboard console to make sure the BMC is sharing the primary NIC and using DHCP?

Alright, some updates on this. It has been kind of a pain since my two systems are in different locations.

I confirmed that all the bios settings are the same between the two (working iKVM and nonworking iKVM.)

The non-working one gets an IP address from the DHCP server, but, still nothing on port 80 (or other common http ports according to port scans.) Tried with static, still no love. So I feel like this is going to be a question for the ebay seller...



Justin, as for your question, I've been running into that on some switches as well with eth0, but the issues go away after the machine is turned on (and eth1 never has had issues)

 

[Alan Mattson]

Now this is something I would like to see a link Did you buy them from ebay/amazon etc? If you have the link, could you please share? 3K rpm is very low! I recall starting around 4500 then jumping above 5K rpm (ok, in a normal room with no a/c). Blade 2 run hot for me as well, I guess because of having a blade on top and another at it's side, doesn't help. But then, I would assume it's poor design on airflow? Again, I'm talking about no a/c in the room.
But I have to admit... the noisiest part is the Dell 6224 switch! It's like a leaf blower!! (1RU and so much noise...)

I 've seen a couple reports of the blade 2 running hot, but I actually had an issue after swapping out the "jet 9k" fans with some desktop 3K fans that the SR5650_TEMP on blade 1 would stay around 90C without any load on the server, and with a little load, it'd hit the 98C mark and shutdown. I ended up buying a couple 30x30x10mm fans and attached those to the heatsinks of the SR5650s, and now I'm staying down below 60C, which is a great improvement. I used the power header next to the memory, and so far it looks like it's going to work out well.

 

[naecoblue]

I recently picked up one of the servers for an ESXi lab. The server I purchased didn't come with RAM. Can someone give me a model number or the specs of the RAM that was included with your server? I can't find anything online, and what I had on hand isn't working. I'm thinking it should be PC2-6400P.

 

[kdieiev2013]

Hello everyone. OK so I have a problem with this server reliably connecting to my switches, I have a cisco sg200-26 gig port, and it is not linking up when I plug the nodes in. if I change the port speed from 1000m to 100m its magically comes online? Anyone come across this before?


Thank you.

hi, YES, I confirme that I have simular issue too, only static speed of 100 is working, if anybody knows solution please post here

 

[Mike]

hi, YES, I confirme that I have simular issue too, only static speed of 100 is working, if anybody knows solution please post here

Ethtool could show you what options the NIC actually supports in it's current state.

 

[kdieiev2013]

Guys can you share your jumper setting config which your motherboards have, is it possible to disable BMC|IPMI for clear NIC operation
When I changed speed to 1000 in esxi VMWARE via VSphere, I lost my connection at all,
i need to reinstall esxi, reseting via esxi management interface for network config dont give any positive feedback

 

[kdieiev2013]

anybody trying update NICs eeprom, could you provide guidance, I download
http://downloadmirror.intel.com/12344/eng/BootAgent.zip
but cant understand how to use it

 

[kdieiev2013]

I observe strange issue, when I connect 2 ethernet ports on board I get 1Gbps link, I can use straight or crossover cable without any problem.
But I am unable to get it working with any switch at 1Gbps, only one working setting is 100Mbps/Half-duplex hardcoded on switch port,
as I understand I have some issues with EEPROM, I plan to use BOOTUTIL -DEFCFG,
when I manually set speed/duplex via ethtool nothing change,
Please tell me is it safe to manipulate with EEPROM, could anybody share with me working hex-DUMP, so I am truing to flash it,
I am install test windows x64, I think that may be after instaling Intel`s drivers I will get advanced features to set, but unfortunately
install program cant recognize that my mobo had Intel 82576 controller.
I saw before some people play already with net configuration, may be YOU can post it here, thanks

 

[unsobill]

Hello everyone, I also picked one of this bad boys.
Question is my ubuntu host complains that there's no VT enabled and I could not find any settings in BIOS on that.
Can I also upgrade CPU to X5650 on this platform?