Dell C6100 XS23-TY3 2U 4-Node (8 CPU) Cloud Server

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Fzdog2

Member
Sep 21, 2012
92
14
8
Dang, all I have is a 9211-4i and a 9750-4i with SAS ports at the back which hit the CPU heatsink.
 

Rain

Active Member
May 13, 2013
276
124
43
Does anyone know what the PEF Action sensor/alert is in the BMC? Sometimes when I boot the system, it immediately has a status of "PEF Action" and all the lights (even on powered off nodes) flash amber (running nodes alternate).

Occasionally, when I fire a node up, this does not happen and the lights remain green for some amount of minutes, and then the PEF Action sensor switches from "Normal" to "PEF Action" and the lights switch to amber again.

Any ideas? Google has failed me (Silly Google thinks PEF means PDF, even with -pdf) and the manual doesn't offer that much information, unless I missed it.
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
Does anyone know what the PEF Action sensor/alert is in the BMC? Sometimes when I boot the system, it immediately has a status of "PEF Action" and all the lights (even on powered off nodes) flash amber (running nodes alternate).

Occasionally, when I fire a node up, this does not happen and the lights remain green for some amount of minutes, and then the PEF Action sensor switches from "Normal" to "PEF Action" and the lights switch to amber again.

Any ideas? Google has failed me (Silly Google thinks PEF means PDF, even with -pdf) and the manual doesn't offer that much information, unless I missed it.
PEF Action refers to a "platform event filter" being triggered. It is a core concept of IPMI-based management. See pages 65ff of this document for more information.

In the case of the C6100 it means one or more BMC alarms have been triggered, and the flashing yellow light on the node indicates which node raised the alarm. Usually this is a sensor alarm. Log into the BMC and look at the "Event Log" under the "Server Health" tab. For a more real-time look with specific data open the "sensor readings" tab and find the sensor with an abnormal status.
 

Rain

Active Member
May 13, 2013
276
124
43
PEF Action refers to a "platform event filter" being triggered. It is a core concept of IPMI-based management. See pages 65ff of this document for more information.

In the case of the C6100 it means one or more BMC alarms have been triggered, and the flashing yellow light on the node indicates which node raised the alarm. Usually this is a sensor alarm. Log into the BMC and look at the "Event Log" under the "Server Health" tab. For a more real-time look with specific data open the "sensor readings" tab and find the sensor with an abnormal status.
Thanks for the quick reply! I checked the sensor tab and two strange things stand out: ACPI Pwr State is "Legacy ON State" and "PSU 1 AC Status" and "PSU 1 Present" are both "Not Available". (Link to screenshot)

Something to note: When I first powered on the server (one node w/ Win7; the rest off), I had no amber lights and the BMC did not report any PSU information either. I never noticed ACPI Pwr State until now; maybe this is the problem?

Also, I did swap the fans out with the Evercool fans you mentioned using the the Taming the c6100 thread. The lowest I've seen the fans spin is 1900RPM (~100RPM less than spec), so they shouldn't be triggering the Low-Critical RPM alarm @ 1500RPM. Currently, I have the threshold lowered to 850RPM, but nothing has changed.
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
ACPI Pwr State of "Legacy ON State" is normal.

If PSU 1 is inserted and plugged in then "PSU 1 AC Status" should be "normal" and "PSU 1 Present" should say "Presence Detected". If you have two PSUs you might want to pull PSU 1 and run it only on PSU 2 to see if things change.

If that doesn't narrow it down then look in the event log. It should list the specific event that raised the alarm. If there are too many events listed just clear the log and power off (all the way off - unplug both PSUs). If the alarm returns you'll see the power light go flashing amber. Look in the event log again and you should see the alarm.
 
Last edited:

Rain

Active Member
May 13, 2013
276
124
43
Interesting, apparently two of the fans have triggered the lower-critical alarm. Not quite sure how this is possible given that I monitored the fans right at boot time through the BMC (and ipmitools) and they immediately jump to 1900RPM.

Clearing the log has stopped the amber lights from blinking. The dates on the alarms seem to suggest that they were triggered a while ago (at least a few reboots while I was trying to diagnose the problem). Do the amber lights stay flashing until the user has "acknowledged" the log by clearing it?
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
Normally the flashing amber will clear on its own if the alarm clears - e.g., if the fans return to a non-alarm speed. Don't know why clearing the log turned off the flashing lights, but glad it cleared it. Hope it stays clear for you.
 

pyite

New Member
May 15, 2013
9
1
3
You have to populate both CPUs to use all 12 DIMM slots in a node. With only one CPU you can only use 6 DIMMs.

Max is 12x 8gb per node when using X55xx or L55xx CPUs.

FWIW I was able to use 8x16GB with a single L5520 CPU. I haven't yet tried two 5520's but I'll report the results here if/when I do. This was an XS23-TY3.
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
FWIW I was able to use 8x16GB with a single L5520 CPU. I haven't yet tried two 5520's but I'll report the results here if/when I do. This was an XS23-TY3.
I think you need to double check that you actually see and can use all of the memory you have plugged in. You likely have a system that POSTs and boots but only actually "sees" 48gb of the 128 you've got installed.

Two problems. One is a physical interconnection problem. The memory chips associated with the second socket are only wired to the 2nd socket. The CPU in the first socket can only access them via the QPI lanes that interconnect the two CPUs. So it is impossible for CPU 1 to make use of any more than the 6 DIMM sockets it is connected to unless there is a second CPU active in the CPU 2 socket.

Second, the Xeon 55xx chips don't ever see more than 8gb/DIMM.

If you really have it working I'd love to see screenshots confirming the memory configuration and CPU type...
 

Jaknell1011

New Member
May 14, 2013
18
0
0
Looking to run ESX off of 3 nodes using USB flash drives for the ESX install. Can someone tell me if there are any internal USB ports in these nodes? If not, are there any headers available on which I could put one of these:

StarTech USBMBADAPT2 Adapter - Newegg.com

I have seen this page:

http://forums.servethehome.com/processors-motherboards/1638-c6100-extender-board-usb-2.html

but I'm not sure if this is only relating to the extender board. Any info on internal USB ports or headers is appreciated.
 

Jaknell1011

New Member
May 14, 2013
18
0
0
You found it - the c6100 extender board USB thread above has the information that you need.
So no onboard internal USB ports or headers, the only option is soldering our own? Kind of a bummer there. I'm really not too keen on experimenting with soldering on one of these right out of the gate. Oh well.
 

Jaknell1011

New Member
May 14, 2013
18
0
0
No need to mess with the motherboard - there is a connector already in place. The only soldering you'll need to do is one cable. See post number twenty four by chuckleb.
The suspense is killing me, but your pic is dead...

...or is it just me?

**Scratch that, it's me** :eek: I will take a look at this in a bit, when I am not within the confines of my firewall. :rolleyes:

Thank you!
 
Last edited by a moderator:

pyite

New Member
May 15, 2013
9
1
3
If you use more than 3 DIMMs/CPU and the DIMMs are quad-rank (most RDIMMs are) then memory speed will be reduced to DDR3-800. Some people have reported that using Dual-Rank RDIMMs avoids this speed downgrade, but dual-rank are hard to find at higher capacities.
I have 12x 16GB of dual rank memory in a set of C6100's and XS23-TY3's, and they all run at 1333 mhz. The memory isn't hard to find at all, it just costs more.
 

gtallan

New Member
Apr 25, 2013
12
0
1
Minnesota
Drive activity LEDs without SGPIO

Like many here I've been working on re-cabling a C6100 to have two nodes connected to 6 drives each. I have one node with the LSI 1068E mezzanine card and the other with the correct Dell cable to connect all 6 on-board SATA ports to the midplane.

Since the Dell-specific cables for this config from midplane to backplane (HJ6F0 and 334VV) are basically unavailable or obscenely overpriced, I got a couple of the monoprice mini-SAS to 4xSATA cables referenced earlier in the thread (product id 8186), but I was really unhappy with the idea of having no drive activity lights.

Well... on the backplane there is a jumper labelled "LED control". No description of what that does, or apparent google hits about it. On a whim, I enabled that jumper, and... I now have activity LEDs on all drives. Seems to work fine for drives connected to onboard SATA or the LSI controller. Presumably it means the LED is controlled by the drive itself rather than the controller...

Maybe this is widely known already, but I hadn't seen anything about it.

I *had* hoped to re-use the standard Dell mini-SAS to 3xSATA cable for drives 5-6 on each node, but it's really too short to route it sensibly. probably better just to get a couple more of the 4-way monoprice cables for this purpose.

Hope someone finds this useful.

Graham
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
Drive activity LEDs without SGPIO

Like many here I've been working on re-cabling a C6100 to have two nodes connected to 6 drives each. I have one node with the LSI 1068E mezzanine card and the other with the correct Dell cable to connect all 6 on-board SATA ports to the midplane.

Since the Dell-specific cables for this config from midplane to backplane (HJ6F0 and 334VV) are basically unavailable or obscenely overpriced, I got a couple of the monoprice mini-SAS to 4xSATA cables referenced earlier in the thread (product id 8186), but I was really unhappy with the idea of having no drive activity lights.

Well... on the backplane there is a jumper labelled "LED control". No description of what that does, or apparent google hits about it. On a whim, I enabled that jumper, and... I now have activity LEDs on all drives. Seems to work fine for drives connected to onboard SATA or the LSI controller. Presumably it means the LED is controlled by the drive itself rather than the controller...

Maybe this is widely known already, but I hadn't seen anything about it.

I *had* hoped to re-use the standard Dell mini-SAS to 3xSATA cable for drives 5-6 on each node, but it's really too short to route it sensibly. probably better just to get a couple more of the 4-way monoprice cables for this purpose.

Hope someone finds this useful.

Graham
Ah, thank you. Very helpful!
 

Mrlie

Member
Jan 1, 2011
32
2
8
45
Oslo, Norway
I got my C6100 about 2 weeks ago, but when I took it out from the box, I didnt see that the selfexpanding foam that was used, had "wrapped" around the front panel ears. This resulted in me breaking one of the panels when lifting the server out of the box, but thanks to ebay I found some new ones. Now I just wonder if anyone know where I can find a manual of how to replace those? I'd like to read up on the details, before I start to unfasten a bunch of screws.

Patrick: Partnumber is 21GY9 if you want to update the first post.

I'm also gonna have a ssd in each node, and I bought several Kingston SNA-DC/35 Harddrive enclosures, that works perfectly in the C6100 drivebays. Toolless design when it comes to inserting a 2,5" device, and you dont need to remove the C6100 drivecaddy to swap out a 2,5" device.


EDIT
Just replaced the left front panel ear, and it was not very complicated. You need to remove 5 screws on each side of the drivebays, and lift the drivebay out 1-2 inches to get access to the screws that hold the panel ears to the chassis. The cable runs in a grove on the side of the drivebay and across to the controll board.
/EDIT
 
Last edited: