Dell C6100 XS23-TY3 2U 4-Node (8 CPU) Cloud Server

Patrick

Administrator
Staff member
Dec 21, 2010
12,010
4,995
113
Decided to start a mini-thread as I play with this thing.

Getting a Dell C6100 XS23-TY3 inexpensively
I purchased the disk less unit for $1000 + 65 Shipping. Here is a link to the "deal" thread. Here is the configuration:
2x Intel Xeon L5520 (4C/8T 2.26GHz) 60w TDP* (*System and Maximum TDP is based on worst case scenarios. Actual TDP may be lower if not all I/Os for chipsets are used.)
24GB DDR3 Memory
1x 3.5" drive sled, wired for up to 3x drives/ node
2x Intel Gigabit LAN
1x IPMI

Cost is *updated* $200/ node.

Where to Purchase

For those looking for a good ebay search: Here is an ebay Dell XS23-TY3 search. These are now selling for about $800 for a four node unit including CPUs and memory.

Additional links from RimBlock
DELL-POWEREDGE-C6100-XS23-TY3-2U-4-x-HOT-PLUG-NODES-8-x-2-26GHz-QC-L5520-96GB

DELL-POWEREDGE-C6100-XS23-TY3-SERVER-8x-L5520-QUAD-CORE-CPUS-96GB-MEM-4x-TRAYS

Other deals are also available from various sellers supplying less ram, swapping the L5520 for L5420s or by offering only two nodes.

EBay search here.
A note on making offers: The C6100 3.5" option with 8x L5520's and 96GB are allegedly being dumped on the market off Dell leasing. Need to order in 500 unit quantities, but $700-800/ server is what I was told.

Dell C6100 XS23-TY3 or XS23-SB?

Why the Dell C6100 XS23-TY3 over the Dell C6100 XS23-SB? The Dell C6100 XS23-SB generally used the Xeon L5400 CPU. This is a quad core, Core 2 Duo generation Xeon. The Intel Xeon L5420 CPUs were 50w TDP parts which is good, but they used FB-DIMMs. Each Dell C6100 XS23-SB node maxed out at 64GB of RAM. Conversely, the Dell C6100 XS23-TY3 uses the LGA 1366 socket and can accept either Intel Xeon 5500 or Xeon 5600 series processors. With twelve DIMM slots per socket and 8 or 16GB DDR3 RDIMMs accepted per slot (depending on CPU series) the XS23-TY3 has a lot more memory capacity. Also, the Intel Xeon 5500 series introduced Hyper-Threading and the Xeon 5600 series introduced the 32nm process and six core chips.

Common configurations in the cloud deployments of the day were 2x Intel Xeon L5520 (4C/ 8T 60w TDP) per node (8 CPU total), 24GB RAM/ node (96GB total) and four 3.5" drives.

There are a ton of both types of servers on the market, but the Dell C6100 XS23-TY3 is going to have more expansion options.

Dell C6100 XS23-TY3 Chassis
There were two main chassis options, one with 12x 3.5" disks and one with 24x 2.5" disks. The one below has the four stock disk sleds plus four more (purchased at $20/ ea.)



From what I have heard, the alternative 2.5" disk chassis was commonly used with LSI 1068 or 1078 controllers.

Dell C6100 XS23-TY3 Power
Common installations used 1x 1100kw power supply. There are various power supply options and a second power supply can be utilized.

Power Consumption Test Configuration
8x L5520, 24x 4GB DDR3 RDIMM, redundant 1100w PSUs

Idle, 4x IPMI LAN, 8x Gigabit LAN conneted, 4x Intel 320 160GB, 2x WD Red 3TB: 174w
Folding@Home GROMACS 100% CPU load on all 8 CPUs: 489w (power saving enabled)

Dell C6100 XS23-TY3 Internals
Supermicro 2U Twin2 Motherboards in a Dell chassis. These motherboards can be "hot swapped" by pulling them out of the rear of the chassis, just like the power supplies.


Here is an overview shot:


Dell C6100 XS23-TY3 Management
IPMI similar to Supermicro/ Tyan
American Megatrends Based

Default IPMI Username and Password: root // root
Could not find the default login ID and password via searching so hopefully that helps someone.

Dell C6100 XS23-TY3 Sound
Measurements thus far on the Extech
Idle 66dba
100% Load Across 8x CPUs 77.4dba

Dell C6100 XS23-TY3 Expansion
Each Dell C6100 XS23-TY3 comes with a low profile PCIe 2.0 x16 slot on a riser. There is also a PCIe x8 mezzanine card slot with Infiniband and LSI controller options.

Dell C6100 XS23-TY3 Performance
Each dual Intel Xeon L5520 node has about the performance of an Intel Core i7-3770K overclocked to 4.2GHz in highly threaded applications. For example, each L5520 node in Folding@Home each dual Intel Xeon L5520 node does 27K to 30K ppd, about what an overclocked Core i7-3770K will do.

More to come.

Dell C6220 the C6100 Successor
These will cost more but here is a video describing the new C6220 upgrade over the C6100 generation.
[video=youtube_share;tz7R_tAA0QI]http://youtu.be/tz7R_tAA0QI[/video]
From what I have heard, Steve Croce (product manager for the series) left Dell in November 2012 and joined Calxeda. Makes sense as Calxeda has some hot new technology on their hands.

Dell C6100 XS23-TY3 Drivers

Here is a link to current Dell drivers. Most components are very standard.

Dell C6100 XS23-TY3 Spare Parts List
Rackmount Rail Kit - 0Y3DX1
Mid-plane Chassis Fan - 104TT
1.1kw PSU - XVKF0
Motherboard Tray - D61XP
Mellanox dual port QDR Infiniband Daughter Card - JR3P1
SAS Bridge Card - JKM5M
LSI SAS 1068e Daughter Card - Y8Y69
Intel 82599 10GbE Daughter Card - TCK99
3.5" SAS Backplane - V3X78
3.5" Drive Sled - 8TV68
2.5" SAS Backplane - H9TR4
2.5" Drive Tray - D273R

An alternative for the 2.5" chassis is to use the HP 2.5" drive sleds from the HP DL Series G5/ G6/ G7 generations. These run only about $3-4 each on ebay. Discovered by dba and confirmed (with notes on the EMI gasket and drive screws.)
Also additional drive sleds often do not come with screws. We are using the following screws Amazon.com: Supermicro Screw Bag and Label for 24x Hot swap 3.5-Inch HDD Tray Cable (MCP-410-00005-0N): Computers & Accessories to secure drives in trays.

More to come
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,010
4,995
113
Added a few for you. Snowboarding tomorrow.

Very happy to see the sub-430w load numbers as I was getting worried.
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,490
228
63
I want one now. Can these take Xeon L5640's? Very surprised how close in price the L5430 machines are. IIRC Core 2 to Westmere was huge performance and PPW upgrade.

May part one out.
8x Xeon L5520 * $65ea = $520
24x 4GB DIMMS * $20ea = $480
Power supply selling for $80ea
4x 3.5 trays * $20ea = $80
4x Fans * $35ea (ebay price) = $140

Can prob get $100 for each mobo easily and all of those other boards and cables prob $20-60 ea.
 

PigLover

Moderator
Jan 26, 2011
2,976
1,283
113
Very interesting...looks like a 2U mini-bladeserver.

Are the MBs "standardized" (at least within the SuperMicro lineup)?
 

MiniKnight

Well-Known Member
Mar 30, 2012
3,000
910
113
NYC
Wow.... So sub-110w for each system? Best part is can go over 32gb. This is way cheaper than even 4x Core I3s.

Is VTx and VTd support? Would be great if so. Get external controller and connect an extra disk shelf.

Great info. More than anywhere except Dell tech docs
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,490
228
63
I see a stealthy edit on the C6220 series and the product manager.
 

cactus

Moderator
Jan 25, 2011
829
77
28
CA
From the video, I am confused on the power savings for the new gen. He says it saves ~100w at full load? Maybe for equal core count, single E5-2650L vs dual L5520 per node.
Also, are those the elusive full height bracket for LSI9202 like cards in the 2 node? @4:40 and @11:11

@Patrick, the Idle line has "no drives" and then lists drives at the end. 174W for all 4 nodes idle is great.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,010
4,995
113
@Patrick, the Idle line has "no drives" and then lists drives at the end. 174W for all 4 nodes idle is great.
Ha! Edited the end, not the start.

Just installed 2x Xeon L5638's ($125 ea) in one of the nodes. Pulled the motherboard, swapped CPUs, walked to the desk and restarted the box. Awesome experience as it took 1 latch, a tug and then 4 screws for each heatsink. Much easier to swap mobo/ RAM/ CPU than traditional 1Us. Can leave all of the storage and power supply in place with this.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,010
4,995
113
Very interesting...looks like a 2U mini-bladeserver.

Are the MBs "standardized" (at least within the SuperMicro lineup)?
Not 100% sure, need to check the power connectors but that may be the difference. Also, the I/O shield is not replaceable so you would need a MB that fit properly.
 

xnoodle

Active Member
Jan 4, 2011
259
48
28
Were there holes for your 2.5" drives on the 3.5" sleds? (I assume so?)

And hard drives are fed off the motherboard (ICH?), not a add-in mezzanine card?
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,010
4,995
113
Were there holes for your 2.5" drives on the 3.5" sleds? (I assume so?)

And hard drives are fed off the motherboard (ICH?), not a add-in mezzanine card?
I had to use converters to mount the 2.5" drives. They are fed off the ICH.
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,490
228
63
Any idea why Unixsurplus/ mrrackables on ebay is so much more expensive? The other sellers seem to be moving tons of these things while they barely see any sales or offers. Their prices are much higher.
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,490
228
63
Any idea why Unixsurplus/ mrrackables on ebay is so much more expensive? The other sellers seem to be moving tons of these things while they barely see any sales or offers. Their prices are much higher.
Just to add on this. MrRackables is $700 with best offer for the barebones. Theirs has rails ($50), 4x drive caddy ($15/ea), but only 1 psu ($80 ea). Barebones with no rails, no caddy, but redundant psu... $225 on the forums.
 

cactus

Moderator
Jan 25, 2011
829
77
28
CA
I am sort of amazed they work. On Supermicro boards don't spicy chips normally lose support after the initial retail BIOS?