1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Dell C6100 XS23-TY3 2U 4-Node (8 CPU) Cloud Server

Discussion in 'Processors and Motherboards' started by Patrick, Jan 19, 2013.

  1. Patrick

    Patrick Administrator Staff Member

    Decided to start a mini-thread as I play with this thing.

    Getting a Dell C6100 XS23-TY3 inexpensively
    I purchased the disk less unit for $1000 + 65 Shipping. Here is a link to the "deal" thread. Here is the configuration:
    2x Intel Xeon L5520 (4C/8T 2.26GHz) 60w TDP* (*System and Maximum TDP is based on worst case scenarios. Actual TDP may be lower if not all I/Os for chipsets are used.)
    24GB DDR3 Memory
    1x 3.5" drive sled, wired for up to 3x drives/ node
    2x Intel Gigabit LAN
    1x IPMI

    Cost is *updated* $200/ node.

    Where to Purchase

    For those looking for a good ebay search: Here is an ebay Dell XS23-TY3 search. These are now selling for about $800 for a four node unit including CPUs and memory.

    Additional links from RimBlock
    A note on making offers: The C6100 3.5" option with 8x L5520's and 96GB are allegedly being dumped on the market off Dell leasing. Need to order in 500 unit quantities, but $700-800/ server is what I was told.

    Dell C6100 XS23-TY3 or XS23-SB?

    Why the Dell C6100 XS23-TY3 over the Dell C6100 XS23-SB? The Dell C6100 XS23-SB generally used the Xeon L5400 CPU. This is a quad core, Core 2 Duo generation Xeon. The Intel Xeon L5420 CPUs were 50w TDP parts which is good, but they used FB-DIMMs. Each Dell C6100 XS23-SB node maxed out at 64GB of RAM. Conversely, the Dell C6100 XS23-TY3 uses the LGA 1366 socket and can accept either Intel Xeon 5500 or Xeon 5600 series processors. With twelve DIMM slots per socket and 8 or 16GB DDR3 RDIMMs accepted per slot (depending on CPU series) the XS23-TY3 has a lot more memory capacity. Also, the Intel Xeon 5500 series introduced Hyper-Threading and the Xeon 5600 series introduced the 32nm process and six core chips.

    Common configurations in the cloud deployments of the day were 2x Intel Xeon L5520 (4C/ 8T 60w TDP) per node (8 CPU total), 24GB RAM/ node (96GB total) and four 3.5" drives.

    There are a ton of both types of servers on the market, but the Dell C6100 XS23-TY3 is going to have more expansion options.

    Dell C6100 XS23-TY3 Chassis
    There were two main chassis options, one with 12x 3.5" disks and one with 24x 2.5" disks. The one below has the four stock disk sleds plus four more (purchased at $20/ ea.)


    From what I have heard, the alternative 2.5" disk chassis was commonly used with LSI 1068 or 1078 controllers.

    Dell C6100 XS23-TY3 Power
    Common installations used 1x 1100kw power supply. There are various power supply options and a second power supply can be utilized.

    Power Consumption Test Configuration
    8x L5520, 24x 4GB DDR3 RDIMM, redundant 1100w PSUs

    Idle, 4x IPMI LAN, 8x Gigabit LAN conneted, 4x Intel 320 160GB, 2x WD Red 3TB: 174w
    Folding@Home GROMACS 100% CPU load on all 8 CPUs: 489w (power saving enabled)

    Dell C6100 XS23-TY3 Internals
    Supermicro 2U Twin2 Motherboards in a Dell chassis. These motherboards can be "hot swapped" by pulling them out of the rear of the chassis, just like the power supplies.

    Here is an overview shot:

    Dell C6100 XS23-TY3 Management
    IPMI similar to Supermicro/ Tyan
    American Megatrends Based

    Default IPMI Username and Password: root // root
    Could not find the default login ID and password via searching so hopefully that helps someone.

    Dell C6100 XS23-TY3 Sound
    Measurements thus far on the Extech
    Idle 66dba
    100% Load Across 8x CPUs 77.4dba

    Dell C6100 XS23-TY3 Expansion
    Each Dell C6100 XS23-TY3 comes with a low profile PCIe 2.0 x16 slot on a riser. There is also a PCIe x8 mezzanine card slot with Infiniband and LSI controller options.

    Dell C6100 XS23-TY3 Performance
    Each dual Intel Xeon L5520 node has about the performance of an Intel Core i7-3770K overclocked to 4.2GHz in highly threaded applications. For example, each L5520 node in Folding@Home each dual Intel Xeon L5520 node does 27K to 30K ppd, about what an overclocked Core i7-3770K will do.

    More to come.

    Dell C6220 the C6100 Successor
    These will cost more but here is a video describing the new C6220 upgrade over the C6100 generation.
    From what I have heard, Steve Croce (product manager for the series) left Dell in November 2012 and joined Calxeda. Makes sense as Calxeda has some hot new technology on their hands.

    Dell C6100 XS23-TY3 Drivers

    Here is a link to current Dell drivers. Most components are very standard.

    Dell C6100 XS23-TY3 Spare Parts List
    Rackmount Rail Kit - 0Y3DX1
    Mid-plane Chassis Fan - 104TT
    1.1kw PSU - XVKF0
    Motherboard Tray - D61XP
    Mellanox dual port QDR Infiniband Daughter Card - JR3P1
    SAS Bridge Card - JKM5M
    LSI SAS 1068e Daughter Card - Y8Y69
    Intel 82599 10GbE Daughter Card - TCK99
    3.5" SAS Backplane - V3X78
    3.5" Drive Sled - 8TV68
    2.5" SAS Backplane - H9TR4
    2.5" Drive Tray - D273R

    An alternative for the 2.5" chassis is to use the HP 2.5" drive sleds from the HP DL Series G5/ G6/ G7 generations. These run only about $3-4 each on ebay. Discovered by dba and confirmed (with notes on the EMI gasket and drive screws.)
    Also additional drive sleds often do not come with screws. We are using the following screws Amazon.com: Supermicro Screw Bag and Label for 24x Hot swap 3.5-Inch HDD Tray Cable (MCP-410-00005-0N): Computers & Accessories to secure drives in trays.

    More to come
  2. dba

    dba Moderator

    I am intrigured by the idea of a relatively low-cost, high density micro cloud server like the 6100. Will you be posting some photos?
  3. Patrick

    Patrick Administrator Staff Member

    Added a few for you. Snowboarding tomorrow.

    Very happy to see the sub-430w load numbers as I was getting worried.
  4. Jeggs101

    Jeggs101 Member

    I want one now. Can these take Xeon L5640's? Very surprised how close in price the L5430 machines are. IIRC Core 2 to Westmere was huge performance and PPW upgrade.

    May part one out.
    8x Xeon L5520 * $65ea = $520
    24x 4GB DIMMS * $20ea = $480
    Power supply selling for $80ea
    4x 3.5 trays * $20ea = $80
    4x Fans * $35ea (ebay price) = $140

    Can prob get $100 for each mobo easily and all of those other boards and cables prob $20-60 ea.
  5. PigLover

    PigLover Moderator

    Very interesting...looks like a 2U mini-bladeserver.

    Are the MBs "standardized" (at least within the SuperMicro lineup)?
  6. MiniKnight

    MiniKnight Member

    Wow.... So sub-110w for each system? Best part is can go over 32gb. This is way cheaper than even 4x Core I3s.

    Is VTx and VTd support? Would be great if so. Get external controller and connect an extra disk shelf.

    Great info. More than anywhere except Dell tech docs
  7. Patrick

    Patrick Administrator Staff Member

    Updated idle power consumption. Not bad.
  8. Jeggs101

    Jeggs101 Member

    I see a stealthy edit on the C6220 series and the product manager.
  9. cactus

    cactus Moderator

    From the video, I am confused on the power savings for the new gen. He says it saves ~100w at full load? Maybe for equal core count, single E5-2650L vs dual L5520 per node.
    Also, are those the elusive full height bracket for LSI9202 like cards in the 2 node? @4:40 and @11:11

    @Patrick, the Idle line has "no drives" and then lists drives at the end. 174W for all 4 nodes idle is great.
  10. Patrick

    Patrick Administrator Staff Member

    Ha! Edited the end, not the start.

    Just installed 2x Xeon L5638's ($125 ea) in one of the nodes. Pulled the motherboard, swapped CPUs, walked to the desk and restarted the box. Awesome experience as it took 1 latch, a tug and then 4 screws for each heatsink. Much easier to swap mobo/ RAM/ CPU than traditional 1Us. Can leave all of the storage and power supply in place with this.
  11. Patrick

    Patrick Administrator Staff Member

    Not 100% sure, need to check the power connectors but that may be the difference. Also, the I/O shield is not replaceable so you would need a MB that fit properly.
  12. xnoodle

    xnoodle New Member

    Were there holes for your 2.5" drives on the 3.5" sleds? (I assume so?)

    And hard drives are fed off the motherboard (ICH?), not a add-in mezzanine card?
  13. Patrick

    Patrick Administrator Staff Member

    I had to use converters to mount the 2.5" drives. They are fed off the ICH.
  14. MiniKnight

    MiniKnight Member

    Wonder if you can mix and match the 2.5 and 3.5 and get 12 2.5 + 6 3.5
  15. Jeggs101

    Jeggs101 Member

    I think that there is a shared drive backplane so you cannot.
  16. Jeggs101

    Jeggs101 Member

    Any idea why Unixsurplus/ mrrackables on ebay is so much more expensive? The other sellers seem to be moving tons of these things while they barely see any sales or offers. Their prices are much higher.
  17. xnoodle

    xnoodle New Member

    Were those Extra Spicy L5638s?
  18. Patrick

    Patrick Administrator Staff Member

    Possibly. Have to save money somehow.
  19. Jeggs101

    Jeggs101 Member

    Just to add on this. MrRackables is $700 with best offer for the barebones. Theirs has rails ($50), 4x drive caddy ($15/ea), but only 1 psu ($80 ea). Barebones with no rails, no caddy, but redundant psu... $225 on the forums.
  20. cactus

    cactus Moderator

    I am sort of amazed they work. On Supermicro boards don't spicy chips normally lose support after the initial retail BIOS?

Share This Page