Taming the C6100

[PigLover]

I picked up on of the C6100 cloud server chassis as described here and listed here. It arrived today, CPUs and memory coming soon.

I mentioned in the other thread that I plan to see if I can tame it a bit for use in a home lab. The chassis is designed for an aggressive deployment using 15k SAS drives and 90watt Xeon CPUs. In my plan I'll be using "L" series 60 watt Xeons, starting with L5638s and maybe one node with L5640s (if I can source them cheap enough...). I'll be limiting the local disks to SSD and, perhaps, running diskless with PXE and/or iSCSI boot. I have a very fast 40Tb ZFS server node and a 10Gbe network infrastructure so its probably going to work OK.

As is, Dell rates it at 77dba loaded and I've got reliable reports (including Patrick's) that it ranges from 65-77 dba depending on load. Too loud for most home uses - loud enough to be heard through normal household walls (even insulated walls to a garage or basement). I've set a goal for myself to target a 50dba noise profile. I'll probably be happy if I can get to 50dba nominal/light load with it somewhat higher at full tilt (which it will almost never reach).

I've some experience silencing systems. My prior dual-Xeon X5550 workstation hits 35dba at full load - even with 13 spinny disks and 4 SSDs. It runs cool at that noise level and idles at audible but not annoying levels (it has passed the "wife test" - she doesn't complain if its gets left on even though its sits in our family room).

So...here we go! I'll post occasionally as things progress.

 

[gigatexal]

Will dutifully follow what you find. What size are those thick fans, 40mm in diameter and like 30mm in thickness? I'll start googling.

This might work

revs to the higher end of 56 dba http://www.titan-cd.com/Y_product_C_1u2u1155_e.php?id=916

these might be better: http://www.titan-cd.com/Y_product_C_1u2u1155_e.php?id=888

Those aren't the right socket though...

this looks like it might work the best: http://www.frozencpu.com/products/8...Server_Active_PWM_Fan_Copper_CPU_Cooler_JAC7B

no word on how loud she gets tho

 

[PigLover]

Gigatexal,

The C6100 uses 1U server cooling methods for each MB tray. Its very much like having two 1U dual-node servers stacked together with all four nodes sharing a set of common support - redundant PSUs, fans, etc.

Each CPU uses a low-profile heavy copper heatsink like this one (its not exact, but shows the general design). The four nodes have a fan tray sitting right behind the hard drive backplane. The fan tray has four 80mm fans - they appear to be 25mm thick or so but I haven't measured them yet. These fans are high pressure fans that blow air at fairly high speed across the MB, with thin cardstock airflow guides ensuring air is forced over the CPU heatsinks. Everything on the MB is oriented to make use of this front-to-back airflow.

You can't just drop-in a typical quiet CPU cooler like the ones we use on "normal" systems.

The real trick is balancing how much you can slow/modify the fans to make them quieter against getting enough airflow to cool the CPUs. Working in my favor is that Dell designed this to have relatively hot SAS disks in the drive backplane in front of the fans. They planned the required airflow assuming the air is pre-heated by the drives - I won't have that problem. They also designed for 90 watt maximum TDP CPUs. Since I'm using 60 watt CPUs I only need to evacuate 2/3s of the heat they planned for.

Working against me is that I don't have a properly conditioned equipment room. At times I'll be dealing with 90+ degree ambient air.

Before outright replacing the fans I'm going to try a few methods to just slow them down. With most fans you get a rapid reduction in noise by just running them slower but still move enough air to manage the system. I'm more comfortable letting the CPUs get relatively warm than Dell's engineers so I think I'll have a fair amount of margin to work with. The real trick is understanding just how smart Dell's fan controller board is...it might work hard to overcome my efforts to slow them down.

 

[Patrick]

Before outright replacing the fans I'm going to try a few methods to just slow them down. With most fans you get a rapid reduction in noise by just running them slower but still move enough air to manage the system. I'm more comfortable letting the CPUs get relatively warm than Dell's engineers so I think I'll have a fair amount of margin to work with. The real trick is understanding just how smart Dell's fan controller board is...it might work hard to overcome my efforts to slow them down.

Well all three of mine are currently with FedEx so cannot be much help at this point. Here is a picture of the unit with the top off:

The somewhat good news is that the fan controller board is in the top middle chamber of the above picture.

 

[cactus]

Who controls the speed of the fans? Say you had only a single node installed, is there a primary slot where it must be installed? Can you boot the chassis with no nodes? My train of thought is intercepting the signal to the controller board and augmenting it in some way.

 

[PigLover]

No, you can't boot the "chassis" without booting any nodes. Well, not exactly, anyway.

It does appear that the fan control board receives power when the PSUs are plugged in but no nodes are actively booted. My best guess - without schematics - is that the FCB uses standby power to keep itself at the ready.

There are also sensor leads going from the FCB to the disk backplane and to the SMB-bus of each MB. Presumably this is used to collect sensor data from the MBs and also to make fan status/alarm data available over SMB. I won't know for sure what you can actually see (or perhaps even influence) until i some more parts in to populate at least one node and light it up.

The fans also use an 8-pin connector to the FCB instead of the more common four. Two of the pins are not connected, one each to "pulse" and "sense" for the PCM, two power and two common. The fan itself has a more standard 4-wire arrangement, with the two power and two common joined into a single power and single common lead right before reaching the fan. I will presume here that it is done this way to collect power from the two PSUs independently.

The fan itself is a 12v/4.6a high speed fan. If I can't get control through the BMC to slow it through software then I'll try a resistor mod. If that doesn't work I'll start shopping for new fans...

I'll take & post some pictures of the fan and its funky cable arrangement tonight.

 

[Patrick]

PL - have you put a lot of load on these yet and listened to the fans? They certainly kick into high-speed. Not sure how it is measuring.

 

[Aluminum]

I think you will have a hard time making this quiet enough for anything but a basement/garage, also what about the powersupply?

It might be ok with 1 node on and everything slow but why have all that awesome density if not to use it?

Fully populated you'll have 8 times the cpu heat in a 2U as I have inside a 4U, and I had to replace every fan and get a regular PSU (fan only kicks on under load) to make that sensible enough to be in the same room as the media center.

 

[PigLover]

Quiet enough for the living room is not the goal. 50 dba is still quite loud. My target is quiet enough for a dedicated equipment space - garage, basement or dedicated room. In my case its garage.

The goal is quiet enough that is does not produce any disturbing noise through the walls into the house...and it needs to be with all four nodes running (albeit with low power CPUs, limited disk, etc). Its will be sitting in the same rack with a Juniper EX-2500 switch (45 port 10Gbe SFP+) that idles at about 45dba and gets quite a bit louder under heavy load or when the ambient in the garage goes up. If I can get it into that same sound class then I've succeeded.

 

[gigatexal]

Piglover would a drop in hsf as well as lowering the voltage on the fans, that combination, not be a good idea? I see your best case scanario being some sort of hybrid solution or an oversized passive heatsink?

either way we all look foreword to what you come up with

 

[PigLover]

Biggest issue with using HSF is that there are no fan headers on the MB. To use them you have to hack/mod a power source for the fan. Ideally you'd have to take power from somewhere on the MB sled itself so that you could still slide them in/out without having to open the case and remove connectors. It would be an ugly mod.

Second problem is that it would have to be very low profile. With clearances to slide the sled in/out of the chassis you'd have to be smaller than a 1U cooler.

Third problem is that the rest of the components on the sled still need cooling too - and the HSF would interrupt the airflow.

In any case - I'll have some more info after next weekend. I don't think I'll have the parts I need to fire up even one node until next week (a part got delayed by a shipping snafu) and I'm traveling on business T-F next week. I really need to get at least one node fired up to know what controls you do (or don't) have via the BMC. I probably can't really characterize the noise and performance profile of the fans until I have all four lit up.

 

[gigatexal]

I see the problem now, well good luck. You figure this out and I'll buy one if they're still around just for kicks ;-)

 

[Jeggs101]

I see the problem now, well good luck. You figure this out and I'll buy one if they're still around just for kicks ;-)

There are many out there. Other post says they are sold in 500qty lots. Easy to get a C6100 with cpus and ram.

 

[PigLover]

Been working to get the C6100 ready to turn up. I have CPUs and memory installed for two nodes (2x L5638s & 49Gb each). I have two more L5638s but am short on RAM. But the heatsinks were shipped separately and have not arrived yet. So I wait...

When they arrive I'm going to fire it up unmodified to get some baselines on noise, power, etc.

I've ordered 4-pin fan connectors, male and female, so that I can interpose a pot (variable resistor) to try simply slowing down the fans. The more I think about this it probably isn't going to work. Firstly, resistor mods on PWM fans are iffy at best.

I've also ordered a couple of fans to try as replacements. The stock fans used by Dell are 13,500RPM max PWM fans that are spec'd at up to 72dba each. One is an 80x38mm Delta with a lower speed/noise profile. I'm going to try another Delta 9,000RPM fan spec'd at 65dba. I've also found a "low-noise" Enermax fan that runs at 6,500RPM and 52dba. These potential replacements are still high-pressure server fans - hopeful that they will move enough air for the L-series CPUs.

Lastly, I've been looking at the disk cabling. My nodes all came with the LSI 1068e mezzanine card (which I really don't want). I need to re-cable to use the on-board SATA connectors for any disks. I tried to jerry-rig in standard SATA cables to route the on-board SATA plugs to the interposer card, but they are just too bulky for the job. Luckily I found a source with the Dell cables available. This is one of the things I have to wait on before lighting it up.

 

[Patrick]

Just bought #4 for the home lab. Hopefully will get some time to help next week. Monday to Monday I'm in four different cities.

 

[Dragon]

Thanks for the tips, I just heard the news from Patrick and am also looking forward to getting the c6100.

70dBa is WAY too loud, according to the fan's specs, it has built in PWM control and its frequency is controlled through the yellow wire.
http://www.digikey.com/scripts/dksearch/dksus.dll?FullDetail&name=603-1708-ND
http://www.delta-americas.com/products/FanUploads/Specification/PFR0812XHE-SP00-REF..pdf

Quoting from page 6:
Black wire (-)
Red wire (+)
Blue wire (F00)
Yellow wire (PWM)

Page 8:
PWM Control Signal:
Signal voltage range:0-16VDC
The frequency for control signal of the fan shall be able to accept 50hz-100kHz
The preferred operating point for the fan is 25kHz.
At 100% duty cycle, the rotor will spin at maximum speed.
At 0% duty cycle, the rotor will stop.
When control signal lead disconnect, the fan will spin at maximum speed.

I still have some "fan speed control" knob from like 7 years ago similar to this:

Hopefully they'll work just by connecting the wires.

If they don't then I'll have to order replacements.

 

[McKajVah]

I'm planning to do something more radical with it... a 4U Franken-server.

My rack is not full depth, so I have to shorten the case.

What I will do is cut the chassis i two between the fans and the motherboards and put the drive-assembly on top of "motherboards" facing backwards... I will then put three 120mm (or even 140mm if they fit) on the back of the thing... Hopefully I wil get the thing down to 40dba, if the power supplies are not to loud...

And if the whole ting turns into a ugly mess...at least I had some fun doing it.


-Kaj

 

[Dragon]

My rack is not full depth, so I have to shorten the case.

What I will do is cut the chassis i two between the fans and the motherboards

Ouch, that sounds messy, what's wrong with letting it stick out like a sore thumb?

 

[PigLover]

Dragon,

The fan controller idea isn't bad but you'll have to do a bit more than just plug in the controller in-line. As built, Dell uses proprietary 5-pin fan connectors so that they can connect the fans to both of the redundant PSUs. Modding the connectors won't be hard.

Although it seems obvious, for some reason I handn't considered just cutting the pwm control lead and running the fans on a pot (duh!). One thing to be careful of is making sure the pot can handle the fan. That Zalman controller you show in the picture is designed for consumer fans - fan that typically draw less than 0.5amp. The Delta's in the C6100 draw up to 4.9amps each...you'll want to be sure your controller can handle that. You don't want it to smoke!

 

[Dragon]

Dragon,

The fan controller idea isn't bad but you'll have to do a bit more than just plug in the controller in-line. As built, Dell uses proprietary 5-pin fan connectors so that they can connect the fans to both of the redundant PSUs. Modding the connectors won't be hard.

Although it seems obvious, for some reason I handn't considered just cutting the pwm control lead and running the fans on a pot (duh!). One thing to be careful of is making sure the pot can handle the fan. That Zalman controller you show in the picture is designed for consumer fans - fan that typically draw less than 0.5amp. The Delta's in the C6100 draw up to 4.9amps each...you'll want to be sure your controller can handle that. You don't want it to smoke!

Thanks for the warning, luckily I haven't ordered the server yet (still trying to figure out how to draw power for 2 extra SSDs hidden in a node), I have no idea what is this "pot" you're referring to, but I am looking forward to your solution nonetheless.