Home Server Rebuild: S-35-DE5SL, E3-1230v3, E3C224D4I

[matt_garman]

I recently rebuilt my home server. Original build log (with pics) here: U-NAS NSC-800, E3-1230v3, E3C224D4I, 2xSSD. There seem to be a few other people using that U-NAS NSC-800 case. While it's wonderfully compact, it has a couple problems resulting from its size:

• Tiny amount of clearance for CPU heatsink/fan. I was using a Noctua NH-L9i cooler on the E3-1230v3 CPU. This was perfectly fine under idle load and/or short bursts of CPU activity. But it was not capable of comfortably cooling the CPU with all four cores maxed out for a sustained period of time. In the interest of safety, I disabled turbo boost and two cores. Running a Handbrake job under this config would put the CPU around 70 degrees Celsius. Acceptable, but not enough headroom for me to confidently enable the other two cores.
• Any changes to the system (short of hard drive swaps) effectively required a complete tear-down. E.g., a memory upgrade I did about a year ago took a long time.

It seems technology has progressed faster than my needs (finally). In particular, on the storage front: I was previously using:

• Five 3TB drives in a RAID-6 config for media storage
• Two SSDs in RAID-1 for OS
• One big drive for MythTV recordings

That made for eight total drives. That required me to use an additional HBA (only six SATA ports on the motherboard), and have six (or more) 3.5" hotswap bays on the chassis. I found I was only using about 5 TB of the 5x3TB array. So I cut that down two HGST He8 8TB drives in RAID-1. That brought the total drive count down to five, only three of which were big 3.5" drives. It also allowed me to remove the HBA I was using, and get away with only the motherboard SATA ports (saving a bit more electricity and heat).

Reducing the required hotswap bay count opened up more options for small cases. I went with the iStarUSA S-35-DE5. I had previously used a variation of this case, the S-35-DE1 (only three hotswap bays), in a build for my parents. The S-35 case in general is "modular" in that the space for hotswap bays is actually just three generic 5.25" slots, so it will work with any number of X-in-3 units. I believe this current config will tide me over long enough until SSD prices are low enough to retire all 3.5" spinning drives. And then I can get a new multiple 2.5" drive module to fit in this space.

Giving up the hotswap bays bought me a lot more room for a bigger CPU cooler. Based on the SilentPCReview article, I went with the Scythe Kotetsu SCKTT-1000.

With the new CPU cooler in place, I was able to re-enable all four cores and turbo boost. Results:

• Previous idle CPU temps were around 40C. Now they're around 30C.
• Previous 2 core loaded CPUs (HandBrake) were around 70C. Now I'm barely breaking 60C with 4 cores loaded.
• Hard drive temps are roughly the same, maybe a degree or two cooler, in the upper 30s.

The 5-in-3 hotswap module that comes with the S-35-DE5 has a built-in 80mm fan. I removed it to make room for the 120mm fan on the CPU heatsink. You can see, if I really wanted to get crazy with cooling, I could add another 120mm fan to the heatsink. Also, the layout screams ducting: I think if I built a duct that covered the sides and top from the hotswap module to the back of the case, I'd see still more cooling efficiency... but as you can see from the results above, things are great as-is.

That said, anyone doing DIY ducting in their systems? If so, what material are you using? Cardboard is quick and easy, but doesn't "feel" right.

I do have a couple gripes with this case:

• It only has internal mounting for a single 2.5 drive. But there is plenty of room for more. The existing 2.5 mounting bracket could have been extended just a bit and two drives could easily be sandwiched on top of each other (someone willing to go to work with a Dremel and a drill could make this happen). And right next to the 2.5 bracket is room for another 2.5 bracket. It's not shown in the pictures below, but my second 2.5 system drive is actually just "floating" on top of the 5-in-3 module (I was originally planning to use an adapter to put it in one of the hotswap bays, but the adapter I have on hand doesn't put the power/data connectors in the right location).
• The rear fan cutout is 120mm... there's easily room for a 140mm fan. In fact I have a 140mm fan with 120mm hole spacing that I was going to use that in this case. But I could see the diameter of the fan blades was greater than the cutout, and I figured any benefits of the bigger fan would be lost.

I will try to remember to post graphs from Munin, that show the temperature differences.

Also, I need to connect this to a Kill-a-Watt to see how much electricity it's using. (Going from memory, the system previously idled around 70 watts; I'm hoping the rebuild gets below 50.)

On to the pics!

Here's the right side profile shot:

Here's the left side profile shot:

Bird's eye view. As I mentioned above, the second 2.5 SSD is actually floating on top. The SATA cables tuck in nicely when the outer shell is put in place.

Left-side 3/4 view:

Right-side 3/4 view:

 

[gigatexal]

very clean. kudos for taking the time to wire it up well.

 

[matt_garman]

very clean. kudos for taking the time to wire it up well.

Thank you! Although I thought my wiring was just "OK". My main goal was to keep wires out of the airflow path from the HDD module to the back fan. Usually on my builds I start out with high expectations for clean wiring, then become increasingly less ambitious as time goes on.

At least we're well-past the days of those old IDE ribbon cables. I used to spend way too much time doing "cable origami" trying to get neat cabling in those builds (and never really pulled it off anyway).

 

[gigatexal]

i gave up on cabling a long time ago. if the cables are out of the way of providing good cooling, then I called that good. The ROI just wasn't there for me.

again, kudos.

 

[PigLover]

Usually on my builds I start out with high expectations for clean wiring, then become increasingly less ambitious as time goes on. ...I used to spend way too much time doing "cable origami" trying to get neat cabling in those builds (and never really pulled it off anyway).

This. Always this. Story of all my builds.

 

[matt_garman]

Here are some graphs of temperatures from Munin. First a series of graphs recorded before the above changes. These graphs are for the 24-hour period starting at Dec 22 around 4:00pm:

Air Temperature (green is F, blue is C):

CPU Temp. The bump you see in the middle is for a HandBrake job. Keep in mind, these temps are with only two cores.

Now, graphs after the change. First air temp:

CPU temps. Here you can see a HandBrake job running at the beginning of the time period, and the start of another at the end. Now all four cores are enabled:

Lastly, here are HDD temps for the after period. When I did the rebuild, my hard drive enumeration got reset, so I Munin doesn't show any HDD temperature data for the before time period. But going from memory, all drives were generally a degree or two higher in the old config. Drive temps were OK before, and still OK in my opinion, so this is more FYI info.

Lastly, the same air temp and CPU temp graphs as above, but a single graph showing the whole time span before and after the changes. This keeps the Y-scale consistent so the change is a little more obvious.

 

[bash]

We have the same complaints when it comes to how cases handle SSD mounts and wasted space. Usually I just go the double sided velcro route.

It may not be pretty but it works!

 

[gigatexal]

those temp improvements are awesome!

 

[T_Minus]

Which MUNIN sensor plugin is that or do they do sensors now default?

 

[matt_garman]

Which MUNIN sensor plugin is that or do they do sensors now default?

I'm running Munin on CentOS 7.1. CentOS doesn't provide Munin by default, but it's available through EPEL; I'm on version 2.0.25.

The CPU and HDD temp sensors come stock, though I can't remember now which ones I enabled and which ones were enabled by default. Specifically, the CPU temp one is called "sensors_temp" (symlinks to "sensors_"); the HDD temp one is called "hddtemp_smartctl".

The air temperature is custom. I'm using a $15 USB thermometer (called PCSensor or USBTemper). I hacked the software to log to a sqlite database, see here. But even the non-hacked software can easily be used with Munin, you'll just have to write a little script to call the program. The Munin Docs for plugin writing are super straight-forward. Someday, I might get around to further enhancing that code on GitHub with RPM spec files, Munin plugin script, etc. But my ambitions are often greater than the time I have available.

 

[T_Minus]

I'm running Munin on CentOS 7.1. CentOS doesn't provide Munin by default, but it's available through EPEL; I'm on version 2.0.25.

The CPU and HDD temp sensors come stock, though I can't remember now which ones I enabled and which ones were enabled by default. Specifically, the CPU temp one is called "sensors_temp" (symlinks to "sensors_"); the HDD temp one is called "hddtemp_smartctl".

The air temperature is custom. I'm using a $15 USB thermometer (called PCSensor or USBTemper). I hacked the software to log to a sqlite database, see here. But even the non-hacked software can easily be used with Munin, you'll just have to write a little script to call the program. The Munin Docs for plugin writing are super straight-forward. Someday, I might get around to further enhancing that code on GitHub with RPM spec files, Munin plugin script, etc. But my ambitions are often greater than the time I have available.

Awesome thanks! I've gotten some USB temp sensors too, but not gotten around to wiring them up yet... very excited to be able to add sensors and use w/munin w/little effort.

Any chance you looked into rack/data-center type receivers/probes for humidity and temp to use with munin too vs. whatever they use proprietary/built-in?

I understand the ambitions, my to-do list is through 2016 and my project specs just keep growing, ha ha.

 

[matt_garman]

Any chance you looked into rack/data-center type receivers/probes for humidity and temp to use with munin too vs. whatever they use proprietary/built-in?

I looked a little bit. That's actually how I discovered the $15 USB sensor: at work, most of our servers are in professionally managed (offsite) datacenters. But we have a small server room in our office. We needed a way to monitor temperature, but all the true rackmount equipment cost more than he wanted to pay. That's when I stumbled on the USBTemper/PCSensor cheapie (though that link shows they're up to $21 now, at least on Amazon, maybe cheaper on ebay). When we got that at work, I thought it was too cool (and low enough in price) that I had to have one for home!