Help spec my SC846 build! (fileserver + light virtualization)

[Diavuno]

that'll be fine for now....

Remember though - half those dimm slots are connected to the second CPU.

I'd buy a matched pair and only use one until you need the slots.

with VM's ram is huge... and for home use you generally don't need any cpu.

 

[eduncan911]

Just FYI, I have 24GB of 1.35V Low-Voltage 1333 Mhz ECC ram for sale in the classified section here. I specifically bought them for exactly your build, along with 2x L5640s to build a low-power consumption system as the 60W TDP of the L5640 is an excellent power saver (not to mention the 6C/12T each CPU brings, for a total of 12C/24T).

I carefully selected that ram to get the full 1333 Mhz across 2x processors - as if you read the manual of the motherboard you got (and all other Supermicro X8 series), it's difficult to get 1333 Mhz in ECC to work with the Supermicro X8 series. The key is to remember the LGA1366 socket is a tri-channel memory socket: it wants 3 sticks per CPU. If you double up the memory (2 sticks per channel, or 12 sticks total), you drop down to 1066 Mhz or 800 Mhz for your memory. If you go for dense QR (Quad Rank) memory, it also drops down to 800 Mhz for the most part. The memory kit I got is 6x 4GB DIMMs, 3 DIMMs per CPU for tri-channel memory to get the full 1333 Mhz.

The L5640s is a great buy right now IMO. Do it. 12C and 24T, for cheap (I got mine for $80 shipped if you watch for deals on eBay and make offers). And, they overclock very easily if you like to get over 3 Ghz. Power consumption goes way up at that point though.

I ended up going with an X10SRA build instead with an engineering sample of a E5-2690v3 (12C/24T) I got super cheap. Overall build cost with the X10SRA and 64GB of DDR4 ECC ram added $500 to the expense of my build over the cost with the ideal X8D board for 3 GPUs and same 12C/24T; but, I have 64GB of ram instead of 24GB. My goals were a tad different than yours: same lower-power, low-noise + VMs setup but I needed at least 3 GPUs (it's a StableBit DrivePool+Plex build, not FreeNAS, along with a number of NZB and development VMs) for my home lab. The X8 series boards with the perfect x16 layout for me were all $350+ each. I figured if i was spending that much on a board, I might as well go with something newer. Still kind of pissed I didn't go with a 4-way setup; but, this chassis can only do 3-way. I might take out the Dremel one day if I ever want to make it 8-slot (move the mobo tray over 1", and get another mobo).

Also, be sure to read up on the security / hackers-dream of the IPMI of these boards and lock yours down.

 

[NickM]

Thanks for the info/offer I scored a good deal on some non-LV RAM and just went with that. For now, I've got 6x4GB and 1x L5640.

I will note, for anyone else torn between LV/non-LV, that the 6 sticks of 1.5v ECC RAM seem to be about the hottest thing on the motherboard (passive HSF, air shroud in place). Maybe I should've gone LV after all...

Re: IPMI, I've read a bit on the security issues, but most of the articles I found were more along the lines of 'here's the problem', rather than 'here's how to protect yourself'. Any specific tips?

 

[NickM]

Also, for another PSU/noise datapoint....

I've got one of the 1200W gold PSUs (just one). At idle with ~12 drives attached and ~10 spun down, the PSU pulls about 150W from the wall, but is only delivering ~90W DC to the system (per PMBus stats). Very inefficient at low power draw. It's basically dissipating 60W in waste heat inside the PSU.

The server sits about 10 feet away from my desk, in a closet with the door kept open for airflow. The office is directly across the hall from my bedroom. I've got PWM control working on the default case fans, and at around 30% they're comparable to a not-entirely-silent desktop PC.

The PSU on the other hand, is a pain. PSU noise level seems to be dramatically influenced by ambient temperature. Over the past 2 weeks, daytime highs where I live went from 95F+ down to mid-70's, prompting open windows at my house vs air conditioning set to ~78F. At 75-80F ambient in the server closet, the PSU fan spins at 4k-5k rpm, and is very annoying even during the daytime while working. At night, I had to close the office door (or -- gasp! -- turn off the server).

Once I had the windows open, and overnight ambient indoor temps in the server closet dropped into the low 70's/upper 60's, the PSU fans became all-but-unnoticeable in the morning. Comparable to the noise it generates when the PSU is plugged in, but the server itself is powered down. By mid-day when it warms up, the PSU fan sped back up and became annoying again.

Currently contemplating/planning on ghetto-mounting a spare desktop PSU into the server that I know will be quiet, rather than shelling out for a platinum PSU (which may or may not actually be quiet enough)

 

[eduncan911]

I have a large blog post I am making related to that (power usage).

But I can say as part of my tests, yes replacing the PSUs with an aftermarket ATX, any of the PSUs even the "SQ" models, will remove all noise (excluding the PWM fans' noise that is). It's part of my testing rig and data I'll be publishing.

The only issue is you loose some IPMI stats.

 

[eduncan911]

I've got PWM control working on the default case fans, and at around 30% they're comparable to a not-entirely-silent desktop PC.

Can you explain how you got them down to 30%? Or even know the PWM duty cycle is infact 30%?

Using an aftermarket PWM controller?

From my measurements, the lowest I can get them is 50% with Supermicro's onboard BMC, including the CPU fan header.

$ ipmimonitoring -D lan20 -h {masked} -u {masked} -p {masked}
ID   | Name            | Type              | State    | Reading    | Units | Event
4    | CPU Temp        | Temperature       | Nominal  | 38.00      | C     | 'OK'
71   | PCH Temp        | Temperature       | Nominal  | 32.00      | C     | 'OK'
138  | System Temp     | Temperature       | Nominal  | 28.00      | C     | 'OK'
205  | Peripheral Temp | Temperature       | Nominal  | 37.00      | C     | 'OK'
272  | VcpuVRM Temp    | Temperature       | Nominal  | 36.00      | C     | 'OK'
339  | VmemABVRM Temp  | Temperature       | Nominal  | 34.00      | C     | 'OK'
406  | VmemCDVRM Temp  | Temperature       | Nominal  | 30.00      | C     | 'OK'
540  | DIMMA2 Temp     | Temperature       | Nominal  | 34.00      | C     | 'OK'
674  | DIMMB2 Temp     | Temperature       | Nominal  | 34.00      | C     | 'OK'
808  | DIMMC2 Temp     | Temperature       | Nominal  | 32.00      | C     | 'OK'
942  | DIMMD2 Temp     | Temperature       | Nominal  | 31.00      | C     | 'OK'
1009 | FAN1            | Fan               | Nominal  | 300.00     | RPM   | 'OK'
1076 | FAN2            | Fan               | Nominal  | 2600.00    | RPM   | 'OK'
1143 | FAN3            | Fan               | Nominal  | 2700.00    | RPM   | 'OK'
1277 | FAN5            | Fan               | Nominal  | 2400.00    | RPM   | 'OK'
1344 | Vcpu            | Voltage           | Nominal  | 1.81       | V     | 'OK'
1411 | VDIMM           | Voltage           | Nominal  | 1.21       | V     | 'OK'
1478 | 12V             | Voltage           | Nominal  | 11.80      | V     | 'OK'
1545 | 5VCC            | Voltage           | Nominal  | 5.00       | V     | 'OK'
1612 | 3.3VCC          | Voltage           | Nominal  | 3.36       | V     | 'OK'
1679 | VBAT            | Voltage           | Nominal  | 3.04       | V     | 'OK'
1746 | 1.05V PCH       | Voltage           | Nominal  | 1.04       | V     | 'OK'
1813 | AVCC            | Voltage           | Nominal  | 3.34       | V     | 'OK'
1880 | VSB             | Voltage           | Nominal  | 3.27       | V     | 'OK'
1947 | Chassis Intru   | Physical Security | Nominal  | N/A        | N/A   | 'OK'
2148 | PS1 Status      | Power Supply      | Nominal  | N/A        | N/A   | 'Presence detected'

At full speed, the fans spin at about 5,500 RPMs. One could deduct that 50% duty cycle is around 2,750 RPMs, as you can see above (sensor output only shows within 100s).

To set them at this level requires you to use IPMIview (ugh) to set the Chassis Fans to "Optimum", at least for my X10SRA-F. The other options were not available.

FAN1 is a Nochua, and I've adjusted the thresholds accordingly for that low speed to keep the spin up/down cycle from happening.

 

[NickM]

I had no reasonable luck trying to control the fans with IPMI settings. At best, I could set it to 'optimal' (on my X8DTE-F board), which was still in the 4k RPM range and entirely too loud. I wasn't able to figure out how to change any 'threshold' settings with my board/ipmi which would spin the fans down low enough at idle.

After adding the right modules (use "sensors-detect"), the lm-sensors package allowed fine-grained control of the fans:

# modprobe coretemp
# modprobe w83795

# echo 1 > /sys/class/hwmon/hwmon1/device/pwm2_enable

# echo 200 > /sys/class/hwmon/hwmon1/device/pwm2
# cat /sys/class/hwmon/hwmon1/device/fan1_input
5818

# echo 100 > /sys/class/hwmon/hwmon1/device/pwm2

# cat /sys/class/hwmon/hwmon1/device/fan1_input
3846

# echo 500 > /sys/class/hwmon/hwmon1/device/pwm2
# cat /sys/class/hwmon/hwmon1/device/fan1_input
7105

# echo 1000 > /sys/class/hwmon/hwmon1/device/pwm2
# cat /sys/class/hwmon/hwmon1/device/fan1_input
6994

# echo 30 > /sys/class/hwmon/hwmon1/device/pwm2
# cat /sys/class/hwmon/hwmon1/device/fan1_input
2351

# echo 1 > /sys/class/hwmon/hwmon1/device/pwm2
# cat /sys/class/hwmon/hwmon1/device/fan1_input
1724

I'd initally thought the pwm device took values 0-100, but it appears that even from 200 to 500 still has an incremental effect. I'm not really sure what those values "mean".

For my motherboard / package version / fans -- around pwm set to "30" (~2300 rpm), I don't get further audible reduction by dropping the fan speed further, so there's not much point in decreasing it below that. Your Sensor Names May Vary

From here, you'd just want to write/use a script to monitor whatever temperature(s) you're interested in, and adjust the fan speed accordingly

 

[NickM]

But I can say as part of my tests, yes replacing the PSUs with an aftermarket ATX, any of the PSUs even the "SQ" models, will remove all noise (excluding the PWM fans' noise that is).

I'm having a bit of trouble parsing this. Are you essentially saying, "a standard ATX PSU is quieter even than the 'SQ' supermicro PSUs"?

 

[eduncan911]

@NickM: Thanks! Yeah, already found the lm-sensors package. That's what I thought you must have been using (needing an operating system and scripts setup). FYI, I believe the values is 0-255 for PWM. With 255 = 100% PWM duty cycle' and, 0 = 0% PWM, which means the fans will as low as they are programmed to operate at, but not "off". A "PWM fan" has an onboard circuit that monitors the PWM duty cycle and is pre-programmed to spin at X RPMs with 0 PWM being sent.

With that said, I experimented and found the 50% PWM duty cycle that the Optimum setting of the BMC on the X10SRA-F to be sufficient for noise. Still very audible; but, it is going in the basement. I like the BMC controlling the fans in the vent that the operating system freezes or something. I have two fan headers connected to the 3 PWM fans in the mid-plane, and the rear 2 connected to what is considered to be "CPU Fan2", which duplicates CPU Fan1 header. This allows the system to spin up the rear fans in the need for more CPU cooling, which works perfectly with my heatsink setup using the stock fan shroud (i'll post pics later).


Regarding the replacement of the ATX...

Yes, that's what I am saying. I have a few "SQ" models of the stock PSUs and while they are quiet, yes, they still have a low-groan to them - they aren't 100% quiet/silent. I have a few 740W Platinums and only 1 is "quiet", the other 2 have a low-groan to them as well - a tad higher than the SQs. But i do have 1x 740W Platinum which IS quiet - as long as there isn't any real load on the CPU. Running mprime on the CPU eventually makes the 740 Platinum audible a little, but the CPU fans are already kicked in and pretty loud at that point.

Alternatively, just about any decent standard Gold or Platitum ATX PSU is silent - or at least completely inaudible.

As others said though, once you replace the PSUs with SQs, or even the Platinum versions, the chassis' stock PWM fans are the next thing people find annoying and replace all of them with quieter fans, or even PWM fans.

So, mixing in silent PWM fans with a standard ATX PSU gives you a silent 24-bay chassis you can stick under your desk. My problem is if you had 24 HDDs spinning, that would need a LOT of cooling which I'd be concerned with by replacing the PWM fans.

Me, personally, I like the umph (static pressure) the stock PWMs have and will continue to use them in my setup. I'm using StableBit's DrivePool. ZFS and Freenas and alike are nice and have very high bandwidth; but, they spin up all HDDs. What I like about DrivePool is I am able to spin down all HDDs. Each 4 and 6 TB drive pulls about 8W from the wall in my setup, so that's 14x 8W = 112W. When I command all HDDs to spin down, my system goes from 185W down to 74W idle at the wall. So one could replace all PWM fans since only 2 or 3 HDDs would be spun up at any given time. It's when all HDDs spin up is when I'd really want all fans.

 

[eduncan911]

Just for giggles, here's the PWM <> RPM output for the stock fans:

PWM 255 FAN 6164
PWM 240 FAN 6164
PWM 225 FAN 5921
PWM 210 FAN 5648
PWM 195 FAN 5421
PWM 180 FAN 5133
PWM 165 FAN 4909
PWM 150 FAN 4623
PWM 135 FAN 4383
PWM 120 FAN 4066
PWM 105 FAN 3802
PWM 90 FAN 3470
PWM 75 FAN 3169
PWM 60 FAN 2818
PWM 45 FAN 2459
PWM 30 FAN 2080
PWM 28 FAN 1898
PWM 26 FAN 1802
PWM 24 FAN 1737
PWM 22 FAN 1693
PWM 20 FAN 1624
PWM 18 FAN 1578
PWM 16 FAN 1535
PWM 14 FAN 1498
PWM 12 FAN 1467
PWM 10 FAN 1440
PWM 8 FAN 1434
PWM 6 FAN 1422
PWM 4 FAN 1410
PWM 2 FAN 1409
PWM 0 FAN 1415

Taken from lm_sensors.

 

[NickM]

I can happily report that my server is blissfully quiet with the 1200W Gold PSU and housing removed, and a standard ATX PSU installed (a spare Corsair CX600 80+ Bronze I had sitting around -- attached to the interior of the case with velcro strips).

I can now differentiate the noise from built-in case fans all the way down to a setting of '0'. At '0', there's still enough airflow at idle to keep the drive temps in the low 30's C. Total noise output from the server at idle is about equivalent to a laptop with its fans on medium to high. Totally reasonable to have in my office closet with the door open, and me at my desk 8 feet away. The WAF is very high

Doing an array integrity check (12 drives full speed reading), my fan control script spins the case fans up to around a pwm setting of 20-30 (very manageable noise level), which keeps the drive temps in the 35-38C range.

edit: I'm still toying with the idea of installing the Norco 120mm fan wall to get this silent under load too, but can't justify spending $18-20 per fan on the Noctua's that everyone seems to like. If I can figure out / settle on some cheaper 120mm PWM fans that are quiet and still have good static pressure, I might move forward with that too

 

[abstractalgebra]

Nice job betting the power supply in that case. How bad was the power supply mounting with the velcro? A picture would be sweet.

Btw, you got a really nice deal it looks like the SC846 Series with a SAS2 backplane is $470 landed for a barebones without Motherboard now on ebay.

 

[NickM]

I followed this very detailed guide to swap out the PSU. It was quite easy: Are You Looking For A Less Expensive Norco 4220 / 4224 Alternative? - AVS Forum | Home Theater Discussions And Reviews

It's hard to get a picture really showing much, but here ya go:


edit: looking at the pictures, it appears I mis-remembered what PSU I put in there