Supermicro-SC847E16-R1400LPB - Who has succeeded?

[ude6]

Hi all,
this is my first post to the forum.
I am looking to build a (probably too large) NAS for my home use. It would
reside in the basement but if the thing is too loud you would hear it through the
ceiling....
That leeds to my question: I have a good offer for a Supermicro-SC847E16-R1400LPB
chassis. But it seems that the noise of the stock fans and powersupplys is legendary.
There are a number of threads around that start discussing quieting the "beast" but
then seem to lead to nothing.
So my question: Has anyone succeeded? Is it enough to use PWM fans and atch to the
MB? Is exchaning fans an option or do temperatures "explode"?

Thanks for any hint
ude6

 

[MiniKnight]

You can use IPMItool to set fan speeds except for the power supplies.

Swapping those fans is tricky. Since you have drives and a system static pressure matters. I'd advise just to set PWM fan curves. You'd also be better off using 5.4k rpm disks than 7.2k

 

[ude6]

Thanks for the first feedback. That is what I thought.
I will be using WD Reds (home) not the Pro. So they are 5.4k rpm.

 

[Aestr]

The stock PWM fans with proper settings are reasonably quiet and SQ power supplies are great. From there you could look at the fans in the 745SQ chassis (FAN-0104L4) as they are less noisy while still pushing a decent volume of air with good pressure. In my experience Noctua and similar quiet fans were not sufficient to keep drives and add in cards a comfortable level of cool.

 

[ude6]

Thanks Aestr. I guess that is the way to go.
Can I swap just a subset of the 7 fans to the FAN-0104L4, or do they all have to be identical?
Also I am looking at the Xeon E5-1650 v4 as a CPU. Should I be using the active heatsink?

 

[Aestr]

They don't have to be identical, but I'm not sure how much value you'll get out of swapping just some. You could play around with replacing 4 and removing the other fans and seeing how temperatures are. If you're not filling the entire chassis for now you might not need all seven fans.

I'd use the active heatsink if you're going to reduce the fan performance. I've swapped out the fan on the Supermicro active heatsink with a Noctua and it still works quite well.

 

[ude6]

Thanks. Thats good to hear. Which Nctua did you use for the heatsink? I assume the sink is:
Supermicro SNK-P0048AP4?

 

[Aestr]

Yes that's the heatsink. I've linked the fan below.

Amazon.com: Noctua 60x25mm PWM A-Series Blades with AAO Frame, SSO2 Bearing Premium Cooling Fan NF-A6x25 PWM: Computers & Accessories

 

[ude6]

That looks good. I will be using the same I think. Now I will have to decide on the right HBA to use.
I will use Linux/ZFS so its IT mode. Is the LSI Megaraid 9211-8i still the "go-to" card? Or should
I be using something newer? LSI SAS 3008? Is there an advantage to using 2 cards? (One for front 24 drives
and one for the back 12?)

 

[sfbayzfs]

Before you start - backplane check:

Have you confirmed that the backplanes are SAS2 expander, and NOT SAS1 expander? Although the 45-bay JBOD version of the 847 was only sold with SAS2 expander backplanes, there are unfortunately a LOT of 36-bay units out there with SAS1 expander backplanes, which you will need to replace (I am about to e-waste a whole pile of them) I regularly see sellers on ebay trying to get $600+ for units with the old SAS1 expander backplanes in them, but at current prices, you would need to spend $300-400 to replace the 2 backplanes with good SAS2 expander backplanes these days. I would therefore not value a complete SAS1 expander based unit higher than about $300 at the most.

Stock backplanes in the SAS1 units were:
BPN-SAS846-7EL1 / BPN-SAS826-7EL1 for single expander
BPN-SAS846-7EL2 / BPN-SAS826-7EL2 for double expander
...but regular separate models can also be found in them:
BPN-SAS846-EL1 / BPN-SAS826-EL1 for single expander
BPN-SAS846-EL2 / BPN-SAS826-EL2 for double expander

If the rear backplane does NOT have a metal cage supporting it, it is almost definitely the SAS1 expander type.

For fans:

I ran a 36-bay SC847 filled with 5400RPM Hitachi drives for ages in a basement with no heat issues. At some point in this process, I moved the drives into a SC847 45-bay JBOD instead to be able to fit a few more drives in, but the thermal and noise profiles are basically the same.

First, I traded the older 0.8A stock fans for the newer 0.6A stock fans, and I pulled 3 of the 7 fans, and let the motherboard PWM the remaining 4, causing a good degree of noise reduction, and plenty of airflow. I bought a set of 7 Noctuas, but they didn't move quite enough air for me to feel safe with them. Next, I tried four of my Panaflo 0.19A fans and they moved barely enough air and probably would have been fine with 7 of them instead of just 4 (the Panaflos do work great for a SC836 if you trade them in for all 5 fans), so not so long after I swapped the Panaflos for 4x 0.35A Supermicro exhaust fans. I had a rear expander go flakey on me, so in case it was a heat issue with the expander heatsink, I swapped the fan that points at it for a 0.55A Supermicro exhaust fan just in case.

To put any non-38mm thick fans into the stock carriers, you need to mod them by cutting off the finger notch.

Supermicro exhaust fan carriers are not compatible with the mounts (probably to prevent people from accidentally partly reversing airflow) so you do need to trade them into the modded stock carriers. I got a whole bunch of the exhaust fans to retrofit all of my stuff, but as usual, I got too many, so I'll be adding the extras to my FS/FT post at some point.

For controllers:

You get a slight bandwidth boost from using a 2308 controller instead of a 2008, or going to one 2008 card per backplane instead of splitting a card (single link per backplane) or daisy-chaining the backplanes (dual link to the front, and then single link to the rear) but overall a single 2008 card dual linked to the front, daisy chained to the rear is just fine. Going to a 2308 in front and a 2008 for the rear is about the best you can do bandwidth-wise, the only reason to go to a separate 2308 for the rear too would be for IOPS if you have a workload that needs them.

 

[ude6]

Thanks @sfbayzfs :
The backplane Looks good it is a BPN-SAS2-826EL1 / BPN-SAS2-846EL1. Therefore they are SAS2 and should be fine.

As for the Controller I will start using the on-board Controller on the Supermicro X10SRH-CF for a start.
I will order a Special SAS3/SAS2 conection cable and takew it from there.

Fans: That is really my main Point... I guess I will just start by connecting the fans to the MB and see
where that takes me. Do you have a part # for the Panaflows that work for you?

 

[sfbayzfs]

The Panaflos I use are marked:
NMB-MAT
5L20A73 -1A
Model FBA08A12M
12V 0.17A

3-pin only, but they do a great job and definitely keep things cooler than Noctua NF-A8-PWM

I have lots of extras since the only way I was able to find that model when I was originally looking was to get a case of them. I have them listed at $7 each including a Y adapter in my second FS/FT post, but if you need a lot, and don't need the Y cables, I can do a bit better!

In my SC847 chassis' I have moved to FAN-0125L4 in a modded FAN-0126L4 / FAN-0099L4 carrier, which retain hotswap capability, and are 4-pin PWM
Part Number - (A) (RPM) (CFM) Pressure (dBA)
FAN-0125L4 - 0.35 6,700 59.6 0.68 47.0
FAN-0126L4 - 0.60 7,000 72.5 1.09 53.5 (new SC847 stock fans)
FAN-0099L4 - 0.80 6,300 90.3 0.85 51.0 (old SC847 stock fans, which sound louder than the new ones to me despite specs)

Those ratings are at full tilt of course, I find that in a JBOD the 0.35A in a SC847 JBOD PWM down to a lot quieter than the stock ones - the fan drone is a lower pitch and intensity, although still quite noticeable in the same room. In contrast, the stock fans are noticeable 2 rooms away around corners, and a higher pitch as well.

PWM controlled by the motherboard though, even the stock fans can be quiet depending on which motherboard and fan profile you use, especially with X10/X11 fine grained IPMI fan speed control.

That said, I don't have as much time as I would like to mess with this stuff, so after finding how quiet I could safely get my chassis' I decided to take a better safe than sorry approach. Most of my stuff is X8 and X9, so I don't have the fine grained IPMI fan speed control, so in chassis' with motherboards, I use the workstation / quiet fan profile, and use Panaflo, 0.35A or 0.60A fans depending on the chassis. For JBODs, I am going to all 0.35A FAN-0125L4 fans in modded carriers because I know that in my situation they can move enough air even on the hottest days, are quiet enough, and don't draw a ton of power.

Backplane firmware:

A few months ago, I updated the firmware on most of my chassis' SAS2 expander backplanes which were running old versions. You have to flash the firmware into 2 regions, and the "MFG" file for the model specific features such as drive id lights and fan control. The MFG file MUST match / be compatible with the firmware version or drive identification lights do not work. Anyway, there are 2 variants of the MFG file for each backplane - does anyone know what the difference is between the regular and "nofan" versions is? Are you supposed to use nofan with a motherboard when it is controlling the fans, or in a JBOD, or is it related to fan speed control or buzzers / alarms or something else? If we could somehow get better PWM control of fans in a JBOD, even if it requires sending SAS enclosure commands through the HBA, I'm sure lots of folks would be happy

 

[sfbayzfs]

Here are my drive temps the night after two record hot days in a row, things have been holding the heat, and ambient temps in the basement have gotten higher than I have seen before.

These have been running with four of the 0.35A fans instead of 7 of the 0.60A or 0.80A ones, a couple of them may have a single 0.55A fan pointing at the SAS2 expander chip (second fan from the PSU side of the case). Most of them have been actively copying for the last couple of days
# of Drives / temp
Hitachi / HGST 4TB coolspin, 2 units, top of rack and mid-rack:
3 Temperature_Celsius 37
10 Temperature_Celsius 38
11 Temperature_Celsius 39
10 Temperature_Celsius 40
7 Temperature_Celsius 41
2 Temperature_Celsius 42
12 Temperature_Celsius 43
6 Temperature_Celsius 44
3 Temperature_Celsius 45
4 Temperature_Celsius 46
3 Temperature_Celsius 47
2 Temperature_Celsius 48
1 Temperature_Celsius 49
WD80EZZX white label shucked externals, lower middle of rack ** in a SC836 with a motherboard, with the same fans I'm pretty sure, but I need to double-check when I get a chance:
1 Temperature_Celsius 35
3 Temperature_Celsius 36
2 Temperature_Celsius 37
4 Temperature_Celsius 38
3 Temperature_Celsius 39
3 Temperature_Celsius 40
Each of the SC847 JBODs in use right now is full other than 7-9 drive baysI think the cooler 4TBs are all in the mid-rack SC847 JBOD, and the hotter ones are un the unit at the top of the rack where the heat rises to, but I would have to turn on some drive lights to be sure.

Overall that looks pretty good to me, well within operating specs, and nothing even at 50+

 

[K D]

@sfbayzfs

Are these temperatures normal? 4 drives are 7200rpm HGST NAS drives and the rest are all coolspin drives.

I am using the PanaFlo 0.17A fans in an 846 Chassis. Previously when I had smaller number of drives, I never worried about temperatures and have had drives run at 50deg for years without issues. But the word around the internet (especially in the FreeNAS focumes) seems to be that drives should be maintained at ~30degrees and FreeNAS triggers an alarm when a drive hits 40deg.

I am extremely happy with this setup in terms of noise levels. I can actually keep it next to me in my office when I'm working or in conference calls without it being a distraction. The max temp I've seen the drives hit is around 45deg when I replaced a drive and the pool was rebuilding.

da0 : 39 C
da1 : 39 C
da2 : 38 C
da3 : 41 C
da4 : C
da5 : 41 C
da6 : 41 C
da7 : 39 C
da8 : 39 C
da9 : 40 C
da10 : 41 C
da11 : 39 C
da12 : 38 C
da13 : 38 C
da14 : 38 C
da15 : 37 C
da16 : 37 C

 

[sfbayzfs]

Those temps look great to me - My temps were during a heatwave, and I consider them to be great, especially since it was a heatwave, and they are well within the rated operating temps of 5-60C:
https://www.hgst.com/sites/default/files/resources/DS5K4000_ds.pdf

Slight rant:

Unfortunately there are a lot of recommendations in the FreeNAS forums which are misleading IMHO, and push you towards worrying about nonissues, and generally spending a lot more than you need to on a system that may be less stable than a cheaper system would be, such as:

• You must have at least 1GB of RAM per 1TB storage - unless you are running highly random write heavy workloads, this is completely overblown, especially for the most common use case of bulk storage and personal backups. I have run a 114TB zpool with "only" 32GB for years, and knocking it down to 16GB only marginally slows it down.
• You should divide your zpools up into VDEVs of 2, 4, or 8 data drives plus parity - the excuses are that a 128K block doesn't divide easily into non-power of two drive counts, and that parity calculations are too expensive, but the data drive count per VDEV thing has been debunked, if you enable large blocks of 1M, it is even less of an issue, and parity calculations are not that expensive on even older CPUs - a Sandy Bridge Xeon e3-1260L can easily push 1GB/sec on to a single Z3 VDEV of 33+3 drives in my experience.
• Don't use a motherboard which isn't brand new and current generation X11 - In Intel/Supermicro terms, they say Avoton / Rangely / Haswell are already too old (X10 and earlier) , and recommend you buy a new X11 Xeon e3 or e5 board. Actually, I prefer an older board with more stable BIOS, patched CPU issues, etc. so I don't have to worry about or mess with it, and it's cheaper too! An x7 DDR2 board runs the latest FreeNAS great, and is easy to load up with a low power quad core and cheap ECC RAM if you need it - I know several people doing this with great results! The only X7 limitation is it seems the FreeBSD behyve VM hypervisor they use doesn't work well for non-FreeBSD guests on X7 CPUs, but X8 and newer supports everything needed VM-wise, and will probably run FreeNAS perfectly for ages! The X11 Skylake/Kabby Lake hyperthreading problems have only recently been patched, I'm glad I haven't been relying on them since they were released!
• Keep your drives as close to 30C as possible, 40C is a problem...

(end rant)

According to the long term google study on drive failures:
https://static.googleusercontent.com/media/research.google.com/en//archive/disk_failures.pdf
...interpreted and summarized here (and in many other places):
Google’s Disk Failure Experience

...low temperatures were generally much worse for drives than high temperatures, and high temperatures were only bad when the drives got to be 3+ years old at temps of 50C+.

Based on my reading of the stats in that study, running drives below 35C is bad, you should ideally run your drives in the 35-45C range if possible, and temps of 50C should be just fine until your drives have been running for 3 solid years or more.

Yes, the study is ~10 years old now, but while capacity has gone up a lot, there have been no major design changes or other breakthroughs that seem to increase drive reliability in that time. Helium may help, but it will probably leak out eventually, so I would call it a wash or negative once the Helium drives get to be very old.

 

[miked1]

@sfbayzfs

What is your ambient and an idea of an approx dBA for a -847 (either JBOD and/or w/MB) running the -125l4s? I'm currently running 4 -126l4s and while keeping drives cool, their (when the system is mostly idle) drone can be heard clear across the house when it's quiet. I'm looking at the -125l4s to quieten the chassis further.

 

[Churchill]

I built 30 of these for a Openstack Swift (moved to Gluster) project. These are workhorse servers, loud, powerful, and need solid power. These are datacenter servers, not quiet at all.

I wouldn't worry about drive temps till they hit high 40s.

You have 2 options to make them quiet:

1. SQ power supplies
2. Replace all the fans with Noctuas (or similar quiet types)

I managed to silence a SM server to where I barely know it's there from the hurricane that it was.

 

[funkywizard]

Hi all,
this is my first post to the forum.
I am looking to build a (probably too large) NAS for my home use. It would
reside in the basement but if the thing is too loud you would hear it through the
ceiling....
That leeds to my question: I have a good offer for a Supermicro-SC847E16-R1400LPB
chassis. But it seems that the noise of the stock fans and powersupplys is legendary.
There are a number of threads around that start discussing quieting the "beast" but
then seem to lead to nothing.
So my question: Has anyone succeeded? Is it enough to use PWM fans and atch to the
MB? Is exchaning fans an option or do temperatures "explode"?

Thanks for any hint
ude6

Looks like it has 7x 80mm fans -- FAN-0126L4

Seems like overkill.

You can reduce the fan speeds to whatever the motherboard thinks is a good idea. Inside linux with ipmitool installed:

ipmitool raw 0x30 0x45 0

^ "0" sets "standard" fan speed on supermicro. standard means PWM speeds controlled based on system / cpu temperatures. "1" sets fans to full speed. "2" sets to "optimal" which really just means fans it thinks are cooling pcie slots will run a bit slower than when set to "0", under light load anyway.

I would be tempted to remove the second row of 3 fans, but that might not be a good idea.

 

[funkywizard]

The stock PWM fans with proper settings are reasonably quiet and SQ power supplies are great. From there you could look at the fans in the 745SQ chassis (FAN-0104L4) as they are less noisy while still pushing a decent volume of air with good pressure. In my experience Noctua and similar quiet fans were not sufficient to keep drives and add in cards a comfortable level of cool.

I would be a bit worried about using fans that much slower. Maximum 2,800 rpm vs maximum 7,000 rpm. The higher rpms are primarily to produce extra static pressure, moreso than airflow -- i.e. suck / push air through tight spaces. I'm pretty sure this is why they have two rows of fans as well -- this configuration increases static pressure far more than it increases airflow. If you don't need all that cooling power, I would rather remove one row of fans the server comes with, instead of replace them with 7 fans that are that much slower.