SuperMicro SC836 HDD Temps Too High / Fan Replacements / Your Experiences?

[svtkobra7]

Hi,

As my first foray into a non-COTS server, I purchased a used SuperMicro Super Chassis SC836E16-R1200B via ebay. Having sold my QNAP NAS appliances to offset the cost, I was left with 16 HGST Deskstar NAS HDDs - 6TB that needed to be repurposed. I spent considerable time researching and planning my build and it is well noted that the Hardware Guide explicitly recommends NOT using 7200 RPM hard drives due to heat produced; however, in my scenario it would not have made sense to source 5400 RPM drives when I already had several $k worth of 7200 RPM NAS HDDs looking for a home.

I've thought for a number of weeks that I had some sort of hardware or software misconfiguration, as the only fan profile that I'm able to achieve sub-40°C HDD temperatures with is the "Full Speed" fan profile. Needless to say, running the server with that fan profile in a home environment is not a sustainable solution. As of last night, I believe I may have identified the issue, but I'd like to share my issue and data with the community for your thoughts / confirmation.

• Specifically, I'm curious what HDD temperatures other forum members have encountered with the same chassis,
• I'd like your thoughts as to whether not having the correct fans installed is the cause of high HDD temperatures,
• and finally, your recommendation on the the most appropriate (a) chassis middle fan and (b) exhaust fan for this chassis.

Without further preface ...

1. Define: Achieve HDD temperatures of 40°C or lower at a "reasonable" sound level and at desired ambient temperature, 78°F.

• 78°F is the ambient temperature I find most comfortable during the summer and 72°F is far too cold.
  
• I don't want to incur the additional cost of lowering ambient temperature to 72°F to achieve the goal (I'm sure I'm already in for a treat when the power bill arrives with dual E5-2670s and 16 spinning disks @ 7200 RPM ).
  
• Modifications to hardware configuration are absolutely acceptable solution (and one I propose at that). If air shroud removal and usage of active CPU coolers would help, please let me know (I've read differing opinions as to whether the air shroud should be left in place).

2. Measure: HDD temperatures were recorded for each drive at each fan profile with ambient temperatures of 72°F / 75°F / 78°F using Spearfoot's (FreeNAS Forum Member) "get_hdd_temp" script.

• At each intersection of (a) fan profile and (b) ambient temperature combination, 3 measurements were recorded at t0, t+15 min, and t+30 min. For each of those intersections, the mean of those 3 measurements was used for analytical purposes.
  
• Where ambient temperature was decreased or fan profile was changed, a given profile was allowed to operate for a minimum of 30 minutes once ambient temperature was stable.

3. Analyze: The full speed fan profile is the only profile that achieves sub-40°C drive temperatures at the desired ambient temperature, 78°F. The standard and optimal profiles produce results with a maximum variation of 0.4°F at any measured ambient temperature, but fail to achieve sub-40°C drive temperatures even at the lowest ambient temperature used, 72°F. A summary of my data follows, and I'm happy to pass along the Excel file if you would like to review.

4. Improve: To be completed. 5. Control: To be completed

Proposed Explanation

• I believe the explanation as to why I cannot keep my HDDs cooler and why only the "Full Speed" fan profile can achieve sub-40° temperatures has been staring me in my face, I just have been too blind to see it. Other items that suggest fans are the issue: (a) CPU2 (rear) measuring 20°C warmer than CPU1 (front), and (b) realizing that my fans where not spinning at 100% of what I thought should be their maximum rotational speed. Yet, I didn't connect the dots until last night.
  
• After much (over) analysis, I looked at the fans installed in the server (as purchased from ebay), and compared them to the SuperMicro's specifications for the SC836E16-R1200B and noticed that they differed.
• Those familiar with the chassis / backplane know that the "cut outs" in the backplane for airflow are quite small and the "chassis middle" fans pull air from the front of the chassis / hot swap drive bays through the backplane via these "cut outs". As such, I would assume a fan rated at as high a static pressure as possible is needed to effectively cool the HDDs.
• Per the SuperMicro System Fan Matrix, the 3 chassis middle fans fans that came with the system are listed at 0.85 in. H20 static pressure, vs. the specified fan for the SC836E16-R1200B which is rated at 1.09 in. H20 static pressure. The installed fan's speed is lower by 700 RPM and airflow is 17.8 CFM higher than the specified fan.
• Again, per the SuperMicro System Fan Matrix, the 2 exhaust fans that came with the system are listed at 0.49 in. H20 static pressure, vs. the specified fan for the SC836E16-R1200B which is rated at 0.68 in. H20 static pressure. The installed fan's speed is lower by 1.7k RPM and airflow is 0.5 CPM lower than the specified fan.
• In summary, I assume those static pressure and air flow specifications are at maximum speed, where I'm missing 0.24 in. H20 at the backplane (per fan x 3) and I'm missing 0.19 in. H20 at the rear (per fan x 2). The "Optimal Speed" Profile uses a base fan speed at 30% PWM duty cycle, so perhaps the variance in static pressure is much larger at that duty cycle, which would account for its ineffectiveness? My question to you: Does this proposed explanation make sense?
• As an ancillary question, I have the backplane fans connected to FAN2-4 and the exhaust fans connected to FAN5-6, is this the correct? My understanding is that since all fans are deemed "CPU / System" fans, they belong in Zone 0 (FAN1-6) and there is no need to use FANA-B, Zone 1, for Peripheral Zone Fans.
• Note: A full summary of my hardware configuration including fan specification follows at the end of this post.

Proposed Solution

• Replace 2 x Exhaust Fan (80 x 80 x 32 mm). For this fan size, the Fan Matrix shows the only alternative to the incorrect fan I have (FAN-0062L4) to be the same one listed in the specifications for the SC836E16-R1200B. That correct fan is the FAN-0125L4.
  
• Replace 3 x Chassis Middle Fan (80 x 80 x 38 mm). For this fan size, the Fan Matrix shows 17 fans. Should I procure the one specified for the SC836E16-R1200B, 3 x FAN-0126L4, or is there a better alternative?
• Note: I've summarized the Fan Matrix in a shared spreadsheet located here: Fan Matrix - Extract.

Thank you in advance for your review and feedback.
____

[Chassis Middle Fans] / {Specified, per Supermicro's website} or (Installed, as Purchased)

• {Specified} = 3 x FAN-0126L4: 80 x 80 x 38 (mm), Fan Speed (RPM) = 7.0K, Airflow (CFM) = 72.5, Static Pressure (in. H20) = 1.09, Noise (dBA) = 53.5
• (Installed) = 3 x FAN-0094L4: 80 x 80 x 38 (mm), Fan Speed (RPM) = 6.3K, Airflow (CFM) = 90.3, Static Pressure (in. H20) = 0.85, Noise (dBA) = 51

[Exhaust Fans] / {Specified, per Supermicro's website} or (Installed, as Purchased)

• {Specified} = 2 x FAN-0125L4: 80 x 80 x 32 (mm), Fan Speed (RPM) = 6.7K, Airflow (CFM) = 59.6, Static Pressure (in. H20) = 0.68, Noise (dBA) = 47
• (Installed) = 2 x FAN-0062L4: 80 x 80 x 32 (mm), Fan Speed (RPM) = 5.0K, Airflow (CFM) = 59.1, Static Pressure (in. H20) = 0.49, Noise (dBA) = 47

Hardware Configuration

• Chassis: SuperMicro Super Chassis SC836E16-R1200B - 3U 16-bay
• Motherboard: SuperMicro X9DRi-LN4F+, 2 x Intel Xeon Processor E5-2670 (passively cooled), 48GB ECC RAM
• Backplane / HBA: SuperMicro BPN-SAS2-836EL1 / LSI SAS 9211-4i
• Storage Drives: 16 x HGST Deskstar NAS HDD - 6TB (8 in service + 8 in burn-in + 4 in transit)

[FDD/DVD Size Drive Bays Above 3.5" Drive Bays - Position] / {Original} / (Replacement)

• [LEFT SLOT] / {DVM-PNSC-DVD-SBT3: DVD-ROM Drive} / (Open, to be replaced with MCP-290-00036-0B: DVD Dummy Tray for 2.5" HDD)
• [MIDDLE SLOT] / {CP-220-00007-01: USB/COM Port Tray} / (MCP-220-00023-01: USB Dummy Tray for 2.5" HDD, populated with Intel 320 Series SSD - 40 GB)
• [RIGHT SLOT] / {MCP-290-00015-01: FDD Dummy Cover} / (MCP-220-83601-0B: FDD Dummy Tray for 2.5" HDD, populated with Intel 320 Series SSD - 40 GB)

[Cross Posted @ FreeNAS Forum - I've discussed the issue at length with members from both forums and would like to give all the opportunity to view replies]

 

[i386]

836b chassis, a backplane
3x middle wall fans (FAN-0126L4) replaced with 3x FAN-0074L4
hdd position (x):
OOOO
OXOO
OOOO
OOOO

 

[svtkobra7]

View attachment 5573 View attachment 5574
836b chassis, a backplane
3x middle wall fans (FAN-0126L4) replaced with 3x FAN-0074L4
hdd position (x):
OOOO
OXOO
OOOO
OOOO

Interesting, so you went to a fan with less speed, air flow, and static pressure v. the one I have installed, dropped 8.5 dBA in noise (compared to factory spec) and are still cooler than I am, also with 7200 RPM drives.
What is your ambient temp?

 

[i386]

Currently 20°c

 

[Morpheus187]

I'm not sure if you will be able to reach sub 40 C without screaming noises from the fans.

I've had some old test 2 TB 7200 RPM disks installed in my 2u chassis and they were constantly above 40 C when idle. Now I've got my production disks ( 6 TB WED RED ) in place and with a bit tuning ( using tape to force airflow trough the disks and not above them ) I'm at about 34-36 C ( depends on position, bottom right is the hottest disk )
Ambient temperature is about 24-26 C


What I've though of, but not implemented is using some sort of heat-transfer-pads below the hard disks to make physical contact with the chassis to transfer heat, but in your chassis that might only work for the bottom row.


I'm nonetheless interested if you find anything that helps cooling those disks.

 

[svtkobra7]

Now I've got my production disks ( 6 TB WED RED ) in place and with a bit tuning ( using tape to force airflow trough the disks and not above them )

• I was able to drop my temps by 1°C by forcing the backplane fans to pull air through the drive bays and not the DVD-ROM / FDD profile bays above the HDDs. I had the DVD-ROM bay open, which I applied painter's tape to as a temporary solution and covered the grates on the dummy USB & FDD trays with velcro on the inside.
• Care to share more detail about your "tuning"?

What I've though of, but not implemented is using some sort of heat-transfer-pads below the hard disks to make physical contact with the chassis to transfer heat, but in your chassis that might only work for the bottom row.

• Interesting idea, care to share any additional details (either by link to the transfer pad you are referring to or description)?
• I would think this would be tough to accomplish as the underside of spinning disks have a PCB and injection molded form, whereas the top is metal.
• If you reference the attached imagine, you will see that my drive temps are warmest on the bottom row. I'm not sure why this is other than the fact that the bottom row is literally sitting on the "floor" of the chassis and the fans don't extend down to the "floor" all the way.

I'm nonetheless interested if you find anything that helps cooling those disks.

• The correct fans (3 x FAN-0126L4 + 2 x FAN-0125L4) will be delivered tomorrow, so I will certainly reply back with an update.
• Considering they will provide more static pressure, I expect favorable results. Favorable enough to hit sub-40 w/ optimal profile, I hope, but I'm not sure if that will get me there. Additional static pressure = 0.24 in. H20 at the backplane (per fan x 3).

 

[Morpheus187]

I did the same, I used tape to cover the airflow above the hdd's to force the air trough the hdd slots ( I have only 2 HE chassis )


I thought about pads like this: PHOBYA Wärmeleitpad Ultra 100x100x1.5mm (5W/m K) - digitec

You can order then in different sizes and thickness, so in theory it should be possible to stick it under/above the disk and establish contact with the chassis.

In my case the bottom right disk is the hottest because there is no fan for that disk, just the PSU.

Hope you get luck with the better fans.

 

[svtkobra7]

4. Improve: In process. 5. Control: To be completed.

I received the "correct" fans for my SC836E16-R1200B and I would like to share updated HDD temperature measurements with the community. I hope that my efforts here will benefit someone else who encounters a similar issue or that other community members learn something from from my analysis.

I've proposed "Next Steps" and a single "Question" at the end of this post - I would be appreciative if anyone is able to call out any flaw in those proposed next steps and/or answer that question.

As a refresher, my proposed explanation for HDD temperatures being high was my supposition that this was due my SC836E16-R1200B (purchased used via ebay) not being populated with the fans specified for this chassis on Supermicro's website. In my measurements, reference to "NEW" refers to those "correct" (now installed, as specified) fans, and reference to "OLD" refers to the "incorrect" (previously installed, as purchased) fans, specifically as follows:

• "NEW" (Now Installed, Specified per Supermicro's website): Chassis Middle = 3 x FAN-0126L4 + Exhaust = 2 x FAN-0125L4
• "OLD" (Previously Installed, as Purchased) = Chassis Middle = 3 x FAN-0094L4 + Exhaust = 2 x FAN-0062L

Note: A summary of the fan's specifications is included at the end of this post.

Results - Summarized, Qualitative (mostly)

• The results are favorable and attributable to the extreme importance of using fans with sufficient static pressure. With the "OLD" / "incorrect" fans (vs. the "NEW" / "correct" fans), I was missing static pressure = 0.24 in. H20 at the backplane (per fan x 3) and static pressure = 0.19 in. H20 at the rear (per fan x 2).
• Temperatures have decreased -2.9°C or better for each ambient temperature and fan profile combination.
• My target was/is sub-40°C HDD temperatures at an ambient temperature of 78°F using the "Optimal" Fan profile was and I'm very close now (with a mean of 40.3°C).
• As measured by an un-calibrated sound level app on my phone, there is no difference in noise level with the "NEW" fans.

OK - to the data …

Results - Summarized, Quantitative

• Ambient @ 78°F: Mean reduction of -3.1°C with "NEW" fans. ("Full" = -3.1°C, "Standard" = -3.0°C, "Optimal" = -3.2°C)
• Ambient @ 75°F: Mean reduction of -3.0°C with "NEW" fans. ("Full" = -3.0°C, "Standard" = -2.9°C, "Optimal" = -3.1°C)
• Ambient @ 72°F: Mean reduction of -3.8°C with "NEW" fans. ("Full" = -3.0°C, "Standard" = -3,9°C, "Optimal" = -4.4°C)

Results - Ambient @ 78°F

• "Full" Fan Speed Profile: Reduction of -3.1°C with "NEW" fans.
• "Standard" Fan Speed Profile: Reduction of -3.0°C with "NEW" fans.
• "Optimal" Fan Speed Profile: Reduction of -3.2°C with "NEW" fans.

Results - Ambient @ 75°F

• "Full" Fan Speed Profile: Reduction of -3.0°C with "NEW" fans.
• "Standard" Fan Speed Profile: Reduction of -2.9°C with "NEW" fans.
• "Optimal" Fan Speed Profile: Reduction of -3.1°C with "NEW" fans.

Results - Ambient @ 72°F

• "Full" Fan Speed Profile: Reduction of -3.0°C with "NEW" fans.
• "Standard" Fan Speed Profile: Reduction of -3.9°C with "NEW" fans.
• "Optimal" Fan Speed Profile: Reduction of -4.4°C with "NEW" fans.

Next Steps

• Decouple CPU and HDD cooling and manage via fan script (my understanding is that with a X9 board, I will only have the luxury of the script adjusting fan speed between low / medium / high).
• Install the Supermicro 2U Active CPU Heat Sinks (SNK-P0048AP4) that I have purchased.
• Connect 3 Backplane fans to FANA using PWM splitter.
• Connect CPU fans to FAN1-2.
• Connect Exhaust fans to FAN3-4.

Question

• I'm observing significantly higher temperatures for "Slots" 12 and 13, which are the 3.5" HDD Drive Bays at bottom, right. Reference one of the above diagrams for my labeling schema.
• +5.3°C higher than lowest HDD temperature @ 78°F Ambient using the "Full" Speed Fan Mode (40.0°C v. 34.7°C)
• +6.0°C higher than lowest HDD temperature @ 78°F Ambient using the "Standard" and "Optimal" Speed Fan Modes (44.0°C v. 38.0°C)
  
• Is this to be expected and simply because those bays are directly in front of the PDU and there is no backplane fan directly behind them? If that is the explanation, it doesn't quite explain why there is still a temperature variance between the top and bottom bays in that column.

Thank you for all of your input and assistance.

____

Fan Specifications per Supermicro's website:

[Chassis Middle Fans] / {Now Installed, Specified per Supermicro's website} or (Previously Installed, as Purchased)

• {"NEW"} = 3 x FAN-0126L4: 80 x 80 x 38 (mm), Fan Speed (RPM) = 7.0K, Airflow (CFM) = 72.5, Static Pressure (in. H20) = 1.09, Noise (dBA) = 53.5
• ("OLD") = 3 x FAN-0094L4: 80 x 80 x 38 (mm), Fan Speed (RPM) = 6.3K, Airflow (CFM) = 90.3, Static Pressure (in. H20) = 0.85, Noise (dBA) = 51

[Exhaust Fans] / {Specified, per Supermicro's website} or (Installed, as Purchased)

• {"NEW"} = 2 x FAN-0125L4: 80 x 80 x 32 (mm), Fan Speed (RPM) = 6.7K, Airflow (CFM) = 59.6, Static Pressure (in. H20) = 0.68, Noise (dBA) = 47
• ("OLD") = 2 x FAN-0062L4: 80 x 80 x 32 (mm), Fan Speed (RPM) = 5.0K, Airflow (CFM) = 59.1, Static Pressure (in. H20) = 0.49, Noise (dBA) = 47

____

[Cross Posted @ FreeNAS Forum - I've discussed the issue at length with members from both forums and would like to give all the opportunity to view replies]

 

[Rand__]

• Is this to be expected and simply because those bays are directly in front of the PDU and there is no backplane fan directly behind them? If that is the explanation, it doesn't quite explain why there is still a temperature variance between the top and bottom bays in that column.

I always assumed the upper ones would get draft from the PSU's leaving the bottom ones dry in the sun...