Following up:
Full details so others with similar setup can benefit:
Supermicro 836 case, 920-SQ power supplies, SAS2 EL1 backplane.
MB is X9SRL-F, CPU is E5-2680 V2 (10 core, 2.8 GHz base, 3.6 turbo, 115W TDP). I observed this MB reports fan RPM in 75 RPM increments and that's also when the IPMI Fan thresholds can be set in increments of.
Stock fans 3x FAN-0126L4 in fan wall (replacements are FAN-0074L4), 2x FAN-0125L4 in rear (replacements are FAN-0104L4).
CPU cooler is Noctua NH-D9DX i4 3U, with its stock, single 92mm fan.
I am running Unraid (first time user). Tried various fan control methods (IPMI Tools plugin, Dynamix Auto Fan plugin), using ipmitools CLI , other than altering high/low thresholds using the CLI, could not get any to work reliably, so using the stock IPMI "Optimal" fan profile. I may play with user scripting later but for now I am calling it "good enough".
MB has fan headers FAN1-FAN4 and FANA.
Rear fans are both on header FAN4, mid wall fans are individual on FAN1-FAN3.
My understanding is the MB based IPMI fan control algorithm is designed around the CPU temp for FAN1-FAN4. FANA I am uncertain of, but it's not CPU temp. (this per this resource, optimal = 30% PWM:
https://forums.servethehome.com/index.php?resources/supermicro-x9-x10-x11-fan-speed-control.20/ )
Originally had CPU fan on FANA but it would always just run slow and not ramp up. I moved it to the backplane which runs it at 100% all the time. This is fine as it's a quiet fan. I put together a fan cable so the sense signal is on FANA so that I can still get RPM monitoring/alarming. I experimented with the Noctua "low noise adapter", results below.
Put 8 SATA hard drives in the chassis (1 is a "permanent" 8TB WD white label, rest are temporary just for this testing, various old pulls from work, 500GB-1TB 5400 RPM/7200 RPM). The 8 unused drive bays did NOT have the Supermicro "dummy trays" in them, so I covered half of the vent holes in each tray with tape to simulate the air flow restriction of a hard drive being present.
My goal: low enough hard drive and CPU temps with the fans being relatively quiet. This is in a 4 post rack in my utility room, but it's directly adjacent to my home theater (under construction) so both volume and frequency of the sound matters in terms of it being obtrusive. I did not go to the effort of taking SPL or frequency spectrum measurements with REW or anything (at least not yet!), just using my ears.
Wanted HD temps at or below 35 deg C when under load, CPU < about 70 deg C at load but not primary concern.
Utility room is in a basement in Minnesota (northern North America) so temps are pretty low year round (generally 63-72 deg F).
Impressions:
At IPMI Optimal idle the stock SM fans are not too bad. I would not want to sit next to the case all the time, but sitting one floor above I could not hear it. Running flat out of course you can hear them through the whole house.
I ran experiments as follows:
1) Note idle temps and noise and RPM for a particular setup
2) Writing 0s to all 7 temporary hard drives (using the "preclear" plugin). This put the CPU at about 60% utilization. Note temps and noise and fan RPM
3) Swap out the SM fans, first fanwall, then retest. Then rear fans too, retest. Then try Noctua Low Noise Adapter on CPU fan, retest.
4) FAN RPMs and CPU temp from IPMI. Drive temps from Unraid (via SMART). Ambient temps from IoT device.
Conclusions:
0) The advice to just get the quieter SM fans was solid - for my use case there is no need for the surgery and hassles (especially in a 3U case!) involved in retrofitting 120mm fans - thanks
ramblinreck47 and others!
1) Leave the Noctua at 100%. Noise delta is minimal at idle (the SM fans dominate). At 100% CPU utilization, using the low noise adapter causes higher CPU temps (not much - 2.5 deg C) but the SM fans spin up to a higher RPM due to this increase. This more than overwhelms the noise savings from the low noise adapter - just leave it out.
2) The Noctua CPU cooler is super effective for this CPU TDP anyway. Max CPU temp at 100% Noctua fan speed was 57.5 deg C and the SM wall fans barely spun up. If the SM fans were kept at the IPMI Optimal idle speed regardless of CPU temps, then using the LNA may make sense.
3) At 60% CPU usage, CPU temp rose 10 deg C from idle but the IPMI Optimal algorithm left the fans at idle speed. Drive temps writing 0s flat out were only about 3 deg C higher than baseline idle. All were comfortably below 35 deg C with all fan configurations. This actually tells me that my best long term bet may be a fan control that just leaves the SM fans at their idle RPM all the time, no matter the CPU or drive temps. This should not be too difficult to implement (may be able to use simple IPMI commands in a script to do this if I can get ipmi PWM control to work as many others have on X9 and up mainboards).
4) The "fresh after a reboot of the OS" Fan RPMs were back to idle (yes, reboot of the OS, not the BMC). Even though I have uninstalled the plugins I played with, this implies there is still some interaction between the OS and IPMI. I may need to restore the IPMI firmware to factory - more work to do here.
This was not super scientific - none of this is calibrated, I waited about 45 minutes for drive temps to stabilize when writing 0s but not much longer. Probably apply an uncertainty of +/- 1 deg C to each reading due to methodology. i.e. there is no (scientifically/statistically) meaningful difference between a PCH temp of 48 and 50 deg C or drive temp of 29 and 31 deg C.
All temps but ambient are in deg C, ambient is in deg F.
Raw Results:
Config | stock | stock | replaced mid wall | | replaced rear too | | | | | |
Server load / action | idle | write 0s to all HD | idle | write 0s to all HD | idle | write 0s to all HD | Stress 5 minutes + write 0s | Low noise adapter on CPU fan, all idle | Stress docker to put CPU at 100%, (Low Noise Adapter on CPU), drives idle | after testing, 20 hours later, all idle |
ambient temp (basement) | 68 deg F | 68 deg F | 68 deg F | 68 deg F | 69 deg F | 69 deg F | 69 deg F | 69 deg F | 69 deg F | 69 deg F |
CPU % | 1 | 60 | 1 | 60 | 1 | 60 | 100 | 1 | 100 | 1 |
CPU temp | 27 | 36 | 27.5 | 37 | 28 | 37.5 | 57.5 | 29 | 60 | 29 |
PCH temp | 46 | 50 | 48 | 49 | 48 | 49 | 48 | 48 | 45 | 45 |
ST500DM002 | 24 | 27 | 26 | 29 | 26 | 29 | 28 | | | |
ST500DM009 | 26 | 29 | 27 | 31 | 27 | 31 | 30 | | | |
ST500DM002 | 25 | 27 | 25 | 28 | 26 | 29 | 28 | | | |
ST500DM002 | 24 | 27 | 25 | 28 | 26 | 28 | 28 | | | |
TOSHIBA_DT01ACA100 | 27 | 30 | 28 | 32 | 29 | 32 | 32 | | | |
ST500DM002-1BD142 | 27 | 29 | 27 | 31 | 28 | 31 | 31 | | | |
ST500DM002 | 28 | 31 | 28 | 32 | 29 | 32 | 32 | | | |
cpu fan RPM | 1875 | 1875 | 1875 | 1875 | 1875 | 1875 | 1875 | 1350 | 1350 | 1350 |
Fan 1-3 RPM | 2550 | 2550 | 2025 | 2025 | 2025 | 2025 | 2475 | 2025 | 2925 | 2400 |
Fan 4 RPM | 2550 | 2550 | 2550 | 2550 | 900 | 900 | 1200 | 900 | 1500 | 1125 |
Volume | Decent | no change | quieter | no change | even quieter - good enough | | | only slight change for the better, SM fans dominate noise profile | louder! Not worth it due to SM fans at higher RPM | Fans did not spin all way back down, even 20 hours later. Still, only slightly more obtrusive than furnace fan in its "circulate" mode (equivalent to furnace idle) for both volume and annoyance level of pitch |