P

Supermicro X9/X10/X11 Fan Speed Control

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

DaveInTexas

Member
Oct 28, 2021
77
85
18
Thanks for a great explanation of the way BMC is setup. I have several SM mother boards. Perhaps it is because my X9DR3-F board is one of that does not support Fan control but I tried the commands from a previous thread.

when i use the Truenas shell to type "ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x24" command I get an error message I get says:
"unable to send RAW command channel=0x00 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc
invalid data field in request"
That IPMI command is for Supermicro X10/X11 boards, which is why it didn't work on your X9 board.

raw <netfn> <cmd> [data]

"Network Function Codes:
VAL HEX STRING
==============================================
0 0x00 Chassis
2 0x02 Bridge
4 0x04 SensorEvent
6 0x06 Application
8 0x08 Firmware
10 0x0a Storage
12 0x0c Transport"

You can see that my X9DR3-F motherboard does not have a 0x30 netfn?
Yes. You're right. Thanks for digging into that....

Is there a decoder ring for these X9 boards. I did not find it on that server wiki link you shared. Where are the raw commands defined?
There is no decoder manual and unfortunately, as you've experienced it is a very confusing topic. Sometimes it's possible to find information in documentation produced by the BMC manufacturers, but it's rare and would normally be at the level of the motherboard's embedded software (embedded on its hardware) that talks to the BMC directly, such as an IPMI controller.

The most reliable method of discovery is unfortunately, hands-on experimentation or finding someone who has worked directly with a given board and can validate commands they know work with it. This can be tough to know though, because it's possible someone has entered the commands incorrectly and then believes it doesn't work, without realizing their mistake.

I am so glad you asked these questions and provided feedback, as you prompted me to spend more time on this....

Over time, I've stumbled upon conflicting literature on whether the NetFn byte should be 0x30 or 0x32 for X9 motherboards to control the fans. If one doesn't work for you, I would try the other. I've also found conflicting information on the number of bytes. More on that below.

Supermicro's X9 series of motherboards may have no BMC chip, or any one of the following four (4) BMC chips:
  • Nuvoton WPCM450 (or Winbond WPCM450)
  • Nuvoton WPCM450R (or Winbond WPCM450R)
  • Renesas SH7757 (there may be other models, but I've only seen that one)
  • ASPEED AST2400
So, we're talking about an array of 4 different BMC chips on X9 boards made by 3 different manufacturers (4 manufacturers if you differentiate between before and after Winbond became Nuvoton). Winbond/Nuvoton are effectively the same company/product, just different names. Nuvoton was spun-off from Winbond in 2008.

Raw 0x32 0x91 0x00 0x7f
0x32 is the fan device
0x91 is cmd "set"
0x00 is zone
0x7f is 50% set in hex FF is 100%

I'll try my X9DRH-7F and other boards and your specific recommendations very soon.

X9DR3-F
I can use
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x01 and fan goes to full speed
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x00 and fan goes to a normal speed, about 2600rpm
when i send:
  • Set fan zone 1 to duty cycle 7F (50%): raw 0x32 0x91 0x01 0x7f
I get this error code
Unable to send raw command (channel=0x0 netfn=0x32 lun=0x0 cmd=0x91 rep=0xc1):
Let's take a quick look at your X9 board model numbers and their BMC chipsets.

Nuvoton WPCM450R

X9DR3-F

I can use
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x01 and fan goes to full speed
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x00 and fan goes to a normal speed, about 2600rpm
when i send:
  • Set fan zone 1 to duty cycle 7F (50%): raw 0x32 0x91 0x01 0x7f
I get this error code
Unable to send raw command (channel=0x0 netfn=0x32 lun=0x0 cmd=0x91 rep=0xc1):
That board has a Nuvoton WPCM450 BMC chip, so that makes sense after digging further into my notes on Supermicro voodoo-magic.

This should set that board to fan mode FULL:

raw 0x30 0x45 0x01 0x01

This should set it to STANDARD mode:

raw 0x30 0x45 0x01 0x00

This should return the currently set fan mode:

raw 0x30 0x45 0x00

Should return 0, 1, 2, or 4.

X9DRD-7fln4
Same thing happens fans can be set to high and high turned off but same error message.

It took some time for the fan speed to reduce. I think it is faster than before after turning off full speed. That was my quietest system. not any more. it slowed in steps with delay in between. perhaps it will slow down more
Did you mean X9DRD-7LN4F?

It has Nuvoton WPCM450 BMC.

Try using this command to set it to OPTIMAL fan mode:

raw 0x30 0x45 0x01 0x02

X9DRH-7F
swapped some network cards yesterday and it needs some work to get back on the network. I'll try tomorrow. Ironically it is the one with the issue
Nuvoton WPCM450R

So... they're all using a Nuvoton BMC chip. At least that makes things easier, as once you figure out the right commands, they will most likely work for all of them (any nuances between WPCM450 and WPCM450R notwithstanding).

Please try these other combos:

--> set manual fan control mode for all zones
raw 0x32 0x90 0x01

--> exit manual fan mode (set automatic fan mode) for all zones
raw 0x32 0x90 0x00

--> set a specific fan to duty cycle [ MANUAL FAN MODE MUST BE SET FIRST ]
raw 0x32 0x91 {fan_id} {duty_cycle}

--> get fan current duty cycle
raw 0x32 0x92 {fan_id}

IMPORTANT NOTES

--> fan_id starts at 0
--> fans are numbered (i.e. use 0x0 | 0x1 | 0x2 | etc. and do NOT use header names)
--> remember fan numbers are in hex
--> unclear if the duty_cycle range is 0 - 100 in hex (i.e. 0x00 - 0x64) or 0 - 100% as a percentage / 255 (i.e. range 0x00 - 0xFF where FF = 100% fan speed)
 

DaveInTexas

Member
Oct 28, 2021
77
85
18
One more related comment... most Supermicro boards do not support individual fan control, so I'm not certain those commands mentioned above will work. My suspicion is they probably work for one of the 4 possible BMCs on X9 boards, but I don't know which one(s) - if any.

As I mentioned in my last post, these commands are very esoteric to board manufacturer, BMC chipset, board model, etc.

Some motherboards do not even allow you to change fan speeds, even though their BMC might support it. In those cases, the BIOS often controls the fan speeds and in some cases will either prevent IPMI from changing it or override any IPMI commands on the next reboot.
 

DaveInTexas

Member
Oct 28, 2021
77
85
18
@ziggygt you could also try this X9 command variant I found to control fan zone:

Code:
ipmitool raw 0x30 0x91 0x5A 0x03 "0x1{zone}" "0x{duty_hex}"
where "zone" = 0 or 1

where "duty_hex" = 0x00 to 0xFF ( 0-100 converted to a scale of 0-FF or 0-255; i.e. 255 = 100% power)

I can confirm that command this works on an X9DR3-F board, because I own one and I've tested that command string on that board.
 

ziggygt

Member
Jul 23, 2019
62
10
8
Thanks for a great explanation of the way BMC is setup. I have several SM mother boards. Perhaps it is because my X9DR3-F board is one of that does not support Fan control but I tried the commands from a previous thread.

when i use the Truenas shell to type "ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x24" command I get an error message I get says:
"unable to send RAW command channel=0x00 netfn=0x30 lun=0x0 cmd=0x70 rsp=0xcc
invalid data field in request"

raw <netfn> <cmd> [data]

"Network Function Codes:
VAL HEX STRING
==============================================
0 0x00 Chassis
2 0x02 Bridge
4 0x04 SensorEvent
6 0x06 Application
8 0x08 Firmware
10 0x0a Storage
12 0x0c Transport"

You can see that my X9DR3-F motherboard does not have a 0x30 netfn?
Is there a decoder ring for these X9 boards. I did not find it on that server wiki link you shared. Where are the raw commands defined?

Raw 0x32 0x91 0x00 0x7f
0x32 is the fan device
0x91 is cmd "set"
0x00 is zone
0x7f is 50% set in hex FF is 100%

I'll try my X9DRH-7F and other boards and your specific recommendations very soon.

X9DR3-F
I can use
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x01 and fan goes to full speed
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x00 and fan goes to a normal speed, about 2600rpm
when i send:
  • Set fan zone 1 to duty cycle 7F (50%): raw 0x32 0x91 0x01 0x7f
I get this error code
Unable to send raw command (channel=0x0 netfn=0x32 lun=0x0 cmd=0x91 rep=0xc1):

X9DRD-7fln4
Same thing happens fans can be set to high and high turned off but same error message.

It took some time for the fan speed to reduce. I think it is faster than before after turning off full speed. That was my quietest system. not any more. it slowed in steps with delay in between. perhaps it will slow down more

X9DRH-7F ,
swapped some network cards yesterday and it needs some work to get back on the network. I'll try tomorrow. Ironically it is the one with the issue

Thanks,
Testing of X9DR3-F CPU=E5-2695 v2 showed that this is the right solution

This works
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x01 and fan goes to full speed.
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x00 and fan goes to a normal speed, about 21005rpm
  • COMMANDS to set speeds other levels
    • ipmitool raw 0x30 0x91 0x5A 0x3 0x10 0x7F for CPU Zone 0x7F is about 50% 0x20 is about 2000RPM and no setting will move it lower than that
    • ipmitool raw 0x30 0x91 0x5A 0x3 0x11 0x7F for I/O Zone
  • These settings do not survive a reboot. This can be scripted and run on TRUENAS boot?
Testing of X9DRH-7F CPU = E5-2650L v2
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x01 and fan goes to full speed.
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x00 and fan goes to a normal speed, about 3600rpm
  • ipmitool raw 0x30 0x91 0x5A 0x3 0x11 0x7F for I/O Zone does not work. no error but no effect
  • this is CHASSIS loud and not needed, cpus are cool?
  • Seems like this shOuld be the quiet with thelow power CPU
Testing of X9DRD-7fln4 CPU= E5-2620
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x01 and fan goes to full speed.
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x00 and fan goes to a normal speed, about 3000rpm FOR CHASSIS, 1900rpm cpu COOLERS
  • ipmitool raw 0x30 0x91 0x5A 0x3 0x11 0x7F for I/O Zone does not work. error message generated. rsp=0x82, 0x82 unknown
tech support at SM must be a nightmare when similar system perform so differently
 
Last edited:
  • Like
Reactions: DaveInTexas

ziggygt

Member
Jul 23, 2019
62
10
8
Seems my X9DRD doesn't have any way of manually lowering the fan speed. I poked around with it and actually got the fans to run at <1k RPM.... except it was a lie, I somehow got it to report a lower speed without actually changing the fan speeds.

What I'm thinking of is making a simple circuit that just sits on the PWM line and biases it a bit, so that the pulses are effectively shorter. x9DRD-7LN4F
I have a x9DRD-7LN4F that idles at 3500RPM and I cannot get the IPMI commands to alter that. It runs very cool but too loud. I am looking at this thermal controller and a fan header to control the three chassis fans. The SM chassis fans are 0.6A each and this card can handle 3A from a drive power cable. Control works well but challenge is where to mount the thermal sensor to get the best idea of the CPU temp. I plan to put the system under extreme load while watching thermal temps using IPMI. By experimenting with sensor placement, CPU1 usually gets all the load first. I should be able to set the temps safe for CPU and or drives but quiet for my peace of mind. Attached is a photo of the bench test. The board has and LED readout of temp and speed in RPM. In the attached pdf you can see how to set it up.
On this board I do not think is controllable in another way
 

Attachments

DaveInTexas

Member
Oct 28, 2021
77
85
18
Thanks,
Testing of X9DR3-F CPU=E5-2695 v2 showed that this is the right solution

This works
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x01 and fan goes to full speed.
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x00 and fan goes to a normal speed, about 21005rpm
  • COMMANDS to set speeds other levels
    • ipmitool raw 0x30 0x91 0x5A 0x3 0x10 0x7F for CPU Zone 0x7F is about 50% 0x20 is about 2000RPM and no setting will move it lower than that
    • ipmitool raw 0x30 0x91 0x5A 0x3 0x11 0x7F for I/O Zone
  • These settings do not survive a reboot. This can be scripted and run on TRUENAS boot?
Right. That is normal. You need a start-up script, such as a systemd service daemon. Solution options depend on your operating system and technical prowess.

Testing of X9DRH-7F CPU = E5-2650L v2
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x01 and fan goes to full speed.
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x00 and fan goes to a normal speed, about 3600rpm
  • ipmitool raw 0x30 0x91 0x5A 0x3 0x11 0x7F for I/O Zone does not work. no error but no effect
  • this is CHASSIS loud and not needed, cpus are cool?
Check your CPU temps while the fans are at that speed. If the CPU gets too hot, system will override your settings and push fans to max.

Testing of X9DRD-7fln4 CPU= E5-2620
Is your board model actually X9DRD-7LN4F ?

  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x01 and fan goes to full speed.
  • Set IPMI fan mode to FULL: raw 0x30 0x45 0x01 0x00 and fan goes to a normal speed, about 3000rpm FOR CHASSIS, 1900rpm cpu COOLERS
That should be Standard Speed. Try this to see if it sets the fans to Optimal speed mode:

raw 0x30 0x45 0x01 0x02

or

raw 0x30 0x45 0x01 0x04

  • ipmitool raw 0x30 0x91 0x5A 0x3 0x11 0x7F for I/O Zone does not work. error message generated. rsp=0x82, 0x82 unknown
You're telling it to set the fan speeds for Zone 1 (I/O), but that board only has one fan zone (CPU/System or Zone 0). Try this:

ipmitool raw 0x30 0x91 0x5A 0x3 0x10 0x7F

I have a x9DRD-7LN4F ....
On this board I do not think is controllable in another way
Possibly, but try commands above before giving up.

Other thoughts:
  1. FYI, not all BMC's support the 4 fan modes mentioned previously. Some only support 3: Standard | Full | Optimal. They could take set values of 0 | 1 | 2 or 0 | 1 | 4 depending on the BMC implementation.
  2. Nuvoton/Winbond BMC chip with -R designation may function differently or may not allow manual fan manipulation. Your X9DRH-7F board has this chip.
 

ziggygt

Member
Jul 23, 2019
62
10
8
Right. That is normal. You need a start-up script, such as a systemd service daemon. Solution options depend on your operating system and technical prowess.
as expected just checking. did not want to mess up my quiet system

Check your CPU temps while the fans are at that speed. If the CPU gets too hot, system will override your settings and push fans to max.

Is your board model actually X9DRD-7LN4F ?
Yes, just my fat fingers.
That should be Standard Speed. Try this to see if it sets the fans to Optimal speed mode:

raw 0x30 0x45 0x01 0x02

or

raw 0x30 0x45 0x01 0x04

You're telling it to set the fan speeds for Zone 1 (I/O), but that board only has one fan zone (CPU/System or Zone 0). Try this:

ipmitool raw 0x30 0x91 0x5A 0x3 0x10 0x7F
ipmitool raw 0x30 0x91 0x5A 0x3 0x10 0x7F
ipmitool raw 0x30 0x91 0x5A 0x3 0x10 0x20
Those commands have no effect
Possibly, but try commands above before giving up.

Other thoughts:
  1. FYI, not all BMC's support the 4 fan modes mentioned previously. Some only support 3: Standard | Full | Optimal. They could take set values of 0 | 1 | 2 or 0 | 1 | 4 depending on the BMC implementation.
  2. Nuvoton/Winbond BMC chip with -R designation may function differently or may not allow manual fan manipulation. Your X9DRH-7F board has this chip.
Yes it is a X9DRD-7LN4F sorry for the fat fingers. testing raw 0x30 0x45 0x01 0x<yy>
there seem to be 4 modes
yy - CPU - I/O
00 - 3900 / 3750 RPM - Standard?
01 - 7000 / 6600 RPM - Full On?
02 - 3875 / 1875 RPM - Optimal?
03 - Error
04 - 4100 / 6500 RPM - Heavy I/O

CPUs are at 31 and 33 degrees C with fans at 1875 RPM
Choicss:
1) Slave all the fans off the I/O section? They will run at 1875. Plenty to keep the mostly idle CPU cards cool. Will the BMC control the I/O fans if the CPU gets too hot? seems like all fans go into panic mode.so seems like it will work

2) Use the Hardware thermal controller that I posted as a solution. https://forums.servethehome.com/ind...9-x10-x11-fan-speed-control.10059/post-369474

Since I will have 3x 2port Mellanox 10G ethernet cards in this chassis I might use two hardware thermal controllers and put a sensor near the Mellanox cards.

3) Wear ear muffs
 
Last edited:

DaveInTexas

Member
Oct 28, 2021
77
85
18
ipmitool raw 0x30 0x91 0x5A 0x3 0x10 0x7F
ipmitool raw 0x30 0x91 0x5A 0x3 0x10 0x20
Those commands have no effect

Yes it is a X9DRD-7LN4F sorry for the fat fingers. testing raw 0x30 0x45 0x01 0x<yy>
Poking around a bit in the manual for that board... it looks like it could be limited to IPMI 1.5, though there might also be a BIOS level option to change that. If it doesn't support IPMI 2.0, that fact might correlate to those raw commands failing to work, but I don't know for sure if there is a correlation. It certainly seems that a layer between your operating system and the Nuvoton BMC chip is limiting the BMC's capabilities.

Will the BMC control the I/O fans if the CPU gets too hot? seems like all fans go into panic mode.so seems like it will work
Yes. Correct.
 

dogfight

New Member
Feb 26, 2023
10
0
1
I found that on my X10DRL, the minimum fan speed is 50% only. Is there anyway to lower it even more? On consummer mainboards, I can normally lower fan speed to even 0%.

Thank you!
 

DaveInTexas

Member
Oct 28, 2021
77
85
18
I found that on my X10DRL, the minimum fan speed is 50% only. Is there anyway to lower it even more? On consummer mainboards, I can normally lower fan speed to even 0%.
Depends on several factors:
  • Minimum / Maximum physical fan speed range
  • Lower fan speed threshold settings
  • Whether or not your mobo allows you to set fan speeds manually
Presuming your board allows manual fan speed control, the gist of it is to first lower your Lower fan speed thresholds the BMC is monitoring to a level where you will not trigger a lower threshold alert, which will cause your BMC to freak out and ramp all fans to 100% because it believes your CPU is in imminent danger of melting down, and/or one or more fans have failed. Then set your fan speeds manually.

You have to kind of think about this from the viewpoint of a system architect, who works for a large PC box manufacturer. Would you want your customers' PCs to start baking themselves ever? No. So, program them from the factory so that can't happen.

The trick is to reduce your Lower CRitical (LCR) and Lower Non-Recoverable (LNR) fan speed alerts stored in the BMC to a level where your fan speeds will not drop to or below those levels, ever. After that, you can set your fan speed to 0% or 1% (whichever is the lowest duty cycle your board's BMC supports; for most it's zero).

The final wild card is your fans' physical limits of low (stall) and high (max) speed capabilities.
 
  • Like
Reactions: dogfight

dogfight

New Member
Feb 26, 2023
10
0
1
Hi Dave,

I understand that. But I dont know how. Setting using ipmitool can only reduce the fan to 50% minimum.

My fans have max 1700 rpm, it can slow down to ~700 minimum (on consumber mainboard), but when putting into this x10drl board, at idle, it run at 900 rpm. The CPUs idle at 25*C and 30*C, but the fan still 900 rpm and do not want to go lower speed.


Depends on several factors:
  • Minimum / Maximum physical fan speed range
  • Lower fan speed threshold settings
  • Whether or not your mobo allows you to set fan speeds manually
Presuming your board allows manual fan speed control, the gist of it is to first lower your Lower fan speed thresholds the BMC is monitoring to a level where you will not trigger a lower threshold alert, which will cause your BMC to freak out and ramp all fans to 100% because it believes your CPU is in imminent danger of melting down, and/or one or more fans have failed. Then set your fan speeds manually.

You have to kind of think about this from the viewpoint of a system architect, who works for a large PC box manufacturer. Would you want your customers' PCs to start baking themselves ever? No. So, program them from the factory so that can't happen.

The trick is to reduce your Lower CRitical (LCR) and Lower Non-Recoverable (LNR) fan speed alerts stored in the BMC to a level where your fan speeds will not drop to or below those levels, ever. After that, you can set your fan speed to 0% or 1% (whichever is the lowest duty cycle your board's BMC supports; for most it's zero).

The final wild card is your fans' physical limits of low (stall) and high (max) speed capabilities.
 

DaveInTexas

Member
Oct 28, 2021
77
85
18
Hi Dave,

I understand that. But I dont know how. Setting using ipmitool can only reduce the fan to 50% minimum.

My fans have max 1700 rpm, it can slow down to ~700 minimum (on consumber mainboard), but when putting into this x10drl board, at idle, it run at 900 rpm. The CPUs idle at 25*C and 30*C, but the fan still 900 rpm and do not want to go lower speed.
What's the exact model of your server board? Is it X10DRL (no suffix), or X10DRL-i (for example), etc.?

What is the raw IMPItool string you're attempting that does not seem to be working as desired?
 
  • Like
Reactions: dogfight

dogfight

New Member
Feb 26, 2023
10
0
1
The X10drl-i. the command line is:

ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x16.

0x16 means 25%

In fact it will set to 50%. I found that to be able to set to lower, I have to set:

ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x09.

The fan will run at lower than 50%. But the issue is if I use the PC, it will go back to 50% after that. So now I want to change the fan profile behavior. For example, the OPTIMAL is

"The Optimal Speed profile is also temperature controlled, but with a base fan speed at 30% PWM duty cycle, broadening the span to 30-100% to allow the system to save energy while idle."

30% means 0x1D, higher than 0x16.

Is there anyway to change the base fan speed of the Optimal mode?





What's the exact model of your server board? Is it X10DRL (no suffix), or X10DRL-i (for example), etc.?

What is the raw IMPItool string you're attempting that does not seem to be working as desired?
 

DaveInTexas

Member
Oct 28, 2021
77
85
18
The X10drl-i. the command line is:

ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x16.

0x16 means 25%

In fact it will set to 50%. I found that to be able to set to lower, I have to set:

ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x09.

The fan will run at lower than 50%. But the issue is if I use the PC, it will go back to 50% after that. So now I want to change the fan profile behavior.
You 'raw' syntax is correct for the X10 series. Your speed setting of 0x09 versus 0x16 and whether or not you notice a difference is likely due to your specific fans and how they're handling those voltage levels, versus something weird with IPMI.

Your description of the board's behavior makes it clear something is controlling the fan speeds such that it is overriding your IPMI induced setting. It does not sound like a lower fan speed threshold issue, as if that were the case your fans would ramp to 100% and then ramp down again; you'd get the 'yo-yo' effect of fan speed.

Are you certain the fan mode was placed into "FULL" mode via IPMI prior to issuing the IPMI fan speed command? If not, that would likely be the problem.

The next most likely culprit would be the BIOS. This also matches your behavior description. It sounds like the BIOS is overriding your IPMI settings based on thermals.

I would suggest first ensuring you've set the BMC fan controller to "FULL" mode before attempting to force the fan speed to whatever. If that is not a factor, try booting into the BIOS and looking for options that could indicate BIOS level control of the fans based on CPU temperature. Also check the manual.

Is there anyway to change the base fan speed of the Optimal mode?
No.
 
  • Like
Reactions: dogfight

dogfight

New Member
Feb 26, 2023
10
0
1
What I want is making the fan idle at low speed, but will speed up if needed. So set to FULL then set fan speed does not make sense.

I dont really understand the SuperMicro threshold, anyone pls explain it to me? I set the threshold but it does not work as expected,

The FAN1 is set to:
OK | (1277) FAN1 | 1000 RPM | 100 RPM | 1500 RPM |

low threshold is 100, high is 1500

I also set the Lower Non-Recoverable, Lower Critical, Lower Non-Critical to very low (100, 200, 300), as expect that the idle speed will goes down, but it still idles at 1000 RPM.

(The fans can slow down to even 700 rpm)

If it set the fan to 700, ok, it will idle at 700, but any demand on CPU will speed up the fans, then later it will idle at 1000.

Any advice? Thank you!


You 'raw' syntax is correct for the X10 series. Your speed setting of 0x09 versus 0x16 and whether or not you notice a difference is likely due to your specific fans and how they're handling those voltage levels, versus something weird with IPMI.

Your description of the board's behavior makes it clear something is controlling the fan speeds such that it is overriding your IPMI induced setting. It does not sound like a lower fan speed threshold issue, as if that were the case your fans would ramp to 100% and then ramp down again; you'd get the 'yo-yo' effect of fan speed.

Are you certain the fan mode was placed into "FULL" mode via IPMI prior to issuing the IPMI fan speed command? If not, that would likely be the problem.

The next most likely culprit would be the BIOS. This also matches your behavior description. It sounds like the BIOS is overriding your IPMI settings based on thermals.

I would suggest first ensuring you've set the BMC fan controller to "FULL" mode before attempting to force the fan speed to whatever. If that is not a factor, try booting into the BIOS and looking for options that could indicate BIOS level control of the fans based on CPU temperature. Also check the manual.



No.
 

DaveInTexas

Member
Oct 28, 2021
77
85
18
What I want is making the fan idle at low speed, but will speed up if needed. So set to FULL then set fan speed does not make sense.
Then you basically have two viable options:
  1. Set mobo fan mode to OPTIMAL. Fire and forget. BMC controls when it ramps up. Minimum speed 30% on Supermicro boards.
  2. Set mobo fan mode to FULL. Implement a software-based fan controller. These can be run manually or setup to run in the background if you have the knowledge to do so (or learn to). For example, a service daemon on a Linux box. Windows is easier (tag the program in start-up programs list the O.S. processes at start-up).

I dont really understand the SuperMicro threshold, anyone pls explain it to me? I set the threshold but it does not work as expected,

The FAN1 is set to:
OK | (1277) FAN1 | 1000 RPM | 100 RPM | 1500 RPM |

low threshold is 100, high is 1500

I also set the Lower Non-Recoverable, Lower Critical, Lower Non-Critical to very low (100, 200, 300), as expect that the idle speed will goes down, but it still idles at 1000 RPM.

(The fans can slow down to even 700 rpm)

If it set the fan to 700, ok, it will idle at 700, but any demand on CPU will speed up the fans, then later it will idle at 1000.
This is actually a rather complex topic, but I'm going to do my best to provide you with a relatively straight-forward answer.

Lower Threshold Terms
First thing is to understand what these terms mean, from the perspective of the BMC controller (the chipset that monitors and controls your fan speeds).

LNR = Lower Non-Recoverable : fan/fan header has failed and cannot be recovered
LCR = Lower CRitical : fan speed is critically low, and action must be taken to attempt fan speed recovery
LNC = Lower Non-CRitical : fan speed is dangerously low; low enough to alert, but not low enough to take action

BMC Behavior
How does your BMC behave when a fan speed reaches or drops below these lower threshold settings?

LNR = full power to ALL fan headers, ignore fan settings
LCR = full power to ALL fan headers, ignore fan settings
LNC =internal alert within BMC, no action

Lower Threshold Ordering
Now, let's talk about order. It's best to think about them in this order for Lower thresholds. This is how you want to think about them when deciding what to set them to.

LNC = Lower Non-CRitical
LCR = Lower CRitical
LNR = Lower Non-Recoverable

In other words, Lower Non-Critical (LNC) should always be equal to or greater than Lower CRitical (LCR), which should always be equal to or greater than Lower Non-Recoverable (LNR). The LNR should always be your lowest value.

How do you know what to set the Lower thresholds to?
The answer is: it depends :p on your desired behavior of the system if the given fan speed is ever low enough to breach one of these thresholds.

You cannot change the default behavior of the server motherboard's BMC when it believes a fan is failing. The server is going to jack up all your fans to maximum speed no matter what else is going on. Therefore, you need to think about when would you want that to happen, or not happen? And how much can you control that outcome?

Many people want to be able to ramp their fan speeds down as low as they can go without stalling (i.e. as close to their minimum speed as possible). One of the tricks in accomplishing this is understanding there are two factors at work here:
  1. The instruction a program sends to your fans to get them to spin as slowly and quietly as possible; and
  2. The reading of fan speed by the BMC, which is trying to protect your system from overheating.
In order to achieve your goal ( # 1 ) without inadvertently causing condition # 2 to occur, you need to understand how your BMC monitors and interprets your fans' speeds, and then apply that understanding to setting your Lower Thresholds.

Fan Specs Matter
First, you must understand the physical characteristics of your fans sufficiently:
  • What is the minimum viable speed of each fan before it will stall?
    • Your minimum fan speed is just above this level.
    • Some fan manufacturers will inform you of the minimum speed, and some will not.
  • What is the fan hysteresis of the BMC fan controller?
  • What is the smallest multiple of fan hysteresis, greater than the minimum viable fan speed for the given fan?
Understanding Fan Hysteresis is Crucial
Let's talk a bit about fan hysteresis. You may be wondering, "What is fan hysteresis?" I almost never see anyone discuss this, but it's actually very important to understand what it is and how it affects the behavior of your fans when setting fan speed thresholds.

Fan hysteresis is effectively the number of RPMs the BMC rounds a reported fan speed up or down to. When the BMC is reading fan speeds, those speeds are fluctuating slightly many times a second. There is an algorithm in every BMC chip that smooths out that value and approximates it, to the nearest fan hysteresis multiple.

For example, let's say you have a fan your board's BMC is reading 1/60 of a second, and every reading in a given polling period (e.g. 1 second) is in the range of 505 - 523 RPMs. If your BMC's fan hysteresis is 100, then when you query your BMC chip via IPMI, it will report the fan speed as 500 RPM. This is because 500 is the closest multiple of 100 (fan hysteresis) to the actual range of fan speed over the BMC's polling period (1 second).

The rounding method the BMC uses varies depending on the BMC chip and its algorithm. Generally speaking, they take the average RPM over the polling period and then round it to the nearest fan hysteresis multiple. Most if not all BMC chips will round down (<=50% of range between 2 fan hysteresis values). This translates to - in the example above - if the polled fan speed range is 451 - 550 RPM, it will be reported in IPMI as 500 RPM. Common fan hysteresis values are 75 and 100.

What If I Don't Know the Fan Hysteresis?
How do you determine what the fan hysteresis is? There are several ways to do so, including a BMC query. I won't get into the more complex methods though (e.g. positive versus negative hysteresis). The simplest method is to observe the factory settings of either the Lower or Upper thresholds. Their default state should be a 1x fan hysteresis separation between each threshold (e.g. 100 | 200 | 300 or 150 | 225 | 300).

Putting It All Together
Once you have this information - the fan's speed limit range and the BMC's fan hysteresis - it's time to calculate the lowest viable fan speed, which is your starting point to calculate the Lower thresholds you will want to set your fan to. My recommendation is to then take the smallest viable multiple of fan hysteresis the fan can spin at without stalling, and +1 fan hysteresis value to that figure. Why?

Fan speeds fluctuate. This is normal behavior. The reported fan speed at any given moment is influenced by various factors. The exact reasons aren't important here. What is important is the fact the reported fan speed is going to fluctuate +/- 1x fan hysteresis around its mean value during the course of normal operation. Thus, if you set your lowest fan threshold too close to the estimated real lowest speed the fan is capable of operating at, you will be likely to trip that threshold at some point. Adding an extra fan hysteresis multiple to the low end of your threshold range will help to prevent inadvertent and unwanted episodes of your BMC initiating its fan panic mode (i.e. 100% power to all fans).

Once you have that lowest possible, viable RPM value, this becomes your base for the upper end of your Lower threshold values. Starting from that figure, work your way down in increments of 1x fan hysteresis for each subsequently lower of your Lower thresholds.

For example:
  • Fan minimum rated speed: 700 RPM
  • BMC fan hysteresis: 75 RPM
  • Set LNC (the highest of your lower values) to: 550 RPM ( 700 - ( 2 x fan hysteresis of 75 ) )
  • Set LCR to: 475 RPM ( 550 - 75 )
  • Set LNR to: 400 RPM ( 475 - 75)
A few tips:
  • LNC >= LCR >= LNR
  • Remember, when the LCR value is reached, your system will go into panic mode and force all fans to 100% power, until the LCR condition is no longer true
  • Minimum lower threshold value = fan hysteresis (i.e. you cannot set any of them to zero)
 
Last edited:
  • Like
Reactions: dogfight

dogfight

New Member
Feb 26, 2023
10
0
1
I tried to determine the hysteresis of the fans and it is 100. I follow your guide and it still does not resolve the issue. The fan does not slow down to what I want

And now one more issue arises: sometime, when the server in max load, the fan jump to max, and never come back. Even when I shut down the PC and reboot it, the fans still keep its max speed. I have to pull the electric plug.


Then you basically have two viable options:
  1. Set mobo fan mode to OPTIMAL. Fire and forget. BMC controls when it ramps up. Minimum speed 30% on Supermicro boards.
  2. Set mobo fan mode to FULL. Implement a software-based fan controller. These can be run manually or setup to run in the background if you have the knowledge to do so (or learn to). For example, a service daemon on a Linux box. Windows is easier (tag the program in start-up programs list the O.S. processes at start-up).



This is actually a rather complex topic, but I'm going to do my best to provide you with a relatively straight-forward answer.

Lower Threshold Terms
First thing is to understand what these terms mean, from the perspective of the BMC controller (the chipset that monitors and controls your fan speeds).

LNR = Lower Non-Recoverable : fan/fan header has failed and cannot be recovered
LCR = Lower CRitical : fan speed is critically low, and action must be taken to attempt fan speed recovery
LNC = Lower Non-CRitical : fan speed is dangerously low; low enough to alert, but not low enough to take action

BMC Behavior
How does your BMC behave when a fan speed reaches or drops below these lower threshold settings?

LNR = full power to ALL fan headers, ignore fan settings
LCR = full power to ALL fan headers, ignore fan settings
LNC =internal alert within BMC, no action

Lower Threshold Ordering
Now, let's talk about order. It's best to think about them in this order for Lower thresholds. This is how you want to think about them when deciding what to set them to.

LNC = Lower Non-CRitical
LCR = Lower CRitical
LNR = Lower Non-Recoverable

In other words, Lower Non-Critical (LNC) should always be equal to or greater than Lower CRitical (LCR), which should always be equal to or greater than Lower Non-Recoverable (LNR). The LNR should always be your lowest value.

How do you know what to set the Lower thresholds to?
The answer is: it depends :p on your desired behavior of the system if the given fan speed is ever low enough to breach one of these thresholds.

You cannot change the default behavior of the server motherboard's BMC when it believes a fan is failing. The server is going to jack up all your fans to maximum speed no matter what else is going on. Therefore, you need to think about when would you want that to happen, or not happen? And how much can you control that outcome?

Many people want to be able to ramp their fan speeds down as low as they can go without stalling (i.e. as close to their minimum speed as possible). One of the tricks in accomplishing this is understanding there are two factors at work here:
  1. The instruction a program sends to your fans to get them to spin as slowly and quietly as possible; and
  2. The reading of fan speed by the BMC, which is trying to protect your system from overheating.
In order to achieve your goal ( # 1 ) without inadvertently causing condition # 2 to occur, you need to understand how your BMC monitors and interprets your fans' speeds, and then apply that understanding to setting your Lower Thresholds.

Fan Specs Matter
First, you must understand the physical characteristics of your fans sufficiently:
  • What is the minimum viable speed of each fan before it will stall?
    • Your minimum fan speed is just above this level.
    • Some fan manufacturers will inform you of the minimum speed, and some will not.
  • What is the fan hysteresis of the BMC fan controller?
  • What is the smallest multiple of fan hysteresis, greater than the minimum viable fan speed for the given fan?
Understanding Fan Hysteresis is Crucial
Let's talk a bit about fan hysteresis. You may be wondering, "What is fan hysteresis?" I almost never see anyone discuss this, but it's actually very important to understand what it is and how it affects the behavior of your fans when setting fan speed thresholds.

Fan hysteresis is effectively the number of RPMs the BMC rounds a reported fan speed up or down to. When the BMC is reading fan speeds, those speeds are fluctuating slightly many times a second. There is an algorithm in every BMC chip that smooths out that value and approximates it, to the nearest fan hysteresis multiple.

For example, let's say you have a fan your board's BMC is reading 1/60 of a second, and every reading in a given polling period (e.g. 1 second) is in the range of 505 - 523 RPMs. If your BMC's fan hysteresis is 100, then when you query your BMC chip via IPMI, it will report the fan speed as 500 RPM. This is because 500 is the closest multiple of 100 (fan hysteresis) to the actual range of fan speed over the BMC's polling period (1 second).

The rounding method the BMC uses varies depending on the BMC chip and its algorithm. Generally speaking, they take the average RPM over the polling period and then round it to the nearest fan hysteresis multiple. Most if not all BMC chips will round down (<=50% of range between 2 fan hysteresis values). This translates to - in the example above - if the polled fan speed range is 451 - 550 RPM, it will be reported in IPMI as 500 RPM. Common fan hysteresis values are 75 and 100.

What If I Don't Know the Fan Hysteresis?
How do you determine what the fan hysteresis is? There are several ways to do so, including a BMC query. I won't get into the more complex methods though (e.g. positive versus negative hysteresis). The simplest method is to observe the factory settings of either the Lower or Upper thresholds. Their default state should be a 1x fan hysteresis separation between each threshold (e.g. 100 | 200 | 300 or 150 | 225 | 300).

Putting It All Together
Once you have this information - the fan's speed limit range and the BMC's fan hysteresis - it's time to calculate the lowest viable fan speed, which is your starting point to calculate the Lower thresholds you will want to set your fan to. My recommendation is to then take the smallest viable multiple of fan hysteresis the fan can spin at without stalling, and +1 fan hysteresis value to that figure. Why?

Fan speeds fluctuate. This is normal behavior. The reported fan speed at any given moment is influenced by various factors. The exact reasons aren't important here. What is important is the fact the reported fan speed is going to fluctuate +/- 1x fan hysteresis around its mean value during the course of normal operation. Thus, if you set your lowest fan threshold too close to the estimated real lowest speed the fan is capable of operating at, you will be likely to trip that threshold at some point. Adding an extra fan hysteresis multiple to the low end of your threshold range will help to prevent inadvertent and unwanted episodes of your BMC initiating its fan panic mode (i.e. 100% power to all fans).

Once you have that lowest possible, viable RPM value, this becomes your base for the upper end of your Lower threshold values. Starting from that figure, work your way down in increments of 1x fan hysteresis for each subsequently lower of your Lower thresholds.

For example:
  • Fan minimum rated speed: 700 RPM
  • BMC fan hysteresis: 75 RPM
  • Set LNC (the highest of your lower values) to: 550 RPM ( 700 - ( 2 x fan hysteresis of 75 ) )
  • Set LCR to: 475 RPM ( 550 - 75 )
  • Set LNR to: 400 RPM ( 475 - 75)
A few tips:
  • LNC >= LCR >= LNR
  • Remember, when the LCR value is reached, your system will go into panic mode and force all fans to 100% power, until the LCR condition is no longer true
  • Minimum lower threshold value = fan hysteresis (i.e. you cannot set any of them to zero)
 

DaveInTexas

Member
Oct 28, 2021
77
85
18
This sounds like expected behavior. You must understand the primary directive as far as your BIOS is concerned is to prevent your CPU from melting.

If you are using an automated fan speed controller, it's not keeping your CPU cool enough, so the BIOS is taking over to prevent a nuclear meltdown and hosing your CPU.

I tried to determine the hysteresis of the fans and it is 100. I follow your guide and it still does not resolve the issue. The fan does not slow down to what I want

And now one more issue arises: sometime, when the server in max load, the fan jump to max, and never come back. Even when I shut down the PC and reboot it, the fans still keep its max speed. I have to pull the electric plug.
There are only a few possibilities. First, there's the question of whether you can actually control the fan speed via IPMI or if it can only be controlled via the BIOS. That depends on the motherboard.

Presuming you can control it via IPMI, several factors will come into play.
  1. "sometime, when the server in max load, the fan jump to max, and never come back" This is because your fans are going into "panic" mode for one or the other of two reasons (or both reasons, possibly):
    • If you have a program controlling fan speed, the fan speed is spinning up and above upper BMC fan critical threshold, or below lower BMC fan critical threshold. This seems unlikely in the scenario you're describing.
    • Temperature sensors on the motherboard determine your CPU is in danger of overheating, causing the BIOS to force fans to full speed.
  2. "Even when I shut down the PC and reboot it, the fans still keep its max speed." This is because of one or more of the following reasons:
    • On reboot, the PC will default to the fan mode it was in before reboot. If you're running a custom fan management program, on many motherboards (e.g. Supermicro), this requires your fan mode to be set to FULL speed. Thus, when you reboot the server it will default to running the fans at full speed. You must have a method for the fan controller to restart and take over again after reboot or the fans will remain in full speed mode. The easiest method to accomplish an automatic fan controller takeover after reboot is with a system daemon program that is launched at PC start-up. You will still have a period of time when the fans run at full speed, before that automated script is started.
    • If your fan mode was set to something other than FULL, and then your CPU temperature monitor on the motherboard caused the BIOS to force your fan mode to FULL. Then, on reboot you're in FULL mode. For example, if you decided to set the system fan mode to OPTIMAL, then your BIOS panics due to the CPU temp, forces the fan mode to FULL, then you reboot and you're in FULL fan mode because that is what mode it was in before you rebooted the system.
 

dogfight

New Member
Feb 26, 2023
10
0
1
In my case the temps are quite okay: at full CPU and memory load: CPU1 50*C, CPU2 53*C, all other readings are under 60*C, well under upper thresholds. I dont know why the fan jump to max at the moment. It never behaves like that before.

When the fan jumps to max (or even more than max i believe), reboot does not help, only disconnect power from it will help. But previously, when the load reduces, fans will slow down.

The fans are quite slow at a normal restart.

may be a bug like this: https://forums.servethehome.com/index.php?threads/supermicro-fan-speed-wont-slow.5179/


This sounds like expected behavior. You must understand the primary directive as far as your BIOS is concerned is to prevent your CPU from melting.

If you are using an automated fan speed controller, it's not keeping your CPU cool enough, so the BIOS is taking over to prevent a nuclear meltdown and hosing your CPU.



There are only a few possibilities. First, there's the question of whether you can actually control the fan speed via IPMI or if it can only be controlled via the BIOS. That depends on the motherboard.

Presuming you can control it via IPMI, several factors will come into play.
  1. "sometime, when the server in max load, the fan jump to max, and never come back" This is because your fans are going into "panic" mode for one or the other of two reasons (or both reasons, possibly):
    • If you have a program controlling fan speed, the fan speed is spinning up and above upper BMC fan critical threshold, or below lower BMC fan critical threshold. This seems unlikely in the scenario you're describing.
    • Temperature sensors on the motherboard determine your CPU is in danger of overheating, causing the BIOS to force fans to full speed.
  2. "Even when I shut down the PC and reboot it, the fans still keep its max speed." This is because of one or more of the following reasons:
    • On reboot, the PC will default to the fan mode it was in before reboot. If you're running a custom fan management program, on many motherboards (e.g. Supermicro), this requires your fan mode to be set to FULL speed. Thus, when you reboot the server it will default to running the fans at full speed. You must have a method for the fan controller to restart and take over again after reboot or the fans will remain in full speed mode. The easiest method to accomplish an automatic fan controller takeover after reboot is with a system daemon program that is launched at PC start-up. You will still have a period of time when the fans run at full speed, before that automated script is started.
    • If your fan mode was set to something other than FULL, and then your CPU temperature monitor on the motherboard caused the BIOS to force your fan mode to FULL. Then, on reboot you're in FULL mode. For example, if you decided to set the system fan mode to OPTIMAL, then your BIOS panics due to the CPU temp, forces the fan mode to FULL, then you reboot and you're in FULL fan mode because that is what mode it was in before you rebooted the system.
 
Last edited:

DaveInTexas

Member
Oct 28, 2021
77
85
18
In my case the temps are quite okay: at full CPU and memory load: CPU1 50*C, CPU2 53*C, all other readings are under 60*C, well under upper thresholds. I dont know why the fan jump to max at the moment. It never behaves like that before.

When the fan jumps to max (or even more than max i believe), reboot does not help, only disconnect power from it will help. But previously, when the load reduces, fans will slow down.

The fans are quite slow at a normal restart.

may be a bug like this: https://forums.servethehome.com/index.php?threads/supermicro-fan-speed-wont-slow.5179/
Yep. Agreed. Next most likely possibility in trouble-shooting this problem you have is to start looking at potential hardware issues. Could be a bad fan or a bad fan header (motherboard issue).