Server Ignores user set fan mode in IPMI

[fatherboard]

Motherboard: Supermicro H13SSL-N

GPU: MSI Nvidia RTX 5090 Liquid Supreme. → not visible in IPMI

GPU driver nvidia-driver-580-open, in Ubuntu 24.04.3.

Updated BMC to latest.

Problem 1:
The server switches by itself to optimal (loud and noisy) fan mode, although I have set it to standard. It starts standard, and at some point in time it decides to switch to optimal while the IPMI still shows the standard mode is chosen.
Yesterday it decided by itself to go back to standard.
I have to say, the server is not doing absolutely nothing, and is forcing the optimal fan mode.
Why is this behaviour happening? and is it related to problem 2?

Problem 2:
The fans of the 5090 GPU only spin when the server starts, after that they stop, which also mean the pump of the GPU is not working. And they spin again when I turn off the server, but never when the server is running.
How do I get the GPU to spin the fans and the pump to work? how do I know if they only spin when it reaches a certain heat degree? How do I know if what is happening now is normal or the GPU is ignored and isolated and not working?

problem 1 ←> problem 2:
I've checked if these are related and saw that the GPU is not hot at all, actually it felt cold, while the server is forcing optimal fan mode and ignoring my standard fan mode.

Do you have any idea what is going on and how to solve this?

 

[i386]

Optimal is the setting where the fans default to run at 30% pwm (standard is 50%, heavy io splits cpu & system fans pwm, full speed = 100% pwm), the lowest noise setting. The bmc runs fans faster if the cpu gets hotter...

The 5090 is a consumer gpu, nvidia and amd introduced a while ago a feature where the fans stop if there is no or low load on the gpu called "zero fan mode"
The 5090 is probably not made by supermicro and doesn't integrate into the bmc controlled system. You could try to install the supermicro thin-agent services, but I doubt they will support a consumer gpu

 

[Alterra]

Open the Sensor tab and see if some component is very hot? Maybe you put M.2 SSD right under that 5090 and it gets hot because the 5090 heat sink warms everything around it?

It looks like fan mode is Standard, just like you have set. FAN2 is running at 3780 RPM which certainly is very audible. Depending on what kind of fan that is you can figure out what the PWM setting is. If it is running near full speed, this is probably because there is some hot component.

 

[fatherboard]

Yes, there is an M.2 NVMe stuck tight under the GPU. But when I touched the GPU it was cold, as if it was outside disconnected.

Before yesterday: Standard was quiet, Optimal was loud, I didn't bother with the full and heavy OI.
As of yesterday: it starts quiet with standard, then out of the blue, the erver doing almost nothing, starts the repetitive loud-quiet cycles of what I used to hear at the very beginning when I set up the server and the fan mode was optimal.

those % definitions are not what I have experienced in the last weeks, standard was quiet and optimal was loud-quiet cycles.

I've checked the sensors, nothing worrying I guess:


Here is some background information:
This behaviour started only after I tried 4090 instead of 5090, the 4090 was recognised in the IMPI, but the 5090 not.
The fan became louder. So I decided to go back to 5090. It became quiet but then started this uncontrol switching between quiet and loud-quiet cycles.

Any idea?

 

[mattventura]

If you're experiencing a loud-quiet cycle, the most common cause is a fan going below its threshold, causing the system to think that the fan has failed, and ramping all the fans up to compensate.

I don't know what the equivalent page is on the modern web UI, but on the older one, there should be a section for fans, and a button to show thresholds. On this particular server, I had to adjust one of the thresholds downwards (to 100/200 from the default of 300/500) because it is a Noctua fan, and would dip below the threshold otherwise. TMK you can't adjust thresholds via the web UI, you have to use IPMI tools to do so. If you can't adjust the fan thresholds, then you probably need to do some cable modifications to make it not report the RPM.

 

[Alterra]

Those temperatures look fine to me.

I don't know what is wrong. Maybe the BMC remembers the 4090 sensors and thinks they are broken now that you replaced it with something else? Maybe try BMC reset?

As for fan speed alarm threshold setting, you could use ipmitool.

 

[fatherboard]

Here is one image I didn't think of adding before but you guys could understand what's going on:
every time the cycles change between quiet-loud, the readings change from all green to one of them red critical, then back to all green , then back to one critical, and it goes on like this. So what I showed all the way up was only one (all green), below is the other one (one critical):

 

[mattventura]

It's probably a fan threshold issue in that case. The first fan is running at 420RPM which is also its lower critical threshold. That causes the other fans to spin faster to compensate. Adjusting the threshold should fix that.

 

[fatherboard]

You mentioned I can't change the fan threshold in the web UI, I need to use ipmi tools. I can access the server using IPMI tool in Ubuntu 24.04.3, I've used it before only to check what fan mode was selected

ipmitool -I lanplus -H xxx.xxx.x.xxx -U ADMIN -P ADMIN raw 0x30 0x45 0x00

and it returned 00.

what's the line I need to use to set the threshold for Fan 1?

 

[Alterra]

Try ipmitool sensor thresh FAN1 lcr 280 with appropriate options to set the threshold to 280 RPM.

 

[fatherboard]

reduced the threshold to 280 for FAN1 -> still quiet-loud cycle

-> FAN3 showed up as critical
reduced threshold of FAN3 to 280
-> still quiet-loud cycle

-> FANA became critical
reduced threshold of FANA to 280
-> Server got angry and started full, continuously loud, doesn't react to setting back thresholds to 420, doesn't react to changing fan modes, the only way I could reason with the server is by reset to factory defaults: Maintenance -> BMC Reset -> Factory Default -> Remove current settings and restore to factory defaults.

This took it back to optimal (quiet-loud cycle), and I could with one click on standard to get back to quiet. No sure what happened.
The cure was BMC reset.

 

[fatherboard]

Although I reset BMC yesterday, today it's back to the same behaviour of quiet-loud cycles, it started immediately after / coincided with opening google earth in a Firefox browser.
Closed the browser with Google earth, but the behaviour continues.

Now even with a BMC reset, the same behaviour quiet-loud behaviour remains.

Then I tried again

ipmitool sensor thresh FAN1 lcr 280

with FAN1 then server got angry again and switch to continuous full.
I reset BMC again to factory, useless by now, it keeps the quiet-loud behaviour

 

[Trotador]

This work around also works with H133xx

https://forums.servethehome.com/index.php?resources/supermicro-x9-x10-x11-fan-speed-control.20/

 

[fatherboard]

My problem is the override. setting the fan speeds using that script in the link above is one thing, and the BMC override simply ignores whatever choices I make whether fan speeds or fan mode.

Is it safe to use this, which according to ChatGPT tells the BMC: “Don’t override my fan mode when PCIe gets warm.”

ipmitool -I lanplus -H xxx.xxx.xxx.xxx -U ADMIN -P ADMIN raw 0x30 0x91 0x5A 0x03 0x00
ipmitool -I lanplus -H xxx.xxx.xxx.xxx -U ADMIN -P ADMIN mc reset cold

I still quote from Chat GPT:

Then wait about 60–90 seconds for the BMC to reboot.
After that:

• Set the fan mode once (Standard) in the IPMI web UI.
• It will stay fixed — no more quiet–loud cycles.

This is safe as long as your case and GPU have their own cooling.

What not to do anymore

• Don’t lower thresholds with sensor thresh — the H13 BMC doesn’t allow it safely.
• Don’t rapidly switch fan modes trying to “unstick” it — the BMC will panic.
• Don’t rely on the “factory reset” loop — it just resets the symptoms.

knowing that the only PCIe connected is the GPU 5090 and I assume the GPU is taking care of itself (wait, is this even correct?).

I have noticed every time I do anything graphics hungry, the override is initiated, I'm totally ignored, I have to reset the BMC and switch again to Standard, then make another graphics demanding task and back again to the same loop.

is it correct and safe to use these two lines of code?

 

[fatherboard]

Absolutely nothing works, nothing.

I've tried the code in the link above

#set fan mode to "full"
ipmitool raw 0x30 0x45 0x01 0x01
#set fans in "system" zone to 37.5%
ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x24
#set fans in "peripheral" zone to 25%
ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x16

Doesn't work.

Tried the Chatgpt nonsense, unrecognised, turns out it either invented it or it read it from some pre-historic documentation.

Do you know if Asrock Rack motherboards (e.g. TURIN2D16-2T) have the same behaviour?

 

[fatherboard]

Anyone with Asrock Rack motherboard can confirm that you have full control over fan mode and operation as mentioned below and that the BMC is not taking over control in some cases and ignoring your instructions?
https://www.asrockrack.com/support/faq.de.asp?id=41