IPMI help - Really frustrated

[thecoffeeguy]

Folks

I have been trying to figure out how to properly set my fan speed on my CPU for somet ime now and have spent countless hours reading about it, trying to get this to work. I thought I had it then something changed this morning.

I have the following mobo:
SUPERMICRO MBD-X10SRL-F Server Motherboard LGA 2011 R3 -
I have a Noctua cooler on it with two PWM fans.

I walked into my office this morning and could hear some fans 'pulsing'. From what I have learned in the past, I know it is the CPU fan and I cant seem to get it right the way i want (the pulsing sound is driving me crazy).

Here is the output from ipmi tool:

FAN1 | 1700.000 | RPM | ok | 300.000 | 500.000 | 700.000 | 25300.000 | 25400.000 | 25500.000
FAN2 | 1700.000 | RPM | ok | 300.000 | 500.000 | 700.000 | 25300.000 | 25400.000 | 25500.000

At one point, I had them steady at 500rpm, which was great. They were silent. I would like them to be set at 500pm, but go no higher than 900rpm and for it to stop pulsing....

I am really hoping for some help to figure this out. I am really at my wit's end right now with this


I appreciate the help.

TCG

 

[K D]

Did you try adjustjng the fan speed thresholds? Check out this thread.

https://forums.servethehome.com/ind...ns-on-xeon-sp-tower-server.17831/#post-172365

 

[EffrafaxOfWug]

You can't set the speed specifically, you need to configure the PWM duty cycle using ipmitool raws.

But the first problem seems to be the "pulsing" - I'm assuming you mean that every so often the fans ramp up to maximum and this makes the lil' fan on the CPU outrageously noisy? Is this accompanied by fan asserts in the IPMI log (either look int he web GUI or try `ipmitool sel list`)? There can be several reasons for this but if you're aiming to look at ~500rpm when idling, you'll need to sort out your lower threshholds - currently you've got your LNR, LCR and LNC thresholds set to 300, 500 and 700rpm respectively - you'll likely want to lower that so that the LNC threshold is 4-500 or so. Try using something like:

ipmitool sensor thresh "FAN1" lower 200 300 400
ipmitool sensor thresh "FAN2" lower 200 300 400

to change the lower assert thresholds. IIRC you might need to restart the BMC for it to pick up the new thresholds.

If they still pulse after that (along with an accompanying assert) then check if any of your fans are still registering speeds below the LNC threshold - I had a problem with Nocua fans on my system dropping to <300rpm at which point they register as 0rpm to IPMI and triggered an assert.

 

[thecoffeeguy]

You can't set the speed specifically, you need to configure the PWM duty cycle using ipmitool raws.

Yea, I remember tha ti had to set the fans to 'optimal' in the web ui, then use the raw commands to set the speeds. i made sure optimal is set in the web ui.

But the first problem seems to be the "pulsing" - I'm assuming you mean that every so often the fans ramp up to maximum and this makes the lil' fan on the CPU outrageously noisy? Is this accompanied by fan asserts in the IPMI log (either look int he web GUI or try `ipmitool sel list`)? There can be several reasons for this but if you're aiming to look at ~500rpm when idling, you'll need to sort out your lower threshholds - currently you've got your LNR, LCR and LNC thresholds set to 300, 500 and 700rpm respectively - you'll likely want to lower that so that the LNC threshold is 4-500 or so. Try using something like:

ipmitool sensor thresh "FAN1" lower 200 300 400
ipmitool sensor thresh "FAN2" lower 200 300 400

to change the lower assert thresholds. IIRC you might need to restart the BMC for it to pick up the new thresholds.

Correct on the pulsing. I was checking them and seeing the fans spin up from 500rpm to 1700rpm. I checked the logs in the web ui and it had entries like:

Lower critical - Going low - Assertion
Lower critical - Going Low - Deassertion

For both FANS (FAN1 and FAN2)
I have been playing with the RAW commands again this morning, adjusting thresholds and the pulsing seems to have stopped, but the fans are spinning at 1700rpm.

FAN1 | 1700.000 | RPM | nr | 300.000 | 400.000 | 500.000 | 1000.000 | 1200.000 | 1300.000
FAN2 | 1700.000 | RPM | nr | 300.000 | 400.000 | 500.000 | 1000.000 | 1200.000 | 1300.000

As you can see, i adjusted the high and low values (not sure what is right or wrong...hehe.)

For reference, here is the CPU heatsink and fan:
NH-U9DX i4

If they still pulse after that (along with an accompanying assert) then check if any of your fans are still registering speeds below the LNC threshold - I had a problem with Nocua fans on my system dropping to <300rpm at which point they register as 0rpm to IPMI and triggered an assert.

I did chck the web ui again and it is complaining (in the sensor readings that is not the log) about the upper non-recoverable for both FAN1 and FAN2

Thanks for the help.
I thin i just need to figure ouw how to slow the fans down?

Chers

TCG

 

[EffrafaxOfWug]

OK, 1700rpm basically means they're spinning at max revs according to the spec (actually 1600rpm according to Noctua but there's often a lot of rounding going on), so if you want them to go no higher than ~900rpm you can limit the duty cycle with the raws. Retrieve the current setting like so:

ipmitool raw 0x30 0x70 0x66 0x00 0x00
ipmitool raw 0x30 0x70 0x66 0x00 0x01

The last hex above specifies the thermal zone - 0x00 usually meaning the CPU, 0x01 meaning system/peripherals although this depends on the motherboard. 60% is probably a good number to start with which would mean a max speed of approx. 960rpm so set the duty cycle to 60% (0x3c) on both zones like so:

ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x3c
ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x3c

You might want to set a different threshold on the CPU fan vs. the system fans based on their comparative loudness and you'll definitely want to keep an eye on temps after these values have been set - always a good idea to try a soak test after limiting your fans just to make sure you're not going to accidentally cook anything (esp. if you've made an accidental mistake and limited one of the fans to 10%, etc etc).

 

[thecoffeeguy]

OK, 1700rpm basically means they're spinning at max revs according to the spec (actually 1600rpm according to Noctua but there's often a lot of rounding going on), so if you want them to go no higher than ~900rpm you can limit the duty cycle with the raws. Retrieve the current setting like so:

ipmitool raw 0x30 0x70 0x66 0x00 0x00
ipmitool raw 0x30 0x70 0x66 0x00 0x01

Ok. Here is output from that:

➜  ~ ipmitool -H 192.168.1.220  raw 0x30 0x70 0x66 0x00 0x00
 64
➜  ~ ipmitool -H 192.168.1.220  raw 0x30 0x70 0x66 0x00 0x01
 64

The last hex above specifies the thermal zone - 0x00 usually meaning the CPU, 0x01 meaning system/peripherals although this depends on the motherboard. 60% is probably a good number to start with which would mean a max speed of approx. 960rpm so set the duty cycle to 60% (0x3c) on both zones like so:

ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x3c
ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x3c

You might want to set a different threshold on the CPU fan vs. the system fans based on their comparative loudness and you'll definitely want to keep an eye on temps after these values have been set - always a good idea to try a soak test after limiting your fans just to make sure you're not going to accidentally cook anything (esp. if you've made an accidental mistake and limited one of the fans to 10%, etc etc).

I took the values you specified above, and it does not seem to have set it (do i need to reset the BMC?)

➜  ~ ipmitool -H 192.168.1.220 raw 0x30 0x70 0x66 0x01 0x00 0x3c

➜  ~ ipmitool -H 192.168.1.220 raw 0x30 0x70 0x66 0x01 0x01 0x3c

But it doesnt look like it took it:

FAN1             | 1700.000   | RPM        | nr    | 200.000   | 300.000   | 400.000   | 1000.000  | 1200.000  | 1300.000
FAN2             | 1700.000   | RPM        | nr    | 200.000   | 300.000   | 400.000   | 1000.000  | 1200.000  | 1300.000

Did i do something wrong?
Thank you for your help

TCG

 

[EffrafaxOfWug]

What values are returned when you run the first set of read commands again? Hopefully they should now read 0x3c rather than 0x64 (possible I've got my octects mixed up as I was typing that from memory but the page K D linked to earlier should have a full description).

ipmitool raw 0x30 0x70 0x66 0x00 0x00
ipmitool raw 0x30 0x70 0x66 0x00 0x01

Been so long since I did it I can't remember if it requires a BMC reset or not, but just in case it does:

ipmitool mc reset warm

 

[thecoffeeguy]

What values are returned when you run the first set of read commands again? Hopefully they should now read 0x3c rather than 0x64 (possible I've got my octects mixed up as I was typing that from memory but the page K D linked to earlier should have a full description).

ipmitool raw 0x30 0x70 0x66 0x00 0x00
ipmitool raw 0x30 0x70 0x66 0x00 0x01

Been so long since I did it I can't remember if it requires a BMC reset or not, but just in case it does:

ipmitool mc reset warm

Seems to be working now.

FAN1             | 500.000    | RPM        | ok    | 200.000   | 300.000   | 400.000   | 1000.000  | 1200.000  | 1300.000
FAN2             | 500.000    | RPM        | ok    | 200.000   | 300.000   | 400.000   | 1000.000  | 1200.000  | 1300.000

I will keep an eye on it.
Thank you so much!!!

TCG

 

[EffrafaxOfWug]

Glad it's working, but watch out for your upper thresholds there - they're now set to 1000, 1200 and 1300 somehow; note that these aren't specifications of upper limits but rather when asserts should be thrown, so if any of your fans do actually go above 1000rpm you might again see your fans ramping up to max; as you can see from your first post the SM default values are at insanely unreachable speeds so that the upper asserts never fire off.

 

[StevenDTX]

Dont forget, you have to reboot the IPMI after you make the changes for them to stick.

 

[nthu9280]

Sensor thresholds are for alerts. These are permanent until you reset ipmi to factory defaults.

Raw commands are to set a specific PWM speed. These are gone when you reboot.

AFAIK, you first need to set the speed to full and then set the speed for each zone to fixed setting using raw commands. If you have it optimal first, it will constantly monitor and adjusts the speed and overrides your zone settings.

 

[thecoffeeguy]

Thanks for all the help folks.
Here is my latest query to check on the fans:

FAN1             | 700.000    | RPM        | ok    | 200.000   | 300.000   | 400.000   | 1000.000  | 1200.000  | 1300.000
FAN2             | 700.000    | RPM        | ok    | 200.000   | 300.000   | 400.000   | 1000.000  | 1200.000  | 1300.000

Jumped up to 700 but that is ok. Still quiet.

Speaking of the upper threshholds, any recommendations on what i should set those two?

Really appreciate all the help here. I plan to document all the notes here for future reference.

Cheers

TCG

 

[EffrafaxOfWug]

Set them to "I want an alarm to go off when the fan speed hits X" levels - not tested it myself but I think the fans will also spin at max rpm on an upper threshold assert as well, something you don't want, so you might just want to set the same insanely high values that SM use OotB; I've left mine like that.