Supermicro X9/X10/X11 Fan Speed Control

[mattlach]

It will persist across reboots. It is completely removing power from the wall that resets IPMI.

That's what I meant. I hadn't shut it down, and pulled the power.

So I tried it.

I can report that my X9DRI-F can handle being powered fully off (no plug in wall) thus requiring the BMC to reinitialize, and these IPMI fan settings persist.

As mentioned previously, I don't doubt there are other models in which this is not the case. Supermicro has been anything but consistent when it comes to stuff like this over the years.


Right now I am chuckling to myself at my "innovative" solution towards accessing the BMC/IPMI when using the board as a workstation

 

[jonboy345]

Moved to some different fans in my chassis, and have noticed the result from the fan_measurement.sh returns odd results... The fan rpm doesn't consistently track levels...

As in, level 25 will cause the fans to spin faster than level 30, 35, 45, and 55... See the results below. Any ideas?

Level,FAN1,FAN2,FAN3,FAN4,FAN5,FANA,FANB
100,6200,6400,6800,7000,6300,,6100,6900
95,6000,6300,6600,6800,6100,,5900,6700
90,5800,6100,6400,6600,5900,,5800,6500
85,5600,5900,6200,6400,5700,,5500,6200
80,5500,5700,5900,6200,5500,,5400,6000
75,5300,5500,5700,5900,5300,,5200,5700
70,5100,5300,5500,5600,5100,,5000,5500
65,4800,5000,5200,5400,4900,,4700,5200
60,4900,5100,5200,5400,4900,,4500,4900
55,4400,4500,4600,4800,4300,,4300,4700
50,4100,4200,4400,4500,4100,,4000,4400
45,4300,4500,4600,4800,4300,,3700,4100
40,5300,5500,5800,6000,5500,,3500,3800
35,3300,3400,3500,3600,3200,,3200,3500
30,3000,3100,3200,3300,2900,,2900,3200
25,4900,5100,5300,5400,4900,,2600,2800
20,2400,2500,2500,2600,2300,,2300,2500

Edit: Nevermind. Stopped the SMFC container, ran fan_measurement.sh again and results look correct now.

Level,FAN1,FAN2,FAN3,FAN4,FAN5,FANA,FANB
100,6200,6400,6800,7100,6300,,6100,6900
95,6000,6300,6600,6800,6100,,5900,6700
90,5800,6100,6400,6600,5900,,5700,6500
85,5600,5900,6200,6400,5700,,5500,6200
80,5500,5700,5900,6200,5500,,5400,6000
75,5300,5500,5700,5900,5300,,5200,5700
70,5100,5300,5500,5600,5100,,5000,5500
65,4800,5000,5200,5400,4900,,4700,5200
60,4600,4800,4900,5100,4600,,4500,4900
55,4400,4500,4700,4800,4300,,4300,4700
50,4100,4200,4400,4500,4100,,4000,4400
45,3800,4000,4100,4200,3800,,3700,4100
40,3600,3700,3800,3900,3500,,3500,3800
35,3300,3400,3500,3600,3200,,3200,3500
30,3000,3100,3200,3300,2900,,2900,3200
25,2700,2800,2900,3000,2600,,2600,2900
20,2400,2400,2500,2600,2300,,2300,2500

 

[6uwy8uDg]

Being unable to control the fan speeds on the X9SCL(BMC 03.52 2016-11-10) with mentioned commands(so far only readout and mode set or "Invalid data field in request" and "Invalid command", not even a clue how to change the threshold value if even possible), I decided to replace the three 7k fans with 80mm Noctua's, to be shortly after boot being alarmed by the smell of 2.5 SAS HDD's slowly starting to solder-reflux before they would die, because the fan speed only reaching a slilent few hundred rpm.

Since the fans require the included extension cables to reach the mobo. anyway, it was a quick fix to cut the pwm signal(the outter blue wire) from those cables and have the fans run full speed while maintaining SG/PG readout.
I gave the extention cables a PVC tape label that they are pwm-less for future me, and ensured the wire coming out of the connector has some lenght to it, so it can be simply resoldered.

 

[peter_s]

An interesting new experience related to AST2600 BMC chip and X13SAE-F motherboard from an smfc user can be found here.

The point is the fan control methods described here are working on AST2600. It has FULL MODE as well, and fan levels can be controlled properly. The naming of the fans and the definition of the thresholds are different (it implements Lower Critical threshold only!). Here is an example:

Sensor ID              : CPU_FAN1 (0x41)
Entity ID             : 29.1
Sensor Type (Threshold)  : Fan
Sensor Reading        : 420 (+/- 0) RPM
Status                : ok
Lower Non-Recoverable : na
Lower Critical        : 140.000
Lower Non-Critical    : na
Upper Non-Critical    : na
Upper Critical        : na
Upper Non-Recoverable : na
Positive Hysteresis   : 140.000
Negative Hysteresis   : 140.000
Assertion Events      :
Assertions Enabled    : lcr-
Deassertions Enabled  : lcr-

 

[DaveInTexas]

An interesting new experience related to AST2600 BMC chip and X13SAE-F motherboard from an smfc user can be found here.

The point is the fan control methods described here are working on AST2600. It has FULL MODE as well, and fan levels can be controlled properly. The naming of the fans and the definition of the thresholds are different (it implements Lower Critical threshold only!). Here is an example:

Sensor ID              : CPU_FAN1 (0x41)
Entity ID             : 29.1
Sensor Type (Threshold)  : Fan
Sensor Reading        : 420 (+/- 0) RPM
Status                : ok
Lower Non-Recoverable : na
Lower Critical        : 140.000
Lower Non-Critical    : na
Upper Non-Critical    : na
Upper Critical        : na
Upper Non-Recoverable : na
Positive Hysteresis   : 140.000
Negative Hysteresis   : 140.000
Assertion Events      :
Assertions Enabled    : lcr-
Deassertions Enabled  : lcr-

@peter_s thanks for posting this info.

Interesting the fan hysteresis is so high, and an odd number. That implies the max fan speed the BMC can track is 35,700 RPM, unless ASPEED also changed how the BMC reports read speeds.

Are you certain it's not possible to set the other lower and upper threshold values? I'd be surprised if they dropped the uppers entirely. I'd be a bit surprised if they have indeed dropped the usage of LNC and LNR, though it would kind of make sense as they are effectively irrelevant thresholds.

 

[DaveInTexas]

Being unable to control the fan speeds on the X9SCL(BMC 03.52 2016-11-10) with mentioned commands(so far only readout and mode set or "Invalid data field in request" and "Invalid command", not even a clue how to change the threshold value if even possible)

Based on my research, my understanding is that board does not support manual fan speed control via IPMI. I believe they only have limited control via the BIOS or no controls (fully automatic) similar to the current methodology of proprietary Dell and HPE boards.

 

[Negative Entropy]

FWIW I recall only being able to change the lowers. Well, those were also the only ones I was worried about. Also Noctuas as I recall. See my post #140 in this thread:

https://forums.servethehome.com/index.php?threads/supermicro-x9-x10-x11-fan-speed-control.10059/post-339648

 

[Z0eff]

Getting some major headaches with this X11SSi-LN4F. I've had no issues with a different X13 board but this X11 board will only spin the fans from the FAN4 header. If I unplug that while it's on it'll spin up everything else in a panic but besides that there is no actual method to spin up the rest of the fans. They're a mixture of OEM and Noctua fans.

Here's an example command, this works just fine on the X13 board. It ramps up fans to 100%.
sudo ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x64
But absolutely nothing happens on this X11 board.

An interesting point is that if I angle any of the other fan connectors and only insert it partly the fans spin up to 100%. From what I can tell I'm connecting 3 pins and excluding the PWM pin. Is there perhaps a setting somewhere to switch between voltage and PWM?

Here's the output of ipmitool sensor on the X11 board:

IANA PEN registry open failed: No such file or directory
CPU Temp         | 33.000     | degrees C  | ok    | 0.000     | 0.000     | 0.000     | 95.000    | 100.000   | 100.000
PCH Temp         | 34.000     | degrees C  | ok    | 0.000     | 5.000     | 16.000    | 90.000    | 95.000    | 100.000
System Temp      | 28.000     | degrees C  | ok    | -10.000   | -5.000    | 0.000     | 80.000    | 85.000    | 90.000
Peripheral Temp  | 42.000     | degrees C  | ok    | -10.000   | -5.000    | 0.000     | 80.000    | 85.000    | 90.000
VcpuVRM Temp     | 36.000     | degrees C  | ok    | -5.000    | 0.000     | 5.000     | 95.000    | 100.000   | 105.000
DIMMA1 Temp      | na         |            | na    | na        | na        | na        | na        | na        | na
DIMMA2 Temp      | 26.000     | degrees C  | ok    | -5.000    | 0.000     | 5.000     | 80.000    | 85.000    | 90.000
DIMMB1 Temp      | na         |            | na    | na        | na        | na        | na        | na        | na
DIMMB2 Temp      | 25.000     | degrees C  | ok    | -5.000    | 0.000     | 5.000     | 80.000    | 85.000    | 90.000
FAN1             | na         |            | na    | na        | na        | na        | na        | na        | na
FAN2             | na         |            | na    | na        | na        | na        | na        | na        | na
FAN3             | na         |            | na    | na        | na        | na        | na        | na        | na
FAN4             | 1400.000   | RPM        | ok    | 300.000   | 500.000   | 700.000   | 25300.000 | 25400.000 | 25500.000
FANA             | na         |            | na    | na        | na        | na        | na        | na        | na

FAN1-4 are connected to fans, FANA is currently unpopulated. Yet FAN4 is the only one actually doing anything.
When angling the connector as described above of any of the other fan headers I can see RPM values appear which tells me the fan headers are functional and the sensor reporting seems to be working fine.

And the X13 board:

IANA PEN registry open failed: No such file or directory
CPU Temp         | 33.000     | degrees C  | ok    | 5.000     | 5.000     | na        | na        | 100.000   | 100.000
PCH Temp         | 52.000     | degrees C  | ok    | 5.000     | 5.000     | na        | na        | 90.000    | 105.000
System Temp      | 31.000     | degrees C  | ok    | 5.000     | 5.000     | na        | na        | 85.000    | 90.000
Peripheral Temp  | 36.000     | degrees C  | ok    | 5.000     | 5.000     | na        | na        | 85.000    | 90.000
VRM_VCORE Temp   | 36.000     | degrees C  | ok    | 5.000     | 5.000     | na        | na        | 100.000   | 105.000
VRMVIN_AUX Temp  | 33.000     | degrees C  | ok    | 5.000     | 5.000     | na        | na        | 100.000   | 105.000
DIMMAB Temp      | 37.000     | degrees C  | ok    | 5.000     | 5.000     | na        | na        | 85.000    | 90.000
M2SSD1 Temp      | na         |            | na    | na        | na        | na        | na        | na        | na
M2SSD2 Temp      | na         |            | na    | na        | na        | na        | na        | na        | na
M2SSD3 Temp      | na         |            | na    | na        | na        | na        | na        | na        | na
CPU_FAN1         | 840.000    | RPM        | ok    | 140.000   | 140.000   | na        | na        | 35560.000 | 35700.000
CPU_FAN2         | na         |            | na    | na        | na        | na        | na        | na        | na
SYS_FAN1         | 700.000    | RPM        | ok    | 140.000   | 140.000   | na        | na        | 35560.000 | 35700.000
SYS_FAN2         | 700.000    | RPM        | ok    | 140.000   | 140.000   | na        | na        | 35560.000 | 35700.000
SYS_FAN3         | 980.000    | RPM        | ok    | 140.000   | 140.000   | na        | na        | 35560.000 | 35700.000

CPU_FAN2 is currently unpopulated as you probably guessed.

Does anyone have any ideas?

 

[RolloZ170]

An interesting point is that if I angle any of the other fan connectors and only insert it partly the fans spin up to 100%.

in PWM mode there is full +12V to the FAN.
without PWM pin the FAN is detected as 0 RPM = Panik.

 

[Z0eff]

in PWM mode there is full +12V to the FAN.
without PWM pin the FAN is detected as 0 RPM = Panik.

Ah, so that's just expected behavior then.
(For that particular quirk, not my issue)

 

[i386]

without PWM pin the FAN is detected as 0 RPM = Panik.

Not sure if this is correct: 3 pin fans have usually a pin each for 12v, ground and sensor (for rpm, it's triggered twice per one full rotation of the fans/blade)

 

[RolloZ170]

Not sure if this is correct: 3 pin fans have usually a pin each for 12v, ground and sensor (for rpm, it's triggered twice per one full rotation of the fans/blade)

right, my bad. i meant without TACH.
result is same. if one of the FANs run under critical RPM, panik mode is enabled.

 

[peter_s]

Have you tried to set FAN thresholds or reset BMC?

 

[Z0eff]

Have you tried to set FAN thresholds or reset BMC?

Yep on both. I've even resorted to trying to set the lower threshold of FAN4 to a value higher than what it's reporting just so force it into panic mode but that didn't work either. If that had worked then at least I'd have spinning fans..

 

[DaveInTexas]

Getting some major headaches with this X11SSi-LN4F. I've had no issues with a different X13 board but this X11 board will only spin the fans from the FAN4 header. If I unplug that while it's on it'll spin up everything else in a panic but besides that there is no actual method to spin up the rest of the fans. They're a mixture of OEM and Noctua fans.

Here's an example command, this works just fine on the X13 board. It ramps up fans to 100%.
sudo ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x64
But absolutely nothing happens on this X11 board.

That is to be expected given what you posted afterwards:

Here's the output of ipmitool sensor on the X11 board:

FAN1             | na         |            | na    | na        | na        | na        | na        | na        | na
FAN2             | na         |            | na    | na        | na        | na        | na        | na        | na
FAN3             | na         |            | na    | na        | na        | na        | na        | na        | na
FAN4             | 1400.000   | RPM        | ok    | 300.000   | 500.000   | 700.000   | 25300.000 | 25400.000 | 25500.000
FANA             | na         |            | na    | na        | na        | na        | na        | na        | na

FAN1-4 are connected to fans, FANA is currently unpopulated. Yet FAN4 is the only one actually doing anything.
When angling the connector as described above of any of the other fan headers I can see RPM values appear which tells me the fan headers are functional and the sensor reporting seems to be working fine.

[...snip...]

Does anyone have any ideas?

sudo ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x64 says "set fan zone 1 fans to 100% power"

Fan zone 1 = FANA in the case of your X11SSi-LN4F board. Since there's no fan connected to that fan header - according to the IPMI output above the BMC does not see a fan connected there - then this behavior is as expected.

The "na" in the IPMI readout means "not available" (duh, I know, but bear with me here), which in Supermicro-speak means "no fan connected to this fan header" or "no reading."

The bottom line here is what your IPMI readout is telling you, is that the BMC thinks you only have a fan connected to fan header FAN4.

I would recommend not unplugging or plugging in fans into the fan headers while power is applied to the board. Treating fans like hot-swap hard drives is risky. Regardless of that however, if you plug in fans and they are not identified, the next thing to try would be shut down the server and restart it (cold boot). You may need to shutdown, unplug power for 30-60 seconds, re-apply power, and then turn on, but for most boards doing all that is not necessary, though a warm or cold boot may be. What you're endeavoring to do is to force the BMC chip to re-inventory the fan header states.

IF none of that works, I'd try testing the fans on another board. If they work on another board, then the evidence points to a bad X11 mobo (possibly the BMC chip has gone bad or possibly something else is wrong on the board, but either way you're screwed).

 

[Z0eff]

That is to be expected given what you posted afterwards:



sudo ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x64 says "set fan zone 1 fans to 100% power"

Fan zone 1 = FANA in the case of your X11SSi-LN4F board. Since there's no fan connected to that fan header - according to the IPMI output above the BMC does not see a fan connected there - then this behavior is as expected.

The "na" in the IPMI readout means "not available" (duh, I know, but bear with me here), which in Supermicro-speak means "no fan connected to this fan header" or "no reading."

The bottom line here is what your IPMI readout is telling you, is that the BMC thinks you only have a fan connected to fan header FAN4.

I would recommend not unplugging or plugging in fans into the fan headers while power is applied to the board. Treating fans like hot-swap hard drives is risky. Regardless of that however, if you plug in fans and they are not identified, the next thing to try would be shut down the server and restart it (cold boot). You may need to shutdown, unplug power for 30-60 seconds, re-apply power, and then turn on, but for most boards doing all that is not necessary, though a warm or cold boot may be. What you're endeavoring to do is to force the BMC chip to re-inventory the fan header states.

IF none of that works, I'd try testing the fans on another board. If they work on another board, then the evidence points to a bad X11 mobo (possibly the BMC chip has gone bad or possibly something else is wrong on the board, but either way you're screwed).

I've also tried zone 0, nothing

Have also cold booted a few times.

I'm thinking of just getting a fan controller of some kind and bypassing the motherboard alltogether. But I do worry what else isn't working properly and that it's not just the BMC chip...

Edit: something I forgot to add is that FAN4 only runs at 1400 rpm at all times. It doesn't spin up to 100% during boot or change when I switch the fan profile around. Which worries even more.

EDIT2: Have updated the IPMI firmware from 1.58 to 1.68. No difference.

 

[DaveInTexas]

I've also tried zone 0, nothing

Have also cold booted a few times.

I'm thinking of just getting a fan controller of some kind and bypassing the motherboard alltogether. But I do worry what else isn't working properly and that it's not just the BMC chip...

Edit: something I forgot to add is that FAN4 only runs at 1400 rpm at all times. It doesn't spin up to 100% during boot or change when I switch the fan profile around. Which worries even more.

EDIT2: Have updated the IPMI firmware from 1.58 to 1.68. No difference.

Yeah, I wish you good luck, but share your sentiment. I'd be inclined to RMA that board if that's an option. Otherwise it's likely best used in non-critical applications and try to get it to run all fans at full speed. If it's not a financial burden for you, I would frankly chuck it and get a different board from a reputable source and start over. Just my $0.02

 

[nabsltd]

It looks like the X9SRL-F falls into the "can't manually control the duty cycle" category.

I can set the thresholds (although there is no real need to, as the built-in lower fan thresholds are 15% or less), and set the general mode (standard, full, optimal), but can't set the duty cycle to a specific percentage. I used the following commands, and none did anything to the fan speed, but none returned errors (which the X10/X11 variants do):

ipmitool raw 0x30 0x91 0x5A 0x03 0x00 0x40
ipmitool raw 0x30 0x91 0x5A 0x03 0x01 0x40
ipmitool raw 0x30 0x91 0x5A 0x03 0x10 0x40
ipmitool raw 0x30 0x91 0x5A 0x03 0x11 0x40

I have version 3.19 of the BMC firmware, and that is the latest. The motherboard is in a Supermicro 826 chassis, with the stock fan wall fans replaced with the lower noise (but nearly the same performance) FAN-0074L4.

 

[DaveInTexas]

It looks like the X9SRL-F falls into the "can't manually control the duty cycle" category.

Appreciate the effort to test your board, and to report your findings on this forum.

FWIW, my understanding is neither the X9SRL (without the -F designation), nor the X9SRW-F (sister boards to yours) support manual fan speed control either. Many more X9 boards don't. Most X9 boards have a Nuvoton brand BMC: either WPCM450 or WPCM450R.

AFAIK, Supermicro X9 server boards came with one of the following BMC chips:

• Nuvoton WPCM450
• Nuvoton WPCM450R
• Renesas SH7757

X9 board support for manual fan speed control is a box of chocolates for sure, but some do allow it (e.g. X9DR3-F, X9DRE-LN4F, and at least some - if not all - of the X9DRG series). There are others, but at least half of all X9 board models do NOT support manual fan speed control, unfortunately.

 

[Z0eff]

Yeah, I wish you good luck, but share your sentiment. I'd be inclined to RMA that board if that's an option. Otherwise it's likely best used in non-critical applications and try to get it to run all fans at full speed. If it's not a financial burden for you, I would frankly chuck it and get a different board from a reputable source and start over. Just my $0.02

To give an update on this, turns out it wasn't the motherboard at all! The 2 fans that were attached to the FAN4 header seem to draw an insane amount of power while only spinning at 1400 RPM. I only found out about this when I tried a stand alone fan controller and got the exact same issue of the fans not spinning up except for the fans previously connected to the FAN4 header. The fan controller's IC got super hot as well which is why I suspect these fans are drawing way too much power. So much in fact that not enough current is getting to the other fans it seems?!

I've since replaced these (Broken?) fans with a silverstone branded one and it's running just fine on the FAN4 header. Can control the speed and everything. I'm leaving the other fans on the separate fan controller just so it doesn't make a ton of noise when it boots up haha.