Limit boost to 3800 for Threadripper PRO 5965WX

[jnolla]

Hi Team,

We are running into an issue with our Threadripper PRO servers. We are trying to ensure they do not go above 3800 on their boost. We can set the speed manually on the BIOS, but if we want to allow the CPU to run at lower speeds and just boost to 3800, it goes past it up to 4400 causing thermal throttling.

This will always go above 3.8GHz
cpupower frequency-set -g schedutil -d 1.8GHz -u 3.8GHz
cpupower frequency-set -g ondemand -d 1.8GHz -u 3.8GHz

This works, cpu stays within 2.8GHz
cpupower frequency-set -g schedutil -d 1.8GHz -u 2.8GHz

Manufacturer: ASRockRack
Product Name: WRX80D8-2T

bios-version
10.27

OCTweaker
CPU Frequency and Voltage(VID) Change
Manual 3800 - keeps all cores at 3.8Ghz all the time.

 

[Wasmachineman_NL]

Best to lower Precision Boost limits instead.

 

[jnolla]

Best to lower Precision Boost limits instead.

Thank you for the quick response. How can we configure properly to achieve this? We have no experience with this.

 

[Wasmachineman_NL]

Thank you for the quick response. How can we configure properly to achieve this? We have no experience with this.

I don't know anything about TR but if it's the same as Ryzen you should go into the Precision Boost Overdrive menu and set lower EDC/PPT values there.

 

[CyklonDX]

I would recommend just running following (to set max clock frequency) without playing with scaling governors... (its likely a reason why you aren't getting locked in at max frequencies you want.)

cpupower frequency-set -u 3800MHz


(and set yourself ondemand governor)
cpupower frequency-set -g ondemand


If you want the change to be permanent just add it to sysctl or tuned.

 

[jnolla]

cpupower frequency-set -g ondemand -d 1.8GHz -u 3.8GHz
This parameter is not working, keep boosting right pass 3.8GHz

 

[RolloZ170]

limit cpu power can not work, if only one core has workloads even a few watts are enough to get up to 3,8Ghz,
but should not overheating.

 

[RolloZ170]

i am also not and AMD TR expert, but can you just disable the Turbo in BIOS ?

 

[CyklonDX]

cpupower frequency-set -g ondemand -d 1.8GHz -u 3.8GHz
This parameter is not working, keep boosting right pass 3.8GHz

I wrote that you should just execute this

cpupower frequency-set -u 3800MHz

 

[jnolla]

It the same command, does not work. It's either boost enabled or manual clocking, which as per the initial post, it causes all CPU to run at 3800 all the time.

 

[RolloZ170]

tried to disable turbo boost in BIOS yet ?

 

[CyklonDX]

it causes all CPU to run at 3800 all the time.

Yes, and no.

This is motherboard's pstates - The package power will be much lower that's why its able to keep its clock up-high as there's no load on them - that doesn't mean they are "hot", they are actually cold at that time - and do use less power just as if it was running on low clocks.
As load lands on cpu the clocks will go down/keep at specified limit if it meet package wattage limitations.

That is the correct behavior under linux to show you max clock when idle.

 

[bayleyw]

well, perhaps thermal throttling is the right behavior? modern CPUs will boost up to max clocks until they get too hot, then roll back clocks until the CPU tracks Tjmax. If you don't like thermal throttling for whatever reason, you can manually set EDC/PPT to track a lower power point. All Threadrippers will lock to 280W by design so if your thermal solution can't dissipate 280W you will run at Tjmax under load.
what cooler are you using? a 24c Threadripper should be easy to deal with thermally unless you are running in a 2U server or something.

 

[jnolla]

I am not sure what you are referring too. OC Tweaker if manual, multiplier and voltage will be set based on user selection. If we set to 3800 it will stay at that clock speed.

Yes, and no.

This is motherboard's pstates - The package power will be much lower that's why its able to keep its clock up-high as there's no load on them - that doesn't mean they are "hot", they are actually cold at that time - and do use less power just as if it was running on low clocks.
As load lands on cpu the clocks will go down/keep at specified limit if it meet package wattage limitations.

 

[CyklonDX]

I am not sure what you are referring too. OC Tweaker if manual, multiplier and voltage will be set based on user selection. If we set to 3800 it will stay at that clock speed.

This isn't overclocking your system. This is just setting max turbo state at 3800MHz. It locks max clock state at 3800

echo value | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq

cpupower frequency set -u -- means to set scaling max frequency, it doesn't force the cpu to hang around there - it just sets max it allows turbo to go. Turbo scales with max package wattage allowed (i.e. in your case thats 280W) It won't go over that value.
If there's no load on chip/core it reports highest allowed frequency as ready to receive work-time order.
(i.e. i'm free, i'm fast -> based on system work scheduler)
This is a default behavior, it also doesn't mean its going to hit 3.8GHz when work hits it. As whole package will be limited by your 280W.
Small work likely will see only few secs at 3.8GHz and effective clock will be around 1-2GHz.

 

[jnolla]

@CyklonDX your information is correct. No one stated this is overclocking the system, just that this in the only value in BIOS we have tweaked to keep clock at 3800Mhz.

The issue as stated above is that the system is not respecting when UP limit is set to 3800MHZ


----WORKING AT 2800MHZ----

cpupower frequency-set -u 2800MHz

cpupower frequency-info
analyzing CPU 0:
driver: acpi-cpufreq
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 1.80 GHz - 7.02 GHz
available frequency steps: 3.80 GHz, 2.80 GHz, 1.80 GHz
available cpufreq governors: conservative ondemand userspace powersave performance schedutil
current policy: frequency should be within 1.80 GHz and 2.80 GHz.
The governor "schedutil" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.69 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: no

cat /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_freq
2759731
2780149
2687364
2794119
2793863
2501965
2614949
2759600
2606696
2736015
2794085
2794555
2794715
2629936
2673019
2763344
2794053
2771195
2793996
2767226
2546229
2763711
2783499
2716963
2774756
2792284
2794181
2606665
2734249
2750456
2766333
2690292
2785724
2712893
2690547
2680289
2721949
2318590
2764037
2779805
2566010
2747828
2754455
2793788
2641967
2653045
2793225
2548927



----NOT WORKING AT 3800MHZ----

cpupower frequency-set -u 3800MHz

cpupower frequency-info
analyzing CPU 0:
driver: acpi-cpufreq
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 1.80 GHz - 7.02 GHz
available frequency steps: 3.80 GHz, 2.80 GHz, 1.80 GHz
available cpufreq governors: conservative ondemand userspace powersave performance schedutil
current policy: frequency should be within 1.80 GHz and 3.80 GHz.
The governor "schedutil" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 3.62 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: no


cat /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_freq
4133211
4138924
4498175
4190910
4524725
4384317
3908282
4204444
3833672
4426025
4124712
4139393
4410932
4432982
4444728
4387443
4270089
4037267
4456204
3842129
4240215
4527916
4505368
4051587
3454037
4536241
4070933
4449132
3850176
4514589
4318434
4157070
3834634
4006935
4251848
4395145
3819873
4322250
4381026
4438533
4376829
4395695
4297663
4062343
4476898
3632831
4469132
3900826

 

[CyklonDX]

can you get max value from (ensure its actually set on to 3800MHz on all.
/cpufreq/scaling_max_freq

also whats the content of your config file
/etc/sysconfig/cpupower
There's a chance its set with performance governor or something when operation starts.


also whats your system amd_pstate value (passive, active, guided) in kernel params.
If you do not have them you can try by editing your grub startup entry with amd_pstate=passive
(or you can modprobe it to unload, and load back in the driver)

and then you can set which amd scaling policy you want to use for scaling.

amd-pstate CPU Performance Scaling Driver — The Linux Kernel documentation

 

[jnolla]

sudo grep . /sys/devices/system/cpu/cpu0/cpufreq/*
/sys/devices/system/cpu/cpu0/cpufreq/affected_cpus:0
/sys/devices/system/cpu/cpu0/cpufreq/bios_limit:3800000
/sys/devices/system/cpu/cpu0/cpufreq/cpb:1
/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq:2800000
/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq:7021093
/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_min_freq:1800000
/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_transition_latency:0
/sys/devices/system/cpu/cpu0/cpufreq/freqdomain_cpus:0 24
/sys/devices/system/cpu/cpu0/cpufreq/related_cpus:0
/sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies:3800000 2800000 1800000
/sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors:conservative ondemand userspace powersave performance schedutil
/sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:3496301
/sys/devices/system/cpu/cpu0/cpufreq/scaling_driver:acpi-cpufreq
/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:schedutil
/sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq:3800000
/sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq:1800000
/sys/devices/system/cpu/cpu0/cpufreq/scaling_setspeed:<unsupported>
grep: /sys/devices/system/cpu/cpu0/cpufreq/stats: Is a directory

cat /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_freq
4178112
4151253
4140022
4154700
4134418
4128007
4122443
4134892
4110399
4112873
4102547
4095616
4093441
4093606
4094905
4075474
4097770
4091382
4105984
4103430
4108178
4121076
4105690
4108087
4127514
4117855
4117319
4132445
4150355
4136302
4137272
4185686
4196856
4190964
4165231
4163727
4246399
4239247
4243261
4226295
4282347
4311322
4310574
4225080
4287559
4314434
4259106
4267047

 

[CyklonDX]

Get acpi_cpufreq replaced with amd_pstate=passive, and then retry the command (if no change retry with amd_pstate=active.)

(whats the os, kernel? - ensure your bios is up-to-date, are you using efi?)

 

[jnolla]

journalctl -b 0 | grep pstate
Jan 29 09:25:17 x1 kernel: Command line: BOOT_IMAGE=/vmlinuz-5.15.0-91-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro i8042.reset quiet nosplash debug intel_pstate=disable amd-pstate=active initcall_blacklist=acpi_cpufreq_init amd_pstate.shared_mem=1
Jan 29 09:25:17 x1 kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-5.15.0-91-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro i8042.reset quiet nosplash debug intel_pstate=disable amd-pstate=active initcall_blacklist=acpi_cpufreq_init amd_pstate.shared_mem=1


cpupower frequency-info
analyzing CPU 0:
no or unknown cpufreq driver is active on this CPU
CPUs which run at the same hardware frequency: Not Available
CPUs which need to have their frequency coordinated by software: Not Available
maximum transition latency: Cannot determine or is not supported.
Not Available
available cpufreq governors: Not Available
Unable to determine current policy
current CPU frequency: Unable to call hardware
current CPU frequency: Unable to call to kernel
boost state support:
Supported: yes
Active: no


sudo dmidecode -s bios-version
10.27

sudo dmidecode -t 2
# dmidecode 3.3
Getting SMBIOS data from sysfs.
SMBIOS 3.3.0 present.

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
Manufacturer: ASRockRack
Product Name: WRX80D8-2T
Version:
Serial Number: BR80FB004300179
Asset Tag:
Features:
Board is a hosting board
Board is removable
Board is replaceable
Location In Chassis:
Chassis Handle: 0x0003
Type: Motherboard
Contained Object Handles: 0