Problems with C package states and energy saving.

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
N

Nozle

Member
Jul 16, 2024
82
1
8
Hello,

I have a cluster of three Lenovo M90Q Gen 4 motherboards, each with an Intel 13500 processor and 64GB of RAM, along with two NVMe drives and an Intel x710 DA2 GPU updated to the latest version.

The problem is that I can't get the package state (c) to be higher than c2. I've configured PowerTop Autotune and PowerSave, everything is up to date, but I can't get it to go beyond c2.

Any recommendations on how to solve this?

Thanks in advance.
 
N

nexox

Well-Known Member
May 3, 2023
1,871
921
113
It's likely ASPM not working correctly somewhere, there are a lot of options for what can go wrong, I suggest you start with this blog post and see what you can find: The Curse of ASPM
 
N

Nozle

Member
Jul 16, 2024
82
1
8
I was looking at that post months ago, but I didn't get very far; as you can see, I'm still stuck on the lc2 package.





Pkg(HW) | Core(HW) | CPU(OS) 0 CPU(OS) 1
| | C0 active 1.0% 0.0%
| | POLL 0.0% 0.0 ms 0.0% 0.0 ms
| | C1_ACPI 0.1% 0.1 ms 0.1% 0.4 ms
C2 (pc2) 84.8% | | C2_ACPI 1.5% 0.9 ms 0.4% 0.9 ms
C3 (pc3) 0.0% | C3 (cc3) 0.0% | C3_ACPI 97.8% 34.6 ms 99.5% 154.7 ms
C6 (pc6) 0.0% | C6 (cc6) 1.1% |
C7 (pc7) 0.0% | C7 (cc7) 96.9% |
C8 (pc8) 0.0% | |
C9 (pc9) 0.0% | |
C10 (pc10) 0.0% | |

| Core(HW) | CPU(OS) 2 CPU(OS) 3
| | C0 active 0.2% 0.2%
| | POLL 0.0% 0.0 ms 0.0% 0.0 ms
| | C1_ACPI 0.1% 0.3 ms 0.2% 0.3 ms
| | C2_ACPI 1.4% 0.9 ms 1.5% 0.6 ms
| C3 (cc3) 0.0% | C3_ACPI 98.2% 88.7 ms 97.9% 18.2 ms
| C6 (cc6) 1.5% |
| C7 (cc7) 92.1% |
| |
| |
| |

| Core(HW) | CPU(OS) 4 CPU(OS) 5
| | C0 active 0.3% 0.0%
| | POLL 0.0% 0.0 ms 0.0% 0.0 ms
| | C1_ACPI 0.1% 0.2 ms 0.0% 0.2 ms
| | C2_ACPI 1.8% 0.9 ms 0.1% 0.7 ms
| C3 (cc3) 0.0% | C3_ACPI 97.6% 94.0 ms 99.8% 81.0 ms
| C6 (cc6) 1.1% |
| C7 (cc7) 96.2% |
| |
| |
| |

| Core(HW) | CPU(OS) 6 CPU(OS) 7
| | C0 active 0.5% 0.1%
| | POLL 0.0% 0.0 ms 0.0% 0.0 ms
| | C1_ACPI 0.1% 0.2 ms 0.0% 0.2 ms
| | C2_ACPI 2.2% 0.9 ms 0.2% 1.0 ms
| C3 (cc3) 0.0% | C3_ACPI 97.0% 48.1 ms 99.6% 67.5 ms
| C6 (cc6) 1.6% |
| C7 (cc7) 95.4% |
| |
| |
| |

| Core(HW) | CPU(OS) 8 CPU(OS) 9
| | C0 active 0.2% 0.1%
| | POLL 0.0% 0.0 ms 0.0% 0.0 ms
| | C1_ACPI 0.2% 0.2 ms 0.1% 0.3 ms
| | C2_ACPI 0.8% 0.8 ms 0.6% 0.9 ms
| C3 (cc3) 0.0% | C3_ACPI 98.7% 54.8 ms 99.2% 136.3 ms
| C6 (cc6) 0.8% |
| C7 (cc7) 97.3% |
| |
| |
| |

| Core(HW) | CPU(OS) 10 CPU(OS) 11
| | C0 active 0.2% 0.2%
| | POLL 0.0% 0.0 ms 0.0% 0.0 ms
| | C1_ACPI 0.1% 0.2 ms 0.1% 0.2 ms
| | C2_ACPI 0.5% 0.8 ms 1.1% 0.8 ms
| C3 (cc3) 0.0% | C3_ACPI 99.1% 82.5 ms 98.4% 17.4 ms
| C6 (cc6) 0.6% |
| C7 (cc7) 94.4% |
| |
| |
| |

| Core(HW) | CPU(OS) 12
| | C0 active 0.4%
| | POLL 0.0% 0.0 ms
| | C1_ACPI 0.1% 0.2 ms
| | C2_ACPI 1.6% 0.9 ms
 
I

Icipher

New Member
Jun 15, 2022
6
1
3
Are you by any chance, using the sata port as well? if you are it will block anything higher than C2. This is assuming there is still the ribbon connector on the gen 4 boards.
 
N

Nozle

Member
Jul 16, 2024
82
1
8
Icipher said:
Are you by any chance, using the sata port as well? if you are it will block anything higher than C2. This is assuming there is still the ribbon connector on the gen 4 boards.
Click to expand...
No, I'm only using PCI for the Intel x710da2, nothing else. There's no USB, no HDMI, nothing. Ethernet and integrated audio are disabled.

M.2 slots 1 and 2 are being used with NVMe drives.
 
I

Icipher

New Member
Jun 15, 2022
6
1
3
Have you tried booting an Ubuntu live iso, and installing powertop in there to rule it out being whatever OS you have running?
 
EffrafaxOfWug

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,432
544
113
I can see C3 and lower are there in your powertop output (but dear god man learn to use the [ code ] tags!) so on the face of it it looks like it should be available - probably you have a device or driver in there somewhere that's blocking the lower states from being reached. For instance, IIRC the class LSI 9211-8i would stop C3 and lower being reached.

As suggested by nexox, there's a wealth of info out there from people who've grappled with the same thing - sometimes unsuccessfully, sometimes opening a whole Pandora's box of hackery. But the lspci output showing all your hardware and its ASPM state is a good place to start, as is checking dmesg/kernel logs for any errors/warnings about ASPM or APCI.

Out-of-the-box settings on my Framework desktop; as you can see some devices are auto-enabled for ASPM whereas others aren't:
Code: 
effrafax@wug:~# lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM |Disabled;)'
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo PCIe USB4 Bridge (rev 02) (prog-if 00 [Normal decode])
                LnkCap: Port #247, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo PCIe USB4 Bridge (rev 02) (prog-if 00 [Normal decode])
                LnkCap: Port #247, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo GPP Bridge (rev 02) (prog-if 00 [Normal decode])
                LnkCap: Port #1, Speed 16GT/s, Width x2, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
00:02.5 PCI bridge: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo GPP Bridge (rev 02) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo GPP Bridge (prog-if 00 [Normal decode])
                LnkCap: Port #1, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo Internal GPP Bridge to Bus [C:A] (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo Internal GPP Bridge to Bus [C:A] (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo Internal GPP Bridge to Bus [C:A] (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
bf:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8126 5GbE Controller (rev 01)
                LnkCap: Port #0, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c0:00.0 Non-Volatile memory controller: Micron/Crucial Technology T500 NVMe PCIe SSD (rev 01) (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
c1:00.0 Non-Volatile memory controller: Micron/Crucial Technology T500 NVMe PCIe SSD (rev 01) (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
c2:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Strix Halo [Radeon Graphics / Radeon 8050S Graphics / Radeon 8060S Graphics] (rev c1)
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c2:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Radeon High Definition Audio Controller [Rembrandt/Strix]
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c2:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Strix/Krackan/Strix Halo CCP/ASP
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c2:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Strix Halo USB 3.1 xHCI (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c2:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h/19h/1ah HD Audio Controller
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c3:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo PCIe Dummy Function
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c3:00.1 Signal processing controller: Advanced Micro Devices, Inc. [AMD] Strix/Krackan/Strix Halo Neural Processing Unit (rev 11)
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c4:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Strix Halo USB 3.1 xHCI (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c4:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Strix Halo USB 3.1 xHCI (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c4:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Strix Halo USB 3.1 xHCI (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c4:00.5 USB controller: Advanced Micro Devices, Inc. [AMD] Strix Halo USB4 Host Router (prog-if 40 [USB4 Host Interface])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c4:00.6 USB controller: Advanced Micro Devices, Inc. [AMD] Strix Halo USB4 Host Router (prog-if 40 [USB4 Host Interface])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
Some devices play nice with ASPM, some don't. If you're feeling brave you can bosh some stuff in manually via sysfs e.g.
Code: 
effrafax@wug:~# echo 1 > /sys/bus/pci/drivers/r8169/0000\:bf\:00.0/link/l1_aspm

# ASPM now enabled for the realtek NIC listed above
bf:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8126 5GbE Controller (rev 01)
                LnkCap: Port #0, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
...or there's some really handy helper scripts like aspm-tool to force ASPM on during the OS run (risking instability or crappy performance in devices that don't properly support it however).

Assuming your BIOS nicely supports ASPM (I haven't read the manual but since <C3 states are exposed let's assume it does), and I know the X710 does, my immediate suspect would be your NVME drives. What model are they and what does NVME CLI or smartctl say about their power states?
Code: 
effrafax@wug:~# nvme get-feature /dev/nvme0 -f 0x0c -H
get-feature:0x0c (Autonomous Power State Transition), Current value:0x00000001
        Autonomous Power State Transition Enable (APSTE): Enabled
        Auto PST Entries        .................
        Entry[ 0]
        .................
        Idle Time Prior to Transition (ITPT): 100 ms
        Idle Transition Power State   (ITPS): 3
        .................
        Entry[ 1]
        .................
        Idle Time Prior to Transition (ITPT): 100 ms
        Idle Transition Power State   (ITPS): 3
        .................
        Entry[ 2]
        .................
        Idle Time Prior to Transition (ITPT): 100 ms
        Idle Transition Power State   (ITPS): 3
        .................
        Entry[ 3]
        .................
        Idle Time Prior to Transition (ITPT): 2000 ms
        Idle Transition Power State   (ITPS): 4
        .................
        Entry[ 4]
        .................
        Idle Time Prior to Transition (ITPT): 0 ms
        Idle Transition Power State   (ITPS): 0
        .................
...
...
...
 
  • Like
Reactions: Whaaat and nexox
N

Nozle

Member
Jul 16, 2024
82
1
8
EffrafaxOfWug said:
I can see C3 and lower are there in your powertop output (but dear god man learn to use the [ code ] tags!) so on the face of it it looks like it should be available - probably you have a device or driver in there somewhere that's blocking the lower states from being reached. For instance, IIRC the class LSI 9211-8i would stop C3 and lower being reached.

As suggested by nexox, there's a wealth of info out there from people who've grappled with the same thing - sometimes unsuccessfully, sometimes opening a whole Pandora's box of hackery. But the lspci output showing all your hardware and its ASPM state is a good place to start, as is checking dmesg/kernel logs for any errors/warnings about ASPM or APCI.

Out-of-the-box settings on my Framework desktop; as you can see some devices are auto-enabled for ASPM whereas others aren't:
Code: 
effrafax@wug:~# lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM |Disabled;)'
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo PCIe USB4 Bridge (rev 02) (prog-if 00 [Normal decode])
                LnkCap: Port #247, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo PCIe USB4 Bridge (rev 02) (prog-if 00 [Normal decode])
                LnkCap: Port #247, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo GPP Bridge (rev 02) (prog-if 00 [Normal decode])
                LnkCap: Port #1, Speed 16GT/s, Width x2, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
00:02.5 PCI bridge: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo GPP Bridge (rev 02) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo GPP Bridge (prog-if 00 [Normal decode])
                LnkCap: Port #1, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo Internal GPP Bridge to Bus [C:A] (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo Internal GPP Bridge to Bus [C:A] (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo Internal GPP Bridge to Bus [C:A] (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
bf:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8126 5GbE Controller (rev 01)
                LnkCap: Port #0, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c0:00.0 Non-Volatile memory controller: Micron/Crucial Technology T500 NVMe PCIe SSD (rev 01) (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
c1:00.0 Non-Volatile memory controller: Micron/Crucial Technology T500 NVMe PCIe SSD (rev 01) (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
c2:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Strix Halo [Radeon Graphics / Radeon 8050S Graphics / Radeon 8060S Graphics] (rev c1)
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c2:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Radeon High Definition Audio Controller [Rembrandt/Strix]
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c2:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Strix/Krackan/Strix Halo CCP/ASP
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c2:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Strix Halo USB 3.1 xHCI (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c2:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h/19h/1ah HD Audio Controller
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c3:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Strix/Strix Halo PCIe Dummy Function
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c3:00.1 Signal processing controller: Advanced Micro Devices, Inc. [AMD] Strix/Krackan/Strix Halo Neural Processing Unit (rev 11)
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c4:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Strix Halo USB 3.1 xHCI (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c4:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Strix Halo USB 3.1 xHCI (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c4:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Strix Halo USB 3.1 xHCI (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c4:00.5 USB controller: Advanced Micro Devices, Inc. [AMD] Strix Halo USB4 Host Router (prog-if 40 [USB4 Host Interface])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
c4:00.6 USB controller: Advanced Micro Devices, Inc. [AMD] Strix Halo USB4 Host Router (prog-if 40 [USB4 Host Interface])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
Some devices play nice with ASPM, some don't. If you're feeling brave you can bosh some stuff in manually via sysfs e.g.
Code: 
effrafax@wug:~# echo 1 > /sys/bus/pci/drivers/r8169/0000\:bf\:00.0/link/l1_aspm

# ASPM now enabled for the realtek NIC listed above
bf:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8126 5GbE Controller (rev 01)
                LnkCap: Port #0, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
...or there's some really handy helper scripts like aspm-tool to force ASPM on during the OS run (risking instability or crappy performance in devices that don't properly support it however).

Assuming your BIOS nicely supports ASPM (I haven't read the manual but since <C3 states are exposed let's assume it does), and I know the X710 does, my immediate suspect would be your NVME drives. What model are they and what does NVME CLI or smartctl say about their power states?
Code: 
effrafax@wug:~# nvme get-feature /dev/nvme0 -f 0x0c -H
get-feature:0x0c (Autonomous Power State Transition), Current value:0x00000001
        Autonomous Power State Transition Enable (APSTE): Enabled
        Auto PST Entries        .................
        Entry[ 0]
        .................
        Idle Time Prior to Transition (ITPT): 100 ms
        Idle Transition Power State   (ITPS): 3
        .................
        Entry[ 1]
        .................
        Idle Time Prior to Transition (ITPT): 100 ms
        Idle Transition Power State   (ITPS): 3
        .................
        Entry[ 2]
        .................
        Idle Time Prior to Transition (ITPT): 100 ms
        Idle Transition Power State   (ITPS): 3
        .................
        Entry[ 3]
        .................
        Idle Time Prior to Transition (ITPT): 2000 ms
        Idle Transition Power State   (ITPS): 4
        .................
        Entry[ 4]
        .................
        Idle Time Prior to Transition (ITPT): 0 ms
        Idle Transition Power State   (ITPS): 0
        .................
...
...
...
Click to expand...

I've done a lot of reading and tried many things before coming here, but I haven't had any luck, which is why I'm here.

In my case, all the devices appear to be ASPM Enabled (L1).

NvMe slot 1 is a Western Digital SN740 256GB and NVMe slot 2 is a Hynix P41 Pro 1TB.

Sorry for the code.

Code: 
root@pve1:~# lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9                                                                          :.]+|ASPM |Disabled;)'
00:01.0 PCI bridge: Intel Corporation 12th Gen Core Processor PCI Express x16 Co                                                                          ntroller #1 (rev 02) (prog-if 00 [Normal decode])
                LnkCap: Port #2, Speed 32GT/s, Width x8, ASPM L1, Exit Latency L                                                                          1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
00:1a.0 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #25                                                                           (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #25, Speed 16GT/s, Width x4, ASPM L1, Exit Latency                                                                           L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
00:1b.0 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #21                                                                           (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #21, Speed 16GT/s, Width x4, ASPM L1, Exit Latency                                                                           L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10Gb                                                                          E SFP+ (rev 01)
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1                                                                           <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10Gb                                                                          E SFP+ (rev 01)
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1                                                                           <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
02:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN770 / PC SN740 2                                                                          56GB / PC SN560 (DRAM-less) NVMe SSD (rev 01) (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L                                                                          1 <8us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
03:00.0 Non-Volatile memory controller: SK hynix Platinum P41/PC801 NVMe Solid S                                                                          tate Drive (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L                                                                          1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
 
N

Nozle

Member
Jul 16, 2024
82
1
8
I just got a new NVMe drive and installed Ubuntu (to avoid using a live USB), and after configuring Autotune and PowerTop, I can see C8 states included!

Now I'm completely lost here (Proxmox runs from an SN740 NVMe drive and has a second NVMe drive for storage); that's the only difference.
 
EffrafaxOfWug

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,432
544
113
Well that's certainly peculiar.

Sadly, how well things like SSDs respond to power saving features is often poorly documented; TechPowerUp for example are one of the few sites that include power consumption data for ASPM on/off in their SSD reviews, and even that methodology is too new for their Hynix P41 review - although we can see from this page that its successor the P51 doesn't look like it supports ASPM properly, so potentially that's a culprit there.

Of course as you say it could also be the USB controller. Sadly nailing these things down is often a matter of trial and error because so much of the detail is opaque, but at least you've found a combo that works without having to hack your ACPI tables or similar.
 
N

Nozle

Member
Jul 16, 2024
82
1
8
EffrafaxOfWug said:
Well that's certainly peculiar.

Sadly, how well things like SSDs respond to power saving features is often poorly documented; TechPowerUp for example are one of the few sites that include power consumption data for ASPM on/off in their SSD reviews, and even that methodology is too new for their Hynix P41 review - although we can see from this page that its successor the P51 doesn't look like it supports ASPM properly, so potentially that's a culprit there.

Of course as you say it could also be the USB controller. Sadly nailing these things down is often a matter of trial and error because so much of the detail is opaque, but at least you've found a combo that works without having to hack your ACPI tables or similar.
Click to expand...
I just verified that on Ubuntu I can boot into deeper C states (up to C8) if I don't connect the Intel x710da2; if I connect it, then I'm stuck in C2...
 
I

Icipher

New Member
Jun 15, 2022
6
1
3
x710 usually needs aspm L1 and L2 forcing using the enable aspm script that's floating about in various forms, then it will happily go lower
 
N

Nozle

Member
Jul 16, 2024
82
1
8
So I just removed the x710 da2 and started Proxmox without it, and it still won't pass c2, however in Ubuntu with the same configuration it reaches c8
 
N

Nozle

Member
Jul 16, 2024
82
1
8
Update: I installed PowerTop 2.14 (previously 2.15) and now I can see C3_ACPI at 97.7%.

I don't understand why (I also can't see anything above C3_ACPI; it's the highest status that appears).

Intel x710 da2 install.
 
N

Nozle

Member
Jul 16, 2024
82
1
8
  • Sad
Reactions: luckylinux
L

luckylinux

Well-Known Member
Mar 18, 2012
1,473
455
83
Nozle said:
I ran the aspm.py script but it still doesn't get past c2.
Click to expand...
Then read ALL of the Troubleshooting and linked Resources. There can be A LOT of Reasons why it doesn't want to go into ASPM, from BIOS BUGs (e.g. FADT), non-compliant PCIe Device, PCIe Bridges, etc.
 
  • Like
Reactions: nexox
You must log in or register to reply here.
Top Bottom