Is ASPM working?

[tcpluess]

Hi,
I am currently trying to optimise my home server a bit for more power saving, as it produces lots of heat.
So I enabled ASPM in the BIOS, switched to "powersave" CPU frequency governor using intel_pstate, and enabled energy saving options in the BIOS.
According to powertop, my processor now spends most of its time (>95%) in C6 state, I think this is good as it allowed me to save some 50 to 60 Watts, which is quite substantial.

I am wondering if I can save more power on the network cards and the SAS cards. Not because I hope my electricity bill becomes much lower, but because I hope I can lower a bit the temperature of these cards. Electronics love it when it is not too hot, so I think I can enhance a bit the endurance of my components if they run cooler.

I did a bit of research and found the following about ASPM in my server:

root@pve0:~# lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'
00:1c.0 PCI bridge: Intel Corporation C620 Series Chipset Family PCI Express Root Port #1 (rev fa) (prog-if 00 [Normal decode])
                LnkCap: Port #1, Speed 8GT/s, Width x1, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:1c.1 PCI bridge: Intel Corporation C620 Series Chipset Family PCI Express Root Port #2 (rev fa) (prog-if 00 [Normal decode])
                LnkCap: Port #2, Speed 8GT/s, Width x1, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:1c.2 PCI bridge: Intel Corporation C620 Series Chipset Family PCI Express Root Port #3 (rev fa) (prog-if 00 [Normal decode])
                LnkCap: Port #3, Speed 8GT/s, Width x1, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk-
00:1c.3 PCI bridge: Intel Corporation C620 Series Chipset Family PCI Express Root Port #4 (rev fa) (prog-if 00 [Normal decode])
                LnkCap: Port #4, Speed 8GT/s, Width x1, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk-
00:1c.5 PCI bridge: Intel Corporation C620 Series Chipset Family PCI Express Root Port #6 (rev fa) (prog-if 00 [Normal decode])
                LnkCap: Port #6, Speed 8GT/s, Width x1, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
01:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
05:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 06) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <32us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
50:02.0 PCI bridge: Intel Corporation Device 347a (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #5, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
51:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
89:02.0 PCI bridge: Intel Corporation Device 347a (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #13, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
89:04.0 PCI bridge: Intel Corporation Device 347c (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #15, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
8a:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
8b:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
                LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
8b:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
                LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+

so it seems, my two SAS3008 cards do not support ASPM. I am not surprised about this and I now bought a new 9400-16i card. It probably does also not support ASPM, but at least I found in its datasheet that this card has way less power dissipation than my two SAS3008. So I expect already a bit of improvement from this.

Further, I can see that my two Intel X520 NICs do support ASPM, however, lspci claims "ASPM Disabled". Why is this, can I enable it somehow, and is it worth it or will it lead to the network connection being interrupted all the time?

So I have my 2 NICs and 2 SAS cards that probably cannot do ASPM; if we look at the above list, we find that these devices remain

50:02.0 PCI bridge: Intel Corporation Device 347a (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #5, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
89:02.0 PCI bridge: Intel Corporation Device 347a (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #13, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
89:04.0 PCI bridge: Intel Corporation Device 347c (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #15, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+

that support ASPM but have it disabled. However I don't know what these devices are, it is probably something from the chipset. Would it be worth to enable ASPM on these as well, is it something I can probably find in the BIOS or can it be enabled only manually from the command line?

At least I can say that, enabling ASPM and powersave frequency governor saved not only a lot of electricity, but it also makes my little closet where I run this server much much cooler. I now probably can even install more quiet fans. I check the system temperatures using ipmitool, and I have only one part that worries me, it's the "CPU_VRMIN Temp" where I wonder if I can get this one also a bit cooler, probably by putting somewhere also another fan.

root@pve0:~# ipmitool sdr
CPU Temp         | 35 degrees C      | ok
PCH Temp         | 36 degrees C      | ok
System Temp      | 31 degrees C      | ok
Peripheral Temp  | 34 degrees C      | ok
CPU_VRMIN Temp   | 52 degrees C      | ok
VRMABCD Temp     | 40 degrees C      | ok
VRMEFGH Temp     | 38 degrees C      | ok
Inlet Temp       | no reading        | ns
DIMMA~D Temp     | 32 degrees C      | ok
DIMME~H Temp     | 33 degrees C      | ok
....

I have a Supermicro X12SPL board, and a Supermicro 743AC-668B chassis where I installed also a rear exhaust fan. I previously used a Papst fan, this was very good (lots of througput) but noisy, so I switched to a Noctua which has less throughput but is almost inaudible.

The goal of this exercise would be that everything becomes even a bit more cooler, so I can probably lower the fan speeds more and get this thing even quieter.

Except the CPU_VRMIN, my two hard disks are now the hottest parts in the system, with 37 degrees C. I have also a couple SSDs which are basically cold (like 27 degrees or so). (By the way - would it be worth it to enable IDLE_A, IDLE_B and IDLE_C on the SSDs, such that they can go to idle power states as well?)

 

[alaricljs]

Useful: The Curse of ASPM

 

[tcpluess]

Thanks for this link. It is an interesting read which I haven't found yet myself!

I tried the python script that enables the ASPM on the SAS cards. Indeed it works fine, my SAS3008 cards in fact do enter L0s and L1. However, read performance from the SSDs drops from 550 MB/sec to 200 MB/sec, so it is too bad.
But still the article is an interesting read.

I wonder where is my lower limit with ECC RAM and Xeon 4310 CPU.
Can I reach 100W? can I go below 100W? to 80W? to 60W? I have no idea.

I have the following other hardware:

• 2x Ultrastar DC HC530 HDD, max. 9W each, 18W total
• 2x SSD HUSSL4040B for the special device, max. 5.5W each, 11W total
• 2x SSD HUSSL4040B for a SSD only pool for VMs, again 11W
• 1x SSD HUSSL4040B for the system itself, 5.5W
• 2x SAS3008 card, I don't know how much this consumes but the datasheet says "power management at 18W", so let's say it's 36W in total
• 1x Intel X520-DA2 network card, 7W max.

this would sum up to ~80W, however, the 9W for the hard disks are only true when they are actively reading/writing, also the same for the SSDs.

Is it worth on the SSDs to enable IDLE_A, IDLE_B, IDLE_C timers with sdparm to have the SSDs go into some sort of sleep?

also, for the ZFS pool, if I shut down the disks: if data that is in the ARC is accessed, will this nevertheless spin up the disks? problem is, I have my Nextcloud data on the hard disks as I have lots of data that I want to have accessible.

 

[pimposh]

Is it worth on the SSDs to enable IDLE_A, IDLE_B, IDLE_C timers with sdparm to have the SSDs go into some sort of sleep?

While these modes actually did something to spinning rust i think for SSD either they do not work at all or difference is just a measurement margin error.

Also IDLE_B and IDLE_C modes requires spinning rust firmware to fully support them (eg. Seagate's SAS drives are fully supporting all modes while SATAs are only limited to one idling mode).

Per specs yours ssd seems to be more efficient than you posted and there is no difference in idle (if it exists at all) vs standby mode.

also, for the ZFS pool, if I shut down the disks: if data that is in the ARC is accessed, will this nevertheless spin up the disks? problem is, I have my Nextcloud data on the hard disks as I have lots of data that I want to have accessible.

Every 5 secs which prevent most of rust to idle/park when ZFS is used. Not the case with ext4.

 

[PANiCnz]

I'm currently going down the APSM / power saving rabbit hole. So far, the key learning seems to be that PCIe cards matter and are the main culprit that prevent the lower C states.

Unfortunately, that brings a few tradeoffs as the enterprise PCIe cards e.g. HBA's, NIC's etc. commonly seem to have problems with APSM.

So far, I've swapped out my LSI SAS 9217-8i for a ASM1166 card and have a TP-Link TL-NT521F on its way to replace my Intel 82599ES network card. The ASM1166 might be a bit of a gamble, but I'm using it with UNRAID and it seems popular amongst that community.

 

[i386]

Servers don't sleep

 

[tcpluess]

thanks for the responses!

Per specs yours ssd seems to be more efficient than you posted and there is no difference in idle (if it exists at all) vs standby mode.

indeed, I just also realised. Thanks for the link anyways. So I cannot save much power from the SSDs.

I'm currently going down the APSM / power saving rabbit hole. So far, the key learning seems to be that PCIe cards matter and are the main culprit that prevent the lower C states.

exactly. I see you switched to a SATA controller. I would like to keep my SAS card, as the SSDs I have are SAS and they are really good. Still 0% wear with insane amount of TB written, so I would like to keep them.

I am now waiting for my LSI 9400-16i HBA to arrive. According to the datasheet, it should support ASPM.
And even if not, it's power consumption should be around 12W, whereas my current 2x 9300-8i has more like 18W each. So even if the new HBA will not properly support ASPM, I will still save some power.
My main goal is not saving power because of the electricity bill (well, sort of it is) but mainly because I don't want to have such a big heat production.

Still I am considering switching over to SATA for the ZFS disks, as this would allow me to kick out one HBA. But sill.... would prefer the SAS approach.

Anyone got some experience with the LSI 9400-16i?

Servers don't sleep

totally true! but this one runs in my apartment and serves as a little space heater. While this will be handy in winter, during summer it is just annoying. Also, the heat generation forces me to use higher fan speeds and therefore, more noise.
I know I cannot have infinite CPU/disk power and no heat generation, but somehow, I would like my setup to be so efficient that I can let it run all day, to have it ready when I access it.
As mentioned in my first message, I previously had a HPE Microserver. A nice machine, but it was too weak for Proxmox and cannot handle enough disks for my use case. However, this was a very efficient device, with only ~22W at the outlet when in idle. I am now far, far away from this. It would be good if I can let my server sleep such that he is ready fast enough when he is accessed, but will not unnecessarily heat my apartment