Interesting, I am pretty sure I tried (and failed) with another Tool (cannot remember which one exactly) on Linux.
Thanks for the Tip
.Did you also try that with a 5000 Series CPUs such as e.g. AMD Ryzen 7 5700X ?
EU ASRock Rack B450D4U-V1LQ5 mATX AM4
- Thread starter Cruzader
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Thanks for the Tip
.Did you also try that with a 5000 Series CPUs such as e.g. AMD Ryzen 7 5700X ?
I agree on the Overclock Part, but to be able to UNDERvolt it would be good to have that Option.
Probably too many Calls from Customers that ran into Stability Issues maybe.
@Cruzader: for the 1U Case with Riser mount that you showed in this Post
Is there space for a 10mm Heatsink (or possibly 3mm Heatsink) plus thermal Pad for the NVMe Device, especially the one facing bottom ?
I tried running some Samsung PM983 960gb and they get quite Hot even at Idle (45-50°C according to one Sensor, 82-88°C according to another Sensor [probably the Controller]).
Thanks in Advance
Is there space for a 10mm Heatsink (or possibly 3mm Heatsink) plus thermal Pad for the NVMe Device, especially the one facing bottom ?
I tried running some Samsung PM983 960gb and they get quite Hot even at Idle (45-50°C according to one Sensor, 82-88°C according to another Sensor [probably the Controller]).
Thanks in Advance
I'm curious if anybody managed to get ASPM working on this ***** Board.
I did a quick Test with a 360W Seasonic 80+ Gold PSU.
I run some ASPM Enabler Script then
Without NVMe / Without Mellanox ConnectX-4 LX NIC: approx 25W Idle
With NVMe / Without Mellanox ConnectX-4 LX NIC: approx 33-35W Idle
With NVMe / With Mellanox ConnectX-4 LX NIC: approx 40W Idle
I find this VERY high considering that a similar Intel Xeon System idles around 20W (granted without NVMe but with SATA SSDs, and Intel X710 instead of the Mellanox ConnectX-4 LX), partly due to this Ryzen Chiplet Design.
Still, the fact that we are unable to turn on ASPM is a Bummer. I might need to give
Weird is that the Samsung SSD claims to NOT support ASPM (Latency is unlimited), yet it claims it within its Capabilities
I even managed to update Firmware to latest EDA7902Q Firmware, but ASPM stayed disabled .
I did a quick Test with a 360W Seasonic 80+ Gold PSU.
I run some ASPM Enabler Script then
powertop --auto-tune but there are apparently still 3 Devices which do NOT want to play along:
Code:
03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01) (prog-if 01 [AHCI 1.0])
Subsystem: ASMedia Technology Inc. 400 Series Chipset SATA Controller
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 81
IOMMU group: 15
Region 5: Memory at fc380000 (32-bit, non-prefetchable) [size=128K]
Expansion ROM at fc300000 [disabled] [size=512K]
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee00000 Data: 0000
Capabilities: [78] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D3 NoSoftRst+ PME-Enable+ DSel=0 DScale=0 PME-
Capabilities: [80] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <32us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap- ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Kernel driver in use: ahci
Kernel modules: ahci
28:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (prog-if 02 [NVM Express])
Subsystem: Samsung Electronics Co Ltd SSD 970 EVO
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 44
IOMMU group: 18
Region 0: Memory at fcb10000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at fcb00000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b0] MSI-X: Enable+ Count=33 Masked-
Vector table: BAR=0 offset=00003000
PBA: BAR=0 offset=00002000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [158 v1] Power Budgeting <?>
Capabilities: [168 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [188 v1] Latency Tolerance Reporting
Max snoop latency: 1048576ns
Max no snoop latency: 1048576ns
Capabilities: [190 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates-
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
L1SubCtl2:
Kernel driver in use: nvme
Kernel modules: nvme
29:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (prog-if 02 [NVM Express])
Subsystem: Samsung Electronics Co Ltd SSD 970 EVO
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 45
IOMMU group: 19
Region 0: Memory at fca10000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at fca00000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b0] MSI-X: Enable+ Count=33 Masked-
Vector table: BAR=0 offset=00003000
PBA: BAR=0 offset=00002000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [158 v1] Power Budgeting <?>
Capabilities: [168 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [188 v1] Latency Tolerance Reporting
Max snoop latency: 1048576ns
Max no snoop latency: 1048576ns
Capabilities: [190 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates-
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-With NVMe / Without Mellanox ConnectX-4 LX NIC: approx 33-35W Idle
With NVMe / With Mellanox ConnectX-4 LX NIC: approx 40W Idle
I find this VERY high considering that a similar Intel Xeon System idles around 20W (granted without NVMe but with SATA SSDs, and Intel X710 instead of the Mellanox ConnectX-4 LX), partly due to this Ryzen Chiplet Design.
Still, the fact that we are unable to turn on ASPM is a Bummer. I might need to give
setup_var.efi (and/or BIOS Patch) another try and turn on Aggressive Link Power Management for SATA as well as manually set ASPM to L0s+L1.Weird is that the Samsung SSD claims to NOT support ASPM (Latency is unlimited), yet it claims it within its Capabilities
Code:
L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates-
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-I don't know, that is a different Model.
I tried the PM983 in a Supermicro X11SCZ-F and after the usual lovely ACPI Patching, ASPM auto-tuning Script and even FORCING ASPM to be enabled on the PM983 (where the Device clearly doesn't state it as being supported), the CPU is stuck at Package C3 (PC3) anyways according to turbostat / cpupower.
So, BIOS or not BIOS on this ASROCK Rack B450D4U-V1LQ5 Motherboard, it still would NOT work I believe
.The only Thing that might help (on OTHER Systems, NOT an Option on this specific Ryzen Motherboard) is to try a PCH connected PCIe Slot instead of a PCIe Slot directly tied to the CPU.
ASPM seems to be easier/more forgiving via the PCH Slots, for instance with the Mellanox ConnectX-4 LX.
I did NOT test without NVMe and WITH the Mellanox ConnectX-4 LX though, but anyways the SATA Aggressive Link Power Management (hidden in BIOS) might need to be turned on anyways first, because that is preventing ASPM as well.
@dbram: I think we need to Undervolt and possible Underclock a bit with that Cooler
Did the Fans ramp up to 100% ? To be honest I'm NOT 100% sure the "Smart Fan" Points in the BIOS Table are set correctly / intuitively.
Did you try using e.g. a Molex 4-pin to Fan Adapter to make sure they run at full speed ?
Did you try removing the existing Thermal Paste and installing your own across ALL the CPU Heatspreader / Cooler Base ?
Did the Fans ramp up to 100% ? To be honest I'm NOT 100% sure the "Smart Fan" Points in the BIOS Table are set correctly / intuitively.
Did you try using e.g. a Molex 4-pin to Fan Adapter to make sure they run at full speed ?
Did you try removing the existing Thermal Paste and installing your own across ALL the CPU Heatspreader / Cooler Base ?
I think Linus (LTT) did that quite a few Years back.something i'd have to look into. for now the workloads don't seem to require 100% cpu over longer periods.
i didn't look at any fan speeds while doing the stress-ng, will have to look later on that. they did start to make a lot more noise when running for a minute or so
the cpu in this system had some thermal paste from being mounted in a midi desktop case before, i just put the gelid cooler with the pre-applied thermal paste on top
I should use the generated heat and transfer it into a hot water tank ...
Not sure I would reccomend it due to all the extra Complexity, Parts (Pumps, Heat Exchangers, Pipes/Tubes, ...), Risk of Leakage (and Corrosion) and lack of flexibility if you ever have to pull out a Server to Fix something / Troubleshoot. I guess there are Water Cooling Quick-Connect Fittings and whatnot, but I think the extra work is not really worth the Trouble. Only Exception could be the Fan Noise ...
Not now that I bought so many SC512 Casesmaybe we should go into the direction that Hetzner did for this boards. they have an open 2U kind of custom frame just one fan combined with a shroud voor memory, cpu and vrm doing the cooling. hot-isle, cold-isle however is less trivial for a homelab
You say PCB Design Skills at zero Level but I only played a bit around with Kicad and my PCB Skills are archived after 15 Years or so of UniversitySome efforts in Kicad for a pcb version of the prototype. Still very early stage and my pcb design skills are beginner level, through-hole components only for the moment.
production of pcbs at a Chinese pcb factory is done and shipped. waiting on arrival (estimated a week or two).
I probably should re-orient the esp32's usb connection and ethernet jack. this prototype should allow for mounting it on a modified io-shield and have the usb connection connected to the one of internal usb headers of the mb (that means that 5V standby power connection might conflict with usb power).
View attachment 44055
I probably should avoid restricting too much cooling air flow
.To be honest I made ZERO Progress here.
My Smartphone broke down and besides the Inconvenience I also need to repair that to recover some Data
.And on the Topic of this Board I also didn't decide what to do yet
. Without knowing if you will Opensource it or not, I will probably have to go through all of the work and reinvent the Wheel and do pretty much what you did.Probably with some Variations e.g. add a Fan Header for having Fan control via the OS, some 16 Pin IDC Connector to directly plug the Supermicro Front Panel (plus either 4 x 2 Pin for the Power/Reset/HDD LED/Power LED Connectors, or some IDC Connector to go to the ASRock Board, etc).
Plus of course all the Software Development. I stopped a couple Weeks ago testing the MS2130 HDMI -> USB Capture Device on a RockPro64 aarch64 Device and it works OK with
ustreamer. I still would need to do the entire things after that (kvmd + the USB-HID Adapter).Although for Security Reasons, even if you plan to go with MQTT, I'd set that up as Authenticated + Encrypted on say 20 different Containers each with their own MQTT Server and Network, so they are (as much as possible) decoupled.
EDIT 1: I was thinking of actually making 2 x IDC Connectors, one for the Main ASRock Front Panel Header, one for the AUX ASRock Header (where we could get the 5Vsb from, in particular).