ASRock Rack B650D4U MicroATX for Ryzen 7000 processors
- Thread starter zer0sum
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
For the record I contacted ASRockRack back in May and they sent me a beta BIOS and BMC firmware that reduced my issues significantly (though not completely). Since then they've release BIOS v21.08 and BMC 06.09.00 as stable which I have updated to and I think my system is finally stable. **fingers crossed**
Last edited:
thought is was the same trouble than at my C741 board, normaly they use chips with fixed quad mode.
well a BMC update fixed this, but replacement to a fixed quad mode BIOS chip would do the job too.
I can't say to much ofc, but on this bord there is a other defect that causes 00 if not taken care of.
Well, we are curently deploying a System with 21.08 / 6.09.00 and randomly Server + BMC / Ipmi crashes.
What's the current state of ECC on this board with EPYC/Ryzen Processors?
Got this board, EPYC and some ECC 5600 memory. Wondering if I wanna swap for faster CPU and faster non-ECC memory. Box won't do anything mission-critical (storage) and memory will be tested before putting it in use.
Got this board, EPYC and some ECC 5600 memory. Wondering if I wanna swap for faster CPU and faster non-ECC memory. Box won't do anything mission-critical (storage) and memory will be tested before putting it in use.
ASRockRack B650D4U power usage and other observations
Owing to the fiasco surrounding my failure to acquire a Supermicro H13SAE-MF, I'm starting my build a year late on an ASRockRack B650D4U. I'd written this board off originally owing to what sounded like a bevvy of hardware failures - seems this was finally corrected on newer hardware revisions a couple of years back. Mine is a rev. 4.01 board with an M80 serial.
Anyway, I haven't seen much in the way of people reporting on board power and ASPM issues so I figured it'd be useful to have it down somewhere, even if I'm catastrophically late to the AM5 server party.
Overall the board seems perfectly happy with turning on ASPM everywhere with the exception of the USB controller which resists all attempts to enable ASPM despite the fact it's listed as supporting L0 and L1; I assume this might a kernel quirk (I've tried the default trixie 6.12 and the trixie-backports 7.1.3) but haven't dug in to this yet; only found this forlorn post on stack exchange from someone with the same issue (also a debian user).
sysfs attrs for the xHCI for L0s and L1 are both set to 1 already so I'm not sure what else there is to be done.
The above is a fairly barebones layout (basic fresh trixie install, no drives attached other than the boot M2, 10Gb SFPs not plugged in). Complete rundown of the hardware in this test config are:
Power stats with the above are as follows (plus a few extra measurements showing the affect the PCI cards have), rounding up:
So whilst the ASPM idle savings aren't great they are there to a degree. TBH I'm a little disappointed the idle draw of the system's still in the 50W range, seems that's still in the realms of slight suckitude, esp. given the low idles that can be seen from the monolithic chips. I was hoping more in the range of 25-30W.
Some other misc oddities/annoyances:
Owing to the fiasco surrounding my failure to acquire a Supermicro H13SAE-MF, I'm starting my build a year late on an ASRockRack B650D4U. I'd written this board off originally owing to what sounded like a bevvy of hardware failures - seems this was finally corrected on newer hardware revisions a couple of years back. Mine is a rev. 4.01 board with an M80 serial.
Anyway, I haven't seen much in the way of people reporting on board power and ASPM issues so I figured it'd be useful to have it down somewhere, even if I'm catastrophically late to the AM5 server party.
Overall the board seems perfectly happy with turning on ASPM everywhere with the exception of the USB controller which resists all attempts to enable ASPM despite the fact it's listed as supporting L0 and L1; I assume this might a kernel quirk (I've tried the default trixie 6.12 and the trixie-backports 7.1.3) but haven't dug in to this yet; only found this forlorn post on stack exchange from someone with the same issue (also a debian user).
Code:
root@testwug:~# lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM |Disabled;)'
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge (prog-if 00 [Normal decode])
LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <64us
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge (prog-if 00 [Normal decode])
LnkCap: Port #0, Speed 32GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge (prog-if 00 [Normal decode])
LnkCap: Port #1, Speed 32GT/s, Width x4, ASPM L1, Exit Latency L1 <32us
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge (prog-if 00 [Normal decode])
LnkCap: Port #2, Speed 32GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A] (prog-if 00 [Normal decode])
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A] (prog-if 00 [Normal decode])
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
01:00.0 Serial Attached SCSI controller: Broadcom / LSI Fusion-MPT 12GSAS/PCIe Secure SAS38xx
LnkCap: Port #0, Speed 16GT/s, Width x8, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
02:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <16us
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
02:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <16us
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <32us
LnkCtl: ASPM L1 Enabled; LnkDisable- CommClk+
04:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #1, Speed 16GT/s, Width x1, ASPM L1, Exit Latency L1 <32us
LnkCtl: ASPM L1 Enabled; LnkDisable- CommClk+
04:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #2, Speed 16GT/s, Width x1, ASPM L1, Exit Latency L1 <32us
LnkCtl: ASPM L1 Enabled; LnkDisable- CommClk+
04:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #3, Speed 16GT/s, Width x1, ASPM L1, Exit Latency L1 <32us
LnkCtl: ASPM L1 Enabled; LnkDisable- CommClk+
04:0c.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #12, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
LnkCtl: ASPM L1 Enabled; LnkDisable- CommClk+
04:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #13, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
LnkCtl: ASPM L1 Enabled; LnkDisable- CommClk+
05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
LnkCap: Port #2, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
07:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 06) (prog-if 00 [Normal decode])
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <32us
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
09:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset USB 3.2 Controller (rev 01) (prog-if 30 [XHCI])
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
0a:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller (rev 01) (prog-if 01 [AHCI 1.0])
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
0b:00.0 Non-Volatile memory controller: Micron/Crucial Technology T500 NVMe PCIe SSD (rev 01) (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
0c:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Granite Ridge [Radeon Graphics] (rev d1) (prog-if 00 [VGA controller])
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
0c:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 19h PSP/CCP
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
0c:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI (prog-if 30 [XHCI])
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
0c:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI (prog-if 30 [XHCI])
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
0c:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] Audio Coprocessor (rev 62)
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
0c:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h/19h/1ah HD Audio Controller (prog-if 00 [HDA compatible])
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
0d:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 2.0 xHCI (prog-if 30 [XHCI])
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+The above is a fairly barebones layout (basic fresh trixie install, no drives attached other than the boot M2, 10Gb SFPs not plugged in). Complete rundown of the hardware in this test config are:
- B650D4U
- Epyc 4545P
- Noctua NH-U9S HSF (thanks to Noctua support for sending me the AMB14 mounting kit for free, required if you want a desktop-style cooler on a server-style front-to-back 90deg rotated socket)
- 2 x Kingston KSM56E46BD8KM-32H 32GB ECC UDIMMs
- 1 x broadcom 9500-8i HBA
- 1 x intel X710-DA2 NIC
- 1 x crucial T500 M.2 boot SSD
- 1 x crappy 2U PSU I had lying around (no SMBus)
Power stats with the above are as follows (plus a few extra measurements showing the affect the PCI cards have), rounding up:
- BMC only, powered off: ~4.3W
- POST: 75-92W
- OS idle, 1xT500 SSD: 42W
- OS idle, 1xT500, 9500-8i: 47W
- OS idle, 1xT500, 9500-8i, X710-DA2: 49.5W
- OS idle + powertop auto tune, 1xT500, 9500-8i, X710-DA2: 49.5W
- OS idle + powertop auto tune + ASPM disabled, 1xT500, 9500-8i, X710-DA2: 57.5W
So whilst the ASPM idle savings aren't great they are there to a degree. TBH I'm a little disappointed the idle draw of the system's still in the 50W range, seems that's still in the realms of slight suckitude, esp. given the low idles that can be seen from the monolithic chips. I was hoping more in the range of 25-30W.
Some other misc oddities/annoyances:
- The DIMMs are rated for 5600MT/s (which the board and CPU ostensibly support) but both run at 5200MT/s for some reason, visible both in the BMC inventory page and via dmidecode; would have hoped/expected 5600 would be sustainable with two dual-rank DIMMs in a 1DpC config:
Code:Handle 0x0021, DMI type 17, 100 bytes Memory Device Array Handle: 0x0017 Error Information Handle: 0x0020 Total Width: 72 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: DIMM 1 Bank Locator: P0 CHANNEL A Type: DDR5 Type Detail: Synchronous Unbuffered (Unregistered) Speed: 5600 MT/s Manufacturer: Kingston Serial Number: Asset Tag: Not Specified Part Number: 9965794-022.A00G Rank: 2 Configured Memory Speed: 5200 MT/s Minimum Voltage: 1.1 V Maximum Voltage: 1.1 V Configured Voltage: 1.1 V Memory Technology: DRAM Memory Operating Mode Capability: Volatile memory Firmware Version: Unknown Module Manufacturer ID: Bank 2, Hex 0x98 Module Product ID: Unknown Memory Subsystem Controller Manufacturer ID: Unknown Memory Subsystem Controller Product ID: Unknown Non-Volatile Size: None Volatile Size: 32 GB Cache Size: None Logical Size: None PMIC0 Manufacturer ID: Bank 11, Hex 0x8C PMIC0 Device Type: Not Installed RCD Manufacturer ID: Unknown RCD Revision Number: Unknown - Adding in the PCIe cards repeatedly changed the device names of the onboard i210 NICs, shunting them both up by 1 each time I added in a new card and rendering me with an offline server. I thought the wholly annoying "let's do away with ethX and have static device names based on <something vague and ill-defined that I've never really understood> was meant to do away with this sort of stupidity?
- Turning off NIC bonding for the IPMI NIC was much more of a faff than it needed to be, and probably would be even more so if I didn't know the magic IPMI raws for turning it back on/off again.
- Even after you've turned off IPMI NIC bonding and successfully stood up a dedicated IPMI IP/VLAN... the NCSI IPMI interface will still happily bond to your regular network via the i210 and also make the IPMI interface visible there again too - so, something to be aware of if you're expecting IPMI to only be accessible on the IPMI NIC. Turning off the NCSI sharing means going in to the BIOS and expressly disabling the BMC's eth1 interface (seemingly no magic I can find in the IPMI interface to do this).
- The BMC inventory doesn't seem to have updated in the last 20 or so reboots; it's still reading the micron 7300 pro I initially had plugged in to do a firmware update (but removed when it became apparent it didn't support ASPM/APST at all). Neither has it picked up the addition of the 9500-8i or the X710-DA2. I'm not aware of any option to force a refresh.
- ASRR have seemingly done a great job with the fan control vs. the previous AST2400 BMC firmware I'm used to, as well as compared to the fustercluck I was preparing to face with the H13SAE-MF; GUI for a multitude of custom fan curves, configurable via any or all of the IPMI temperature sensors (manual control also an option but haven't yet started testing the raws).
Maybe these points are connected - depending on the PSU wattage and it's low power efficiency, you could save a lot of power. I had a system go from ~80W on redundant supermicro 920W PSUs to ~50W on a Corsair SFF Platinum 450W.
It's possible - I'll know better once it's off the test bench and in its "production" spot, I just don't have any spare 80+ gold/platinum/titanium PSUs lying around to try it with.
PSU in question is a TGC SG2U500 80+ bronze that, if memory serves me correctly, came bundled with an inexpensive Logic Case rackmount. But not colossally overpowered or inefficient.
well, 80+ bronze starts at 20% load, so your PSU has 85% efficiency at 100W load. Below that, who knows. The supermicro platinum 920W had atrocious efficiency at 50W load.
I've since learned that this appears to be an AGESA bug introduced in AGESA!V9 ComboAm5PI 1.3.0.0; Level1Techs apparently did a video about it but the summary from this post on HardOCP tells you everything you need to know I think. Seemingly it only affects ECC memory. Reddit thread here also - the chap in that thread is using the same memory modules as me, 2x32GB Kingston KSM56E46BD8KM-32HA.
This problem is apparently fixed in 1.3.0.1b according to a couple of people. Of course, the beta firmware from ASRR for the B640D4U is currently 1.3.0.1a :grimace: Might try a downgrade to confirm the memory can indeed train'n'sustain 5600.