Ryzen AM5 boards with IPMI

[twin_savage]

As you asked here is my wish list :

1. What is the supported bifurcation options for the the x8 slots? Can I go 8x/4x/4x?
2. What is the PCIe lane separation situation? Are all the lanes in separate groups and/or which are combine?
3. Does it really support 48GB DIMMS?
4. Does it supports 192GB of RAM?

1. no bifurcation support in current BIOS (or at the very least it is hidden/no menu for it yet)
2.
3. it does support 48GB DIMMs out of the box... sadly 48GB ECC DIMMs aren't very available yet to test though
4.I assume it does (currently running 96GB), but their doesn't seem to be any way to force memory speeds so it will run at only 3600MHz with 4 DIMMs

 

[Ivan Dimitrov]

If you have linux installed on the system could you also check the IOMMU group with this:

#!/bin/bash
for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo "IOMMU group $(basename "$iommu_group")"; for device in $(ls -1 "$iommu_group"/devices/); do echo -n $'\t'; lspci -nns "$device"; done; done

 

[fortezza]

Asked Supermicro and got this reply.

Dear Jason,

Thank you for contacting Super Micro Technical Support!

According to the specification of motherboard, it doesn’t support PCIe bifurcation. Thanks a lot!

Have a great day!

Best Regards,
Kin Yan/Application Engineer
Technical Support Department
*For any urgent issue, please call +1-(408) 503-8000 option 2 to talk to a tech support live person.*
We welcome and appreciate your feedback: Contact | Supermicro

 

[mmaenpaa]

I am building system with:

B650D4U
7950X
4 X KSM48E40BD8KM-32HM (=128GB)
1 X KINGSTON DC1000B 960GB M.2 2280
ASUS HYPER M.2 X16 GEN 4 CARD
4 X SAMSUNG PM9A3 PCIe4.0x4 M.2 1.92TB or 3.84TB
Mellanox 10GB SFP+ PCIE4X

I am really tempted to order 2 pcs of these Axiom 48GB ECC UDIMMs for testing (about 250-350$ per module + taxes + shipment).
So far these are only 48GB ECC ones I have been able to find on online shops. But then again if those do not work...

AX54800E40I/48G
AX55600E46I/48G

 

[unwind-protect]

I am building system with:

B650D4U
7950X
4 X KSM48E40BD8KM-32HM (=128GB)
1 X KINGSTON DC1000B 960GB M.2 2280
ASUS HYPER M.2 X16 GEN 4 CARD
4 X SAMSUNG PM9A3 PCIe4.0x4 M.2 1.92TB or 3.84TB
Mellanox 10GB SFP+ PCIE4X

I am really tempted to order 2 pcs of these Axiom 48GB ECC UDIMMs for testing (about 250-350$ per module + taxes + shipment).
So far these are only 48GB ECC ones I have been able to find on online shops. But then again if those do not work...

AX54800E40I/48G
AX55600E46I/48G

You would be the first person I am aware of to actually go 192 GB on AM5 or LGA1700.

 

[sadamahan]

I've got an ASRock B650D4U here with a 7950X3D and some 48GB DIMMs.

It booted right up with the factory BIOS, to my surprise. I updated to the new BIOS on the website that explicitly the X3D chips and 48GB DIMMs anyway.

The only problem is that I can't get the dedicated IPMI port to work. The activity light flashes but it won't get an IP. If I move the cable over to the other ethernet ports and let it go through that interface it's fine. Only the dedicated port fails to work.

I guess it's my turn to go through ASRock's support system. Hopefully I can get this resolved quickly.

Which 48GB DIMMs did you use with B650D4U? I'm not able to find any. QVL has: SR6G7UD5385MB01 but can't find that anywhere.
Thanks!

 

[Tom S]

Which 48GB DIMMs did you use with B650D4U? I'm not able to find any. QVL has: SR6G7UD5385MB01 but can't find that anywhere.
Thanks!

There were't any 48GB ECC DIMMs available when I built the system, so I bought a temporary non-ECC set.

I bought a G.SKILL 2x48GB kit on NewEgg that was cheap. I ran memtest86 for about a week without problems.

I'll get a proper 96GB ECC set once things settle down, but this works for now. This isn't a mission critical server for me so I'm not worried.

 

[MrBios]

There were't any 48GB ECC DIMMs available when I built the system, so I bought a temporary non-ECC set.

I bought a G.SKILL 2x48GB kit on NewEgg that was cheap. I ran memtest86 for about a week without problems.

I'll get a proper 96GB ECC set once things settle down, but this works for now. This isn't a mission critical server for me so I'm not worried.

would this work with B650D4U ?

G.SKILL Ripjaws S5 Series 96GB (2 x 48GB) 288-Pin PC RAM DDR5 5600 (PC5 44800) Desktop Memory Model F5-5600J4040D48GX2-RS5K - Newegg.com

Buy G.SKILL Ripjaws S5 Series 96GB (2 x 48GB) 288-Pin PC RAM DDR5 5600 (PC5 44800) Desktop Memory Model F5-5600J4040D48GX2-RS5K with fast shipping and top-rated customer service. Once you know, you Newegg!

www.newegg.com

or this

G.SKILL Flare X5 96GB (2 x 48GB) 288-Pin PC RAM DDR5 5600 (PC5 44800) Desktop Memory Model F5-5600J4040D48GX2-FX5 - Newegg.com

Buy G.SKILL Flare X5 96GB (2 x 48GB) 288-Pin PC RAM DDR5 5600 (PC5 44800) Desktop Memory Model F5-5600J4040D48GX2-FX5 with fast shipping and top-rated customer service. Once you know, you Newegg!

www.newegg.com

whats the difference between two cause newegg doesnt list one of them as compatible with ryzen ?

 

[Tom S]

One has XMP 3.0 and the other has AMD EXPO. Both are for extended memory profiles for overclocking.

I got the Intel version and it works fine because I'm not overclocking on this board.

 

[sadamahan]

One has XMP 3.0 and the other has AMD EXPO. Both are for extended memory profiles for overclocking.

I got the Intel version and it works fine because I'm not overclocking on this board.

Curios to know what the Intel version of this board is. What CPU did you pair it with? I am debating between B650D4U+7950X and the intel platform.

 

[Tom S]

Curios to know what the Intel version of this board is. What CPU did you pair it with? I am debating between B650D4U+7950X and the intel platform.

Sorry, I got the Intel version of the memory linked above. I use it in my B650D4U board.

It's the same memory, just different overclocking profiles on the DIMM. Since I'm not overclocking or using those extended timing profiles, the Intel vs AMD branding on the memory doesn't actually matter.

 

[RDshift]

I'm refining my wish list right now, and trying to talk myself into waiting for Ryzen next-gen.

• 2U - easier expansion (and occasionally, Dremeling)
• 2U - 80MM fans are quieter and perhaps more energy efficient
• Need to fit a next-gen Intel GPU for AI+video tasks. HHFL is usually available, and lower end cards often have the same codec capabilities with only mild slowdowns. At today's Intel GPU prices, I'll fit a current-gen Intel GPU as well, for now <3
• 2 SATA 3.0 minimum - not a problem, everybody's got it.
  • 1 minimum, 2 preferred. For general storage 8TB and above.
• 1 or 2 U.2 or U.3 highly desirable. ASRock does this with a low format card to a backplane, as seen here between the CPU and the GPU riser - fine with me: https://www.asrockrack.com/photo/1U4L2E-B650 RPSU-5(L).jpg(with that many heat pipes on that monstrosity, they should just have a vapor chamber [edit: I think that was an airflow/heat dissipation decision to ensure the server could handle more extreme environments without difficulty])
  • +1 Optane Drive of fastness and durability
  • Future expansion capabilities.
• 10 Gig - but I want SFPs, so I'm happy with a PCIe x4 slot. Happier, actually. Adding 10G tends to delete a PCIe slot, which means less flexibility for me. I want an SFP+ 10G or SFP28 25G card because ServeTheHome.
• 1+ M.2 NVMe - Likely boot drive, if the Optane drive isn't boot. YOU get a heatsink, and YOU get a heatsink.
• Open to non-traditional motherboard layouts - just get me what I want.
• Definitely prefer higher end chipsets for expansion, though server w/ryzen definitely leans towards mid-grade for legit reasons that improve cost efficiency and make companies money
  • I prefer Supermicro in the server space, but ASRock Rack is going to be more practical, and likely offer a higher end chipset (see today they offer B650E on 3 standard format boards vs Supermicro offering with B650. !IMHO! both are better quality than Gigabyte, who I have objections to due to a personal history where Gigabyte didn't release BIOS updates for my stuff grrrr. They broke my trust.

 

[mmaenpaa]

Curios to know what the Intel version of this board is. What CPU did you pair it with? I am debating between B650D4U+7950X and the intel platform.

I also wanted to try 48GB ECC memory but due to time constraints I had to settle for 32GB.

Setup so far:

B650D4U
AMD Ryzen 9 7950X BOX AM5 16C/32T 170W
Noctua NH-U12A
4 X KSM48E40BD8KM-32HM
OS : Window Server 2022

I have been testing with Karhu Software RAM Test for 22 hours. CPU stays at 5.3Ghz and Noctua is very quiet. Fan profiles were a bit tricky to understand at first. Latest bios & BMC (2.17 & 02.08.00) did not use any "quiet" profiles, one has to apply it from the BMC setup.

Good clearance on memory DIMMs and Noctua. Screenshot is after installing the second fan (It did not make any change on CPU speed)




Still waiting for these:

OS drive
- KINGSTON DC1000B 960GB M.2 2280 Ent SSD (SEDC1000BM8/960G)
Storage
- ASUS HYPER M.2 X16 CARD Gen 4 (90MC08A0-M0EAY0)
- 4 pieces of SAMSUNG PM9A3 PCIe4.0x4 M.2 3.84TB (MZ1L23T8HBLA-00A07)
Network
Mellanox ConnectX-3 MCX311A-XCAT CX311A 10GBe 10GB SFP+

 

[lostmind]

Asked Supermicro and got this reply.

KY is answering my support questions about this board as well.

Had an issue with onboard ethernet ports getting PCIe link lost, Reset adapter messages... and the server going offline.

I tried a10g intel x710 adapter, wouldn't boot up past the "dxe—superio initialization 99" portion.

Couldn't even get into bios with the card in.

Swapped for an old connectx3. Same issue.

Threw in a dual port supermicro 1gbps card (AOC-SGP-i2 I believe). it booted, but linux only saw 1 of 2 ports and not matter what I did I couldn't get it to work. Messages log said it had renamed usb0 to enp12s0u10u2c2... ethtool said it saw the port was connected but port speed and everything else was unknown. Tried to force the speed etc but no luck.

Updated BIOS from 1.1 to 1.1a. No luck with the addon cards. Went back to the onboard ports overnight hoping they would work with the update. Woke up to the same PCIe link lost Reset adapter messages. Today the server was sitting idle and network has completely died.

Messaged supermicro and KY says to use a supported OS (Alma, rocky, rhel should all be pretty damned close and apparently choosing alma was the wrong choice). Says to try to use the intel linux driver directly. Said to only use approved parts from the AOC lists... uhm, one was... but ok.

Not impressed... and I had high hopes after asrockrack support was garbage... the supermicro board seems much better laid out.

 

[twin_savage]

If you have linux installed on the system could you also check the IOMMU group with this:

#!/bin/bash
for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo "IOMMU group $(basename "$iommu_group")"; for device in $(ls -1 "$iommu_group"/devices/); do echo -n $'\t'; lspci -nns "$device"; done; done

Here's the output:

IOMMU group 17
        05:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        09:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 06)
        0a:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 52)
IOMMU group 7
        00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU group 25
        0f:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b7]
IOMMU group 15
        05:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        07:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU group 5
        00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU group 23
        0f:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]
IOMMU group 13
        04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f4] (rev 01)
IOMMU group 3
        00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU group 21
        0f:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c4)
IOMMU group 11
        00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e0]
        00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e1]
        00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e2]
        00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e3]
        00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e4]
        00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e5]
        00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e6]
        00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e7]
IOMMU group 1
        00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU group 28
        10:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
IOMMU group 18
        05:0b.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
IOMMU group 8
        00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
IOMMU group 26
        0f:00.5 Multimedia controller [0480]: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor [1022:15e2] (rev 62)
IOMMU group 16
        05:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        08:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU group 6
        00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU group 24
        0f:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b6]
IOMMU group 14
        05:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
IOMMU group 4
        00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU group 22
        0f:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
IOMMU group 12
        02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [144d:a80a]
IOMMU group 2
        00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU group 20
        05:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0d:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f6] (rev 01)
IOMMU group 10
        00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
        00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 0
        00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU group 19
        05:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0c:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f7] (rev 01)
IOMMU group 9
        00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
IOMMU group 27
        0f:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]

I finally got ECC memory installed in it as well and it appears to be properly handling it in linux:

Handle 0x0036, DMI type 17, 92 bytes
Memory Device
        Array Handle: 0x0032
        Error Information Handle: No Error
        Total Width: 72 bits
        Data Width: 64 bits
        Size: 48 GB
        Form Factor: DIMM
        Set: None
        Locator: DIMMA2
        Bank Locator: P0_Node0_Channel0_Dimm1
        Type: DDR5
        Type Detail: Synchronous Unbuffered (Unregistered)
        Speed: 5600 MT/s
        Manufacturer: Unknown
        Serial Number: 00000000
        Asset Tag: DIMMA2_AssetTag (date:00/00)
        Part Number: AVW726GUA1K6560P07-H
        Rank: 2
        Configured Memory Speed: 5200 MT/s
        Minimum Voltage: 1.1 V
        Maximum Voltage: 1.1 V
        Configured Voltage: 1.1 V
        Memory Technology: DRAM
        Memory Operating Mode Capability: Volatile memory
        Firmware Version: Unknown
        Module Manufacturer ID: Bank 6, Hex 0xF7
        Module Product ID: Unknown
        Memory Subsystem Controller Manufacturer ID: Unknown
        Memory Subsystem Controller Product ID: Unknown
        Non-Volatile Size: None
        Volatile Size: 48 GB
        Cache Size: None
        Logical Size: None

 

[diskdiddler]

ASRock Rack has another AM5 board on their website: https://www.asrockrack.com/general/productdetail.asp?Model=AM5D4ID-2T/BCM#Specifications

This one is Deep mini-ITX, but retains all 4 DIMM slots. Single M.2 slot, Single 5.0x16 PCIe slot, and an extra OCuLink port for expansion.

This board is especially interesting because it doesn't have a external chipset. It uses the AMD Knoll activator to get the CPU running without a full chipset.

I'm intrigued. This could be a great board for low power systems. Having a 7950X with 2x48GB DIMMs in a small form factor case would make a great build server for me.

Has anyone actually purchased and used this board? What are the implications of the chipset on it?
I am wanting a particularly fast workstation machine, which I remote desktop to, all the time, in an ITX form factor, this appears to be close to what I need.

(I like IPMI a lot so I can remotely fix problems if things crash)

 

[lostmind]

KY is answering my support questions about this board as well.

Had an issue with onboard ethernet ports getting PCIe link lost, Reset adapter messages... and the server going offline.

I tried a10g intel x710 adapter, wouldn't boot up past the "dxe—superio initialization 99" portion.

Couldn't even get into bios with the card in.

Swapped for an old connectx3. Same issue.

Threw in a dual port supermicro 1gbps card (AOC-SGP-i2 I believe). it booted, but linux only saw 1 of 2 ports and not matter what I did I couldn't get it to work. Messages log said it had renamed usb0 to enp12s0u10u2c2... ethtool said it saw the port was connected but port speed and everything else was unknown. Tried to force the speed etc but no luck.

Updated BIOS from 1.1 to 1.1a. No luck with the addon cards. Went back to the onboard ports overnight hoping they would work with the update. Woke up to the same PCIe link lost Reset adapter messages. Today the server was sitting idle and network has completely died.

Messaged supermicro and KY says to use a supported OS (Alma, rocky, rhel should all be pretty damned close and apparently choosing alma was the wrong choice). Says to try to use the intel linux driver directly. Said to only use approved parts from the AOC lists... uhm, one was... but ok.

Not impressed... and I had high hopes after asrockrack support was garbage... the supermicro board seems much better laid out.

Just a heads up for anyone else with this board. Supermicro suggested we ship them the full system to troubleshoot. I declined. Figured it was possibly a bad motherboard. Ordered a second and swapped the boards.

Same issue.

At this point I am unsure what the issue is. Going to head back to the dc, try removing the 2 x m.2 nvme and install on a plain sata SSD, see if that resolves things.

 

[lostmind]

Just a heads up for anyone else with this board. Supermicro suggested we ship them the full system to troubleshoot. I declined. Figured it was possibly a bad motherboard. Ordered a second and swapped the boards.

Same issue.

At this point I am unsure what the issue is. Going to head back to the dc, try removing the 2 x m.2 nvme and install on a plain sata SSD, see if that resolves things.

Troubles with the board are still ongoing.

While on the racks, tried replacing the psu, tried a new motherboard, ram tests fine in memtest, tried a single sata to install onto, tried without a riser... all resulted in the same error "PCIe link lost Reset adapter" after a few hours of running fine.

Took the server back to the office to troubleshoot. And.. it works fine in any config for well over 24 hours.

Took it back downtown, put it on the rack, within 4 hours, "PCIe link lost Reset adapter" error again.

When this error occurs, the network goes completely offline, so it is not ideal for a server.

Only thing I can think of now, is that the power on the racks is 208v and the power in the office is 120v.

Tried two different power supplies. Same issue. KY @ Supermicro has upgraded my support request to a system PM.

They've setup a test barebones in their office. Hopefully they encounter the same issue and can provide a fix. Otherwise, I guess I just got a pair of motherboards that were bad?

 

[masimus]

Troubles with the board are still ongoing.

I bought the same board (Supermicro H13SAE-MF) recently. Wished I had seen your comments first.. I have the same problem with NIC lost after a couple of hours of use.

Jan 20 07:08:02 server kernel: igb 0000:07:00.0 eno1: PCIe link lost
Jan 20 07:08:02 server kernel: ------------[ cut here ]------------
Jan 20 07:08:02 server kernel: igb: Failed to read reg 0xc030!
...

As per this reddit thread:
https://www.reddit.com/r/debian/comments/18bcdvy ,
I have made it working stable by adding the pcie_port_pm=off pcie_aspm.policy=performance arguments to linux kernel.

However, it comes with a big caveat. Since all PCIe power management is affected, it makes my system, with 4x m.2 nvme drives, consume 12-15W more and drives also get 7-10 degrees C hotter - when idling... So not optimal.

 

[lostmind]

yah we have been testing this with the grub conf updated with those lines. It has made it 48 hours without issues so far. What is your solution for fitting 4 x m.2 drives on this board?