Ryzen AM5 boards with IPMI

[rune-san]

I'm also on the hunt for a decent system with Ryzen 7000. Server systems at this point have honestly gone far beyond what I can use in my home, and in many cases, I can benefit more from clock speed than moar cores. Combined with gobs of PCIe 5.0 on newer server platforms, and the cost is just far beyond what I can justify for use. Supermicro has yet to release anything for AM5. ASRock Rack's board checks pretty much all of my boxes, which is enough room to put in an additional 10Gb NIC and an HBA. Unfortunately though the only Server Barebone ASUS offers with it is a 1U, so it only exposes one slot to a riser and the rest goes unused. Would also be great if the drive backplane supported SAS2/3 vs. SATA only.

Obviously there's always compromises, but it would be great if there was a 2U system that supported a Ryzen board with at least some degree of "it can be cabled together and work" without a huge fuss over wiring harness working, standoffs in the right places, and air shroud / cooling being functional.

 

[Tom S]

I'm also on the hunt for a decent system with Ryzen 7000. Server systems at this point have honestly gone far beyond what I can use in my home, and in many cases, I can benefit more from clock speed than moar cores.

The 16-core consumer parts combined with these affordable server boards have been a game changer for affordable servers.

The clock speed advantage of the consumer parts gives a huge performance boost relative to the slower clocked 16 core server parts. You have to get into the very high core counts and have a very parallelizable workload before the big server systems come out ahead.

There are a few exceptions. Workloads that require a lot of memory bandwidth will benefit greatly from the 4-channel and up server systems. If someone needs huge amounts of I/O, the server systems are your only option. However, the latest AM5 and W680 platforms have significant amounts of I/O, so that's really an edge case at this point.

I'm very curious to see how these AM5 boards will handle with 2 DIMMs per channel. The ASRock Rack board has very different DDR5 speed ratings for 1DPC and 2DPC configurations. I'd like to see someone get 4x32GB and try to push the DDR5 clocks. If I can get 128GB of DDR5 and maintain high clock speeds (4500+) I'd be very happy.

 

[unwind-protect]

I'm very curious to see how these AM5 boards will handle with 2 DIMMs per channel. The ASRock Rack board has very different DDR5 speed ratings for 1DPC and 2DPC configurations. I'd like to see someone get 4x32GB and try to push the DDR5 clocks. If I can get 128GB of DDR5 and maintain high clock speeds (4500+) I'd be very happy.

That is still the iffy thing about the ECC-enabled consumer pieces. Yes, they can take 128 GB of memory (and maybe 192 GB later), but there is this catch. Yes, they can have a couple of PCIe slots, but random onboard devices get turned off when you actually fill them. It doesn't help that they often have 2.5 Gbit/s ethernet, so for 10 Gb/s you need PCIe slots.

 

[Tom S]

That is still the iffy thing about the ECC-enabled consumer pieces. Yes, they can take 128 GB of memory (and maybe 192 GB later), but there is this catch. Yes, they can have a couple of PCIe slots, but random onboard devices get turned off when you actually fill them. It doesn't help that they often have 2.5 Gbit/s ethernet, so for 10 Gb/s you need PCIe slots.

The ASRock Rack boards have 10GBe onboard as an option, not 2.5G. They also have models without 10GBe if you’d prefer to have those PCIe lanes for your own use.

The PCIe lane allocations on these boards isn’t really a mystery. They’re clearly documented in the manuals. Gigabyte didn’t do a good job of exposing all of the lanes, but ASRock has been good about making them all available.

 

[tjk]

Guess we'll all be surprised.

Surprised yet?

Now someone needs to release a Ryzen AM5 with U2 NVME support. Asrockrack has a 2x U2 system that isn'y shipping yet, would be nice to see someone release a 4x U2 system.

 

[doppler_shift]

Now someone needs to release a Ryzen AM5 with U2 NVME support. Asrockrack has a 2x U2 system that isn'y shipping yet, would be nice to see someone release a 4x U2 system.

Assuming you're talking about this system then it doesn't natively support U.2; judging by the pictures they seem to be using one PCIe x4 to SFF-8643 riser card for the first U.2 and then an M.2 to U.2 adapter for the second U.2.

If you fancy a 4xU.2 system with one of their AM5 boards you could always use something like this.

 

[tjk]

Assuming you're talking about this system then it doesn't natively support U.2; judging by the pictures they seem to be using one PCIe x4 to SFF-8643 riser card for the first U.2 and then an M.2 to U.2 adapter for the second U.2.

If you fancy a 4xU.2 system with one of their AM5 boards you could always use something like this or this.

Custom building something right now using AOC-SLG4-4E4T | Add-on Cards | Accessories | Products - Super Micro Computer, Inc.

 

[tjk]

Also, Hetzner is custom building their AX102 server offering on an Asrockrack system and offering up to 6 x U2 NVME, which sounds over-subscribed to me.

 

[vincococka]

Custom building something right now using AOC-SLG4-4E4T | Add-on Cards | Accessories | Products - Super Micro Computer, Inc.

Be careful with AOC-SLG4-4E4T / AOC-SLG4-2E4T (single connector) - they need to have proper BIOS support regarding TX Preset setting.
I bought AOC-SLG4-2E4T and wanted to use it on X12SPL-F, and it's BIOS does not have visible option to configure "DN Tx Preset" to "P7".
See user manual on page 17 (section 4-3). So now my 2 pieces of AOC-SLG4-2E4T are laying on shelf and collecting dust .

I found followint to work flawlessly and hassle-free - SlimSAS to PCI-E made by C-Payne - take a look specifically at these:
- PCIe SlimSAS Host Adapter x8 to 8i - straight
- PCIe SlimSAS Host Adapter x16 to 2* 8i - straight

+ cable Supermicro CBL-SAST-0953 which connects two U.2 drives.

This setup works reliably and keeps holding speeds @PCIe 4.0 (not falling back to PCIe 3.0 like other chinese NVME hba's or cables).

 

[tjk]

Be careful with AOC-SLG4-4E4T / AOC-SLG4-2E4T (single connector) - they need to have proper BIOS support regarding TX Preset setting.
I bought AOC-SLG4-2E4T and wanted to use it on X12SPL-F, and it's BIOS does not have visible option to configure "DN Tx Preset" to "P7".
See user manual on page 17 (section 4-3). So now my 2 pieces of AOC-SLG4-2E4T are laying on shelf and collecting dust .

I found followint to work flawlessly and hassle-free - SlimSAS to PCI-E made by C-Payne - take a look specifically at these:
- PCIe SlimSAS Host Adapter x8 to 8i - straight
- PCIe SlimSAS Host Adapter x16 to 2* 8i - straight

+ cable Supermicro CBL-SAST-0953 which connects two U.2 drives.

This setup works reliably and keeps holding speeds @PCIe 4.0 (not falling back to PCIe 3.0 like other chinese NVME hba's or cables).

Mind if I PM you to discuss further?

 

[vincococka]

Mind if I PM you to discuss further?

Feel free - if you have skype search for identical name there, or we can discuss here via PM

 

[Tom S]

ASRock Rack has another AM5 board on their website: https://www.asrockrack.com/general/productdetail.asp?Model=AM5D4ID-2T/BCM#Specifications

This one is Deep mini-ITX, but retains all 4 DIMM slots. Single M.2 slot, Single 5.0x16 PCIe slot, and an extra OCuLink port for expansion.

This board is especially interesting because it doesn't have a external chipset. It uses the AMD Knoll activator to get the CPU running without a full chipset.

I'm intrigued. This could be a great board for low power systems. Having a 7950X with 2x48GB DIMMs in a small form factor case would make a great build server for me.

 

[Y0s]

Is there a reliable source (within the US) for ASRock Rack motherboards (or other IPMI boards such as Gigabyte or Tyan)? There are dubious 3rd party sellers on Amazon or Newegg with high mark-up

 

[doppler_shift]

Looks like there's a new contender in this niche from Supermicro.

 

[tjk]

I've got 6 units of the AM5 Microcloud on order, hope to have them early Sept, can't wait!

Asrockrack has a couple AM5 barebones with nvme (finally) that look interesting:

https://www.asrockrack.com/general/productdetail.asp?Model=4U18N-AM5/2T#Specifications

https://www.asrockrack.com/general/productdetail.asp?Model=1U4L2E-B650 RPSU#Specifications (will order a few of these when theyare shipping to test)

 

[Tom S]

Looks like there's a new contender in this niche from Supermicro.

Good to see more competition.

I like the PCIe configuration of the SuperMicro board slightly better: Both M.2 slots get PCIe Gen 5 straight from the CPU, whereas the second M.2 slot on the ASRock board is Gen 4 and connected through the chipset.

The SuperMicro board bifurcates the Gen 5 x16 bus if you use both of the big slots, but that's fine for most applications.

 

[Ivan Dimitrov]

Looks like there's a new contender in this niche from Supermicro.

It is good that SuperMicro finally starts seeing the light. But it is still annoying that the AMD boards have a "second class citizen" feel.
There is conflict between the claimed supported memory in the manual and the page (128 vs. 192 GB), the block diagram looks worse, etc.
But otherwise this can easily replace a 3647 Xeon from a few years back.

 

[twin_savage]

I just got my hands on a H13SAE-MF for a 32-bay NAS build if there's anything someone wanted to know about it.

 

[Tom S]

I've got an ASRock B650D4U here with a 7950X3D and some 48GB DIMMs.

It booted right up with the factory BIOS, to my surprise. I updated to the new BIOS on the website that explicitly the X3D chips and 48GB DIMMs anyway.

The only problem is that I can't get the dedicated IPMI port to work. The activity light flashes but it won't get an IP. If I move the cable over to the other ethernet ports and let it go through that interface it's fine. Only the dedicated port fails to work.

I guess it's my turn to go through ASRock's support system. Hopefully I can get this resolved quickly.

 

[Ivan Dimitrov]

I just got my hands on a H13SAE-MF for a 32-bay NAS build if there's anything someone wanted to know about it.

As you asked here is my wish list :

1. What is the supported bifurcation options for the the x8 slots? Can I go 8x/4x/4x?
2. What is the PCIe lane separation situation? Are all the lanes in separate groups and/or which are combine?
3. Does it really support 48GB DIMMS?
4. Does it supports 192GB of RAM?