And so it begins... First AMD Ryzen AM4 server motherboard.

dexvx

Member
Mar 6, 2014
43
4
8
The main issue right now is the price of DDR4 UDIMM/ECC.

I'm going back and forth between the X470D4 or an used Asus X99WS/IPMI. The fact the latter takes DDR4 RDIMM/ECC tremendously helps the price, especially as you scale up in memory.
 

zir_blazer

Active Member
Dec 5, 2016
333
117
43
The main issue right now is the price of DDR4 UDIMM/ECC.

I'm going back and forth between the X470D4 or an used Asus X99WS/IPMI. The fact the latter takes DDR4 RDIMM/ECC tremendously helps the price, especially as you scale up in memory.
This is, again, one of the things where a similarly geared EPYC Embedded would be good at.

A Motherboard close to the X470D4U but based on that instead of standard AM4 could be a prosumer monster.
 

dexvx

Member
Mar 6, 2014
43
4
8
The EPYCD8 Epyc board is only about $100 more expensive than the X470D4U, but then you'd have to deal with the super low quantity and selection of Epyc CPU's.

I wonder how a Broadwell Xeon E5 compares against an Epyc.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,395
509
113
BIOS v3.04 has just been posted (along with v3.02 for the X470D4U2-2T) listing 7nm Ryzen support, so that pretty much confirms current BIOS won't work with Ryzen2 off the bat*... so, assuming benches and power levels are all I'm hoping for, likely a couple of months before I get myself one of these... so there's still time for Supermicro to get my money by announcing a mATX Epyc 3000 ;)

Still though... for less than £500 I'm looking at getting an 8C/16T IPMI-enabled home server platform... even with the high cost of ECC UDIMMs it's still a bargain (and I doubt an Epyc 3000 would be that affordable either).

* Many X470 desktop boards are flashable without a CPU installed but I've no idea if that's possible on this board, the manual certainly didn't mention it and I don't know if it's possible through IPMI-sans-CPU either.
 

ReturnedSword

Active Member
Jun 15, 2018
526
226
43
Santa Monica, CA
I had put together a parts list this weekend, but then realized that the 32GB DIMMs were RDIMM not UDIMM :(

I haven't seen Micron announce any 32GB ECC UDIMMs yet. Samsung announced two, with the M391A4G43MB1-CTD 2666 MHz variant being the only one listed in the US, however ofc it's OOS.

I'd love to use this as a compute server, or even as a FreeNAS build seeing as all the Intel vulnerabilities being quite a turn off. The Threadripper mobo seems nice too, however I like to separate my compute and storage servers.

One very minor annoyance for Ryzen is the non-APU CPUs don't have an iGPU for transcodes. A Ryzen APU for a server is a no-go for me since it only exposes a x8 link rather than a x16 link from the CPU itself.
 

Ojref1

New Member
Oct 8, 2018
19
5
3
I wouldn't wait too long on Supermicro releasing an AM4 or a TR-SP4 socket workstation/server board. They don't seem to have any interest in AMD workstation/server stuff outside of EPYC.
 

Egbert

New Member
May 6, 2012
8
1
3
Many X470 desktop boards are flashable without a CPU installed but I've no idea if that's possible on this board, the manual certainly didn't mention it and I don't know if it's possible through IPMI-sans-CPU either.
I hadn't yet installed the CPU into the motherboard, so I gave this a try. The IPMI does have an option to flash the BIOS, which I was able to use to flash v3.04 without the CPU being installed. Interestingly, I did discover that the system can't actually power on without a CPU installed. With power connected, the motherboard's LEDs light up and I am able to access IPMI, but the fans don't spin and the Dr. Debug module doesn't show anything. I have no idea whether this is the case with all motherboards, as I've never not installed the CPU. ;)

So it looks like you'll be able to buy the board with a Ryzen 3000 CPU without any worries.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,395
509
113
Superb info, thank you! I assume you're going for a Ryzen 2000 build as opposed to waiting for the 3000 series?

It's also confirmed that the ASRock motherboards also don't need a bash one-liner £20 license to allow the BIOS flashing via IPMI ;)
 

Egbert

New Member
May 6, 2012
8
1
3
I went with a Ryzen 1600 as the $80 Micro Center price was too good. Even though the 3000 series is supposed to be quite a bit faster, it's not 2.5 times as fast. I may upgrade to a Ryzen 3600 in a few years when it drops to the same price.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,395
509
113
So it looks like you'll be able to buy the board with a Ryzen 3000 CPU without any worries.
I've not confirmed this with a rep from ASRockRack who tested it themselves and confirms the BIOS is flashable via BMC without a CPU installed.

They also pointed me at their own stab at a centralised system management tool here, available as a linux/windows electron package; apparently BIOS flashes will work through this too.
 

ReturnedSword

Active Member
Jun 15, 2018
526
226
43
Santa Monica, CA
I had a conversation with ASRock the other day and they confirmed they will not be making a mITX X470 board with IPMI.

X470-ITX.png

However, they confirmed that a mITX X570 board with onboard dual 10GBASE-T using the Intel X550-AT2 will be available sometime 2019Q4 or later. This is very exciting!

X570D4I-2T.png
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,395
509
113
So just to confirm the prior posts, ASRock support and my own optimism... my motherboard arrived today, as expected with the 1.50 BIOS. Plugged in the IPMI port, scoured the DHCP logs to find the IP address, logged in, upgraded BIOS to v3.04 (no license hack needed, yay!).

bios_upgraded.png

Installed the CPU and memory and we're able to go straight in.

bios_cpuinfo.png

Initial thoughts:
  • Lots of swishy animations in the IPMI GUI that I'm really not a fan of, and the GUI seems to be trying to cram everything into a phone-style display that makes no use of horizontal space, but functionality-wise it's an improvement on their previous offerings. I think I might even prefer their HTML5 GUI to Supermicro's.
  • Many of the onboard sensors don't appear to work in the IPMI interface yet (most importantly CPU and memory temperatures)
    Code:
    root@frogstar:~# ipmitool sensor
    3VSB             | 3.380      | Volts      | ok    | 2.800     | 2.980     | na        | na        | 3.620     | 3.780
    5VSB             | 5.010      | Volts      | ok    | 4.260     | 4.500     | na        | na        | 5.490     | 5.730
    VCPU             | 0.190      | Volts      | ok    | na        | na        | na        | na        | 1.650     | 1.730
    VSOC             | 1.030      | Volts      | ok    | 0.340     | 0.360     | na        | na        | 1.540     | 1.610
    VCCM             | 1.190      | Volts      | ok    | 1.020     | 1.080     | na        | na        | 1.320     | 1.380
    APU_VDDP         | 0.950      | Volts      | ok    | 0.770     | 0.810     | na        | na        | 1.160     | 1.210
    3V               | 3.360      | Volts      | ok    | 2.800     | 2.980     | na        | na        | 3.620     | 3.780
    5V               | 5.130      | Volts      | ok    | 4.260     | 4.500     | na        | na        | 5.490     | 5.730
    12V              | 12.100     | Volts      | ok    | 10.200    | 10.800    | na        | na        | 13.200    | 13.800
    MB Temp          | na         | degrees C  | na    | na        | na        | na        | 55.000    | na        | na
    Card side Temp   | na         | degrees C  | na    | na        | na        | na        | 68.000    | na        | na
    CPU Temp         | na         | degrees C  | na    | na        | na        | na        | 91.000    | na        | na
    DDR4_A2_Temp     | na         | degrees C  | na    | na        | na        | na        | 84.000    | 85.000    | na
    DDR4_A1_Temp     | na         | degrees C  | na    | na        | na        | na        | 84.000    | 85.000    | na
    DDR4_B2_Temp     | na         | degrees C  | na    | na        | na        | na        | 84.000    | 85.000    | na
    DDR4_B1_Temp     | na         | degrees C  | na    | na        | na        | na        | 84.000    | 85.000    | na
    FAN1             | 1000.000   | RPM        | ok    | na        | na        | 100.000   | na        | na        | na
    FAN2             | na         | RPM        | na    | na        | na        | 100.000   | na        | na        | na
    FAN3             | na         | RPM        | na    | na        | na        | 100.000   | na        | na        | na
    FAN4             | na         | RPM        | na    | na        | na        | 100.000   | na        | na        | na
    FAN5             | na         | RPM        | na    | na        | na        | 100.000   | na        | na        | na
    FAN6             | na         | RPM        | na    | na        | na        | 100.000   | na        | na        | na
    ChassisIntr      | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na
    CPU_PROCHOT      | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na
    CPU_THERMTRIP    | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na
    PSU1 Status      | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na
    PSU1 AC lost     | na         | discrete   | na    | na        | na        | na        | na        | na        | na
    PSU2 Status      | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na
    PSU2 AC lost     | na         | discrete   | na    | na        | na        | na        | na        | na        | na
    1.05V_PROM_S5    | 1.060      | Volts      | ok    | 0.890     | 0.950     | na        | na        | 1.160     | 1.210
    2.5V_PROM        | 2.560      | Volts      | ok    | 2.120     | 2.260     | na        | na        | 2.740     | 2.880
    1.05V_PROM_RUN   | 1.010      | Volts      | ok    | 0.890     | 0.950     | na        | na        | 1.160     | 1.210
    BAT              | 3.000      | Volts      | ok    | 2.000     | 2.700     | na        | na        | 3.400     | 3.560
    PSU2 PIN         | na         | Watts      | na    | na        | na        | na        | na        | na        | na
    PSU2 POUT        | na         | Watts      | na    | na        | na        | na        | na        | na        | na
    PSU1 PIN         | na         | Watts      | na    | na        | na        | na        | na        | na        | na
    PSU1 VIN         | na         | Volts      | na    | na        | na        | na        | na        | na        | na
    PSU2 VIN         | na         | Volts      | na    | na        | na        | na        | na        | na        | na
    PSU2 IOUT        | na         | Amps       | na    | na        | na        | na        | na        | na        | na
    PSU1 IOUT        | na         | Amps       | na    | na        | na        | na        | na        | na        | na
    PSU1 POUT        | na         | Watts      | na    | na        | na        | na        | na        | na        | na
  • I'm using the Noctua NH-L9x65 SE-AM4 HSF and the space between it and the first DIMM slot is really tight, to the extent the heatpipe touches the DIMM to the extent you need to bend it slightly
  • A test debian stretch netinstall on to an M2 went with zero problems (even using what some would consider the odd config of "legacy" MBR boot from an NVME drive, which works without issue)
  • Ryzen 3000 is fast. Like, really nicely fast. I already disabled PBO in the BIOS as I don't plan to use (power efficiency ) it but here's some quick'n'dirty openssl scores compared with my current Haswell E3 (same version of openssl also under debian stretch):

    Code:
    type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
    md5             131680.77k   312282.73k   552217.26k   685280.60k   731400.87k   739469.99k
    sha1            141524.19k   335635.35k   694369.54k   925579.61k  1019554.47k  1038150.31k
    sha256           84687.84k   186115.69k   344207.19k   437095.42k   472416.26k   471684.44k
    sha512           54593.59k   218337.73k   384408.40k   571728.55k   680763.39k   684119.38k
    Code:
    type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
    md5             165881.11k   376279.10k   654155.95k   805793.11k   863455.91k   868013.40k
    sha1            289492.20k   744160.96k  1513553.49k  2047567.87k  2281201.66k  2300051.46k
    sha256          263302.86k   666966.55k  1388905.47k  1905337.69k  2137448.45k  2156341.93k
    sha512           80800.28k   323225.73k   543556.52k   796167.51k   918145.71k   928879.96k
  • Idle power usage of the out-of-the-box config is pretty good at ~31W (that's with a single ECC DIMM). I dare say this'll be reduced considerably with some tweaking - especially given that the idle power usage of some of the X570 systems was pushing north of 60W in many of the benches I read (granted, almost all with a GPU as well)
  • ECC is seemingly working, or at least reported as available; lshw and dmidecode report multi-bit ECC although I'm not sure that's correct given this is unbuffered memory (I suspect it's reading what the memory controller/motherboard is reporting rather than what's actually "negotiated"). edac-util doesn't recognise the Ryzen 2 driver when used with kernel 4.9 or 4.19 from backports. I was somewhat surprised to see dmidecode reporting a data width of 128bits rather than 72 so I'm not entirely sure what's going on here:
    Code:
    Handle 0x0019, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x000F
            Error Information Handle: 0x0018
            Total Width: 128 bits
            Data Width: 64 bits
            Size: 16384 MB
            Form Factor: DIMM
            Set: None
            Locator: DIMM 1
            Bank Locator: P0 CHANNEL A
            Type: DDR4
            Type Detail: Synchronous Unbuffered (Unregistered)
            Speed: 2666 MHz
            Manufacturer: Micron Technology
            Serial Number: xxxxxxxxxxxxx
            Asset Tag: Not Specified
            Part Number: 18ASF2G72AZ-2G6D1
            Rank: 2
            Configured Clock Speed: 2666 MHz
            Minimum Voltage: 1.2 V
            Maximum Voltage: 1.2 V
            Configured Voltage: 1.2 V

As ever, more work needed. As per other messages in this thread, I'll try the same fio SSD test when I get a SATA drive plugged in but don't expect me to be moving too quickly ;)
 
  • Like
Reactions: fasting

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,395
509
113
Yup, just as long as you update the BIOS beforehand - this was the first time I've updated a BIOS without even installing the CPU in the socket, doing so through IPMI was rather cool. As per my comment further up the thread there's also ASRock's management utility that can do the same thing indirectly.

It's going to be a while before it achieves replacement status for me but I'm optimistic as of the last hour or two; suspect a couple of BIOS updates are all that's needed to fix the IPMI sensors and a newer kernel to support ECC detection for the memory controller.

P.S. One thing I noticed in the dmidecode output is that the CPU reported supporting a maximum of 128GB so I'm cautiously optimistic that support for 32GB UDIMMs is already baked in to the controller and will arrive at some point in the future.
Code:
root@frogstar:~# edac-util
edac-util: Error: No memory controller data found.
root@frogstar:~# dmidecode --type memory
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 3.2.0 present.
# SMBIOS implementations newer than version 3.0 are not
# fully supported by this version of dmidecode.

Handle 0x000F, DMI type 16, 23 bytes
Physical Memory Array
        Location: System Board Or Motherboard
        Use: System Memory
        Error Correction Type: Multi-bit ECC
        Maximum Capacity: 128 GB
        Error Information Handle: 0x000E
        Number Of Devices: 4
 

rekd0514

New Member
May 11, 2015
25
7
3
I suppose you could use some other monitoring software to check temps unless those in the IPMI are that important to you. I personally won't need more than even 16GB with my server, but it is nice to know it may support more for people in this thread that want to run a lot of VMs. Can you select any faster speeds for memory in BIOS or is it stuck at 2666MHz?

Definitely nice that you can flash through IPMI in case you don't have an older CPU. I know some AM4 boards have a BIOS flashback feature so you can do this with no CPU/memory and some have dual BIOS like GPUs sometimes have.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,395
509
113
2666MHz is the speed of the memory, it's the fastest ECC UDIMM I could find and the fastest supported by ASRock. There's a trillion options in the BIOS that I suspect I'll need days to read up on to understand, but ultimately memory performance is much less of a big deal on Ryzen 2 than it was on previous generations so I'm not overly concerned in that regard.

I certainly didn't intend to run it any faster than 2666, and it's possible it might drop down to 2400 when I add the second DIMM anyway, but even so the new architecture seems to mitigate the need for high-bandwidth, low-latency RAM in order for it to perform well. All the benches I've seen so far (granted, all windows, none for linux) have shown the performance difference between 2400 and 3200 to be under 5% (at least for the kind of workloads I care about which are all compute-heavy and not memory-heavy). Indeed, it was the reviews that showed how much Ryzen was no longer dependent on memory that convinced me to pull the trigger so soon after launch (although I've been planning to upgrade my haswell for over a year now). Great to see AMD back in the game with such aplomb.

I'm not usually the sort of person who worries about temps (quite the opposite really) but given that it's a brand new proc that I bought precisely to chuck a fairly CPU-heavy workload at (mostly ffmpeg) it'd be good to be reasonably sure I'm not cooking things.

The BIOS flash routines via USB were only present on a few of the X470 boards but yeah it's great to see the ASRock kit taking advantage of it. Since I got my Supermicro IPMI BIOS flashing working I've been hooked on it, so it's great that ASRock provide it for free and without making you jump through any hoops.
 

ReturnedSword

Active Member
Jun 15, 2018
526
226
43
Santa Monica, CA
I wonder what the memory speed will be with 4 x DR ECC UDIMM and Ryzen 3000’s improved IMC. Since 32GB ECC UDIMMs are still magical unicorns, acceptable speed with 16GB modules might be the way to go since the maximum memory is 64GB anyway.

I’ve been trying to source this board for a while here in the US, and not hot on paying Newegg’s $400+ current list price. If it takes too long for the price to go back to $250-270 the X570 based ASRock Rack boards might be out by then (Q4).
 

rekd0514

New Member
May 11, 2015
25
7
3
Old v3/v4 xeons are still exceptionally useful.

Ryzen competes in the e3 range more than e5 price wise and pcielane wise...
That said, in no universe is that 2650 v3 winning single or multi-threaded against a r7 2700.
The would be fairly close in multithreaded due to the 2c/4t advantage the xeon has but the clocks of the r7 2700 are significantly higher... even a r7 1700 (65w) chip beats the 2650 v3 in multithreaded applications.
And those are around $130... so yeah, cheap light compute.

Midyear Zen2 (ryzen 3000) and Epyc releases should shake things up some more.
It is on Amazon for $270. Not much benefit in getting X570 though except for PCIE 4.0. It looks like the all have active cooling on the chipset and use a lot of power too.

https://www.amazon.com/ASRock-Rack-X470D4U-Asrock-Motherboard/dp/B07PNFTPGB
 
Last edited: