The IPMI one. IIRC proper Zen 2 support for k10temp (the sensor used by lm-sensors) only landed in kernel 5.4.
[v2,2/2] hwmon/k10temp: Add support for AMD family 17h, model 70h CPUs - Patchwork
And so it begins... First AMD Ryzen AM4 server motherboard.
- Thread starter McKajVah
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
[v2,2/2] hwmon/k10temp: Add support for AMD family 17h, model 70h CPUs - Patchwork
My observations on the X470D4U:
CPU: Ryzen 5 3600
Cooler: Dynatron A24
RAM: (2) Samsung M391A2K43BB1-CRC 16GB DDR4-2400 CL17 ECC UDIMM
Slot 6: Mellanox ConnectX-3
Slot 5: empty
Slot 4: LSI SAS2308
I was able to overclock the RAM to 3200 CL 18 at stock voltage and 3400 was mostly stable. I expect I'll be able to hit 3600 at a higher voltage. There is currently no EDAC support for Zen2 in the linux kernel (coming in 5.4) so I didn't want to mess with the RAM too much without getting feedback from EDAC.
Unless I'm missing something, the max you can push the RAM voltage is 1.20V +/- 124mV so 1.324V is as high as it'll go. I'm surprised you can't set it to 1.35V.
In order to use both the x16 (physical) slots you must configure the 16 lanes as 2x8 in the BIOS. I'm used to that being automatic on other boards. The options aren't terribly flexible: 1x16, 2x8, 4x4. It would be nice if you could able to do 1x8 + 2x4 so you could have a single x8 card plus a dual NVMe 2x4 card in the other slot.
I'm getting 512 byte PCIe packet sizes for the Mellanox and LSI HBA cards, which is nice, but the max packet size for devices hanging off the X470 SB is stuck at 256 bytes. Fortunately, the packet overhead is less painful with PCIe3 than it was with PCIe2.
I ran into a strange compatibility issue with my ASUS VN248H-P monitor. When it was plugged in the BMC would crash regularly during reboots and would take several minutes to recover. The BMC has been perfectly stable since I unplugged the monitor. Strange.
Fan connector FAN2 only works correctly in smart mode (ie temperature following mode). When set to a fixed speed the fan will only spin at the minimum. The same fan connected to either FAN3 or FAN4 works as expected when set to a fixed speed (eg full on). Perhaps FAN2 is intended for coolers with dual fans. I did not test FAN5 or FAN6.
My 2U chassis doesn't have enough airflow to handle the 65W Ryzen 5 3600. Under mprime load, the CPU temp hits 90-93C (thermal limit) and and the all core boost drops from 4GHz-ish to 3.8GHz. That's still better than the base clock which is 3.6GHz. Perhaps fans optimized for high dynamic pressure would help but ultimately I think my chassis is just not airflow friendly.
CPU: Ryzen 5 3600
Cooler: Dynatron A24
RAM: (2) Samsung M391A2K43BB1-CRC 16GB DDR4-2400 CL17 ECC UDIMM
Slot 6: Mellanox ConnectX-3
Slot 5: empty
Slot 4: LSI SAS2308
I was able to overclock the RAM to 3200 CL 18 at stock voltage and 3400 was mostly stable. I expect I'll be able to hit 3600 at a higher voltage. There is currently no EDAC support for Zen2 in the linux kernel (coming in 5.4) so I didn't want to mess with the RAM too much without getting feedback from EDAC.
Unless I'm missing something, the max you can push the RAM voltage is 1.20V +/- 124mV so 1.324V is as high as it'll go. I'm surprised you can't set it to 1.35V.
In order to use both the x16 (physical) slots you must configure the 16 lanes as 2x8 in the BIOS. I'm used to that being automatic on other boards. The options aren't terribly flexible: 1x16, 2x8, 4x4. It would be nice if you could able to do 1x8 + 2x4 so you could have a single x8 card plus a dual NVMe 2x4 card in the other slot.
I'm getting 512 byte PCIe packet sizes for the Mellanox and LSI HBA cards, which is nice, but the max packet size for devices hanging off the X470 SB is stuck at 256 bytes. Fortunately, the packet overhead is less painful with PCIe3 than it was with PCIe2.
I ran into a strange compatibility issue with my ASUS VN248H-P monitor. When it was plugged in the BMC would crash regularly during reboots and would take several minutes to recover. The BMC has been perfectly stable since I unplugged the monitor. Strange.
Fan connector FAN2 only works correctly in smart mode (ie temperature following mode). When set to a fixed speed the fan will only spin at the minimum. The same fan connected to either FAN3 or FAN4 works as expected when set to a fixed speed (eg full on). Perhaps FAN2 is intended for coolers with dual fans. I did not test FAN5 or FAN6.
My 2U chassis doesn't have enough airflow to handle the 65W Ryzen 5 3600. Under mprime load, the CPU temp hits 90-93C (thermal limit) and and the all core boost drops from 4GHz-ish to 3.8GHz. That's still better than the base clock which is 3.6GHz. Perhaps fans optimized for high dynamic pressure would help but ultimately I think my chassis is just not airflow friendly.
Interesting, I hadn't really considered trying to overclock the RAM, mostly because I didn't think it was needed - almost all of the testing I've seen of Ryzen 3000 shows marginal benefits from higher RAM clocks except in a very few scenarios.
The PCIe slot split bit me as well - I was convinced one of my cards had died at first. I'm pretty sure that in the beta 3.04 BIOS the split across both of the 16x physical slots was automatic but in 3.20 it needed to be set manually.
I'm surprised at you hitting the thermal limits though - I've not tried using mprime ("lies, damned lies and synthetic benchmarks") but even maxing out my 3700X with 16 threads of ffmpeg I've not hit the thermal throttle threshold (well, except when I was on BIOS 3.04 which refused to ramp up the CPU fan because sensors weren't working properly). However, from a quick read around, mprime seems somewhat notorious for creating much higher temperatures than most other workloads so I wouldn't be overly concerned with high temps in that regard myself. Depends a lot on what the air intakes on your chassis is like though I suppose.
The PCIe slot split bit me as well - I was convinced one of my cards had died at first. I'm pretty sure that in the beta 3.04 BIOS the split across both of the 16x physical slots was automatic but in 3.20 it needed to be set manually.
I'm surprised at you hitting the thermal limits though - I've not tried using mprime ("lies, damned lies and synthetic benchmarks") but even maxing out my 3700X with 16 threads of ffmpeg I've not hit the thermal throttle threshold (well, except when I was on BIOS 3.04 which refused to ramp up the CPU fan because sensors weren't working properly). However, from a quick read around, mprime seems somewhat notorious for creating much higher temperatures than most other workloads so I wouldn't be overly concerned with high temps in that regard myself. Depends a lot on what the air intakes on your chassis is like though I suppose.
I've been following this thread and have almost convinced myself to go with this board plus a 3900x instead of a Xeon e-2288g for an upcoming server build. I do have one requirement that I'm hoping someone here can help me with. I need to use the onboard raid along with server 2019. It looks like only consumer operating systems are supported but that doesn't necessarily mean the drivers won't work on 2019. Has anyone had experience with this?
The local IPMI interface(/dev/ipmi0) on my server's board seems to become unavailable during periods of high disk load. Specifically when I'm running backups to a zfs array connected to an HBA.
Any ideas what's causing this?
Any ideas what's causing this?
According to the block diagram, there shouldn't be any connection at all between the PCIe slots used for your HBA (which comes from the CPU) and the PCIe for the BMC (which comes from the southbridge) so I'd be inclined to think it was an OS problem rather than a hardware problem on the face of it.
Is the BMC web interface still functional during your backups?
Is the BMC web interface still functional during your backups?
I haven't checked that yet, I'll have to trigger a backup while I'm awake.
On my desktop, which uses the same board but earlier BMC version, I don't see this issue when backing up from the nvme devices, which are in a two-drive mirrored zpool.
Different OS as well. Desktop runs arch and server runs proxmox 6.
On my desktop, which uses the same board but earlier BMC version, I don't see this issue when backing up from the nvme devices, which are in a two-drive mirrored zpool.
Different OS as well. Desktop runs arch and server runs proxmox 6.
Been a while since I've last posted anything meaningful here but the long and the short of it is I had to send my X470D4U in for a replacement. Essentially, it seemed that video-out failed completely and I'd get no output from either the IPMI iKVM nor the VGA-out. This would seemingly halt the POST with five beeps and a 70 on the seven segment LED.
I've now received the replacement board; it's identical to the board I originally bought (hardware revision 1.01, came with IPMI v1.50 and BIOS v1.50) so hopefully the fix for this error is purely in software.
One thing that has been fixed is the NIC bonding - eth0 and eth1 are now both available straight out of the box without me having to use the S_BMCMAC.EXE/BMCMAC util to rewrite the IPMI NIC MAC address. Hopefully this oversight was just a config defect with one of the earlier batches of boards.
I am currently testing the not-yet-released IPMI v1.90 and BIOS v3.25. This comes with PSP firmware 0.13.0.29 and microcode version 08701013 under my 3700X; this still shows up as AGESA "Combo-AM4 1.0.0.3", I'm not sure how to verify whether or not this might be ABB or ABBA.
One obvious regression I've run in to already with the current board and software combination (that definitely didn't happen on my previous board) is that whenever the box is rebooted/goes through POST, the IPMI OS will also restart - you'll see the "Waiting for BMC MAC Initializing" message, at which point the BMC loses connection and appears to reboot (web interface stops working, interface stops responding to pings), followed by five beeps from the speaker. Not a deal breaker for me at the moment but very annoying, and will likely stop most people entering the BIOS via the HTML5 GUI if there are bootable devices attached since you won't be able to press the keyboard button in time. I can confirm that unplugging all bootable devices and setting the BIOS option Server Mgmt > Wait for BMC to Disabled does NOT work as a workaround. Flashing back to ye olde BIOS v3.20 and IPMI v1.60 and the bug is still there.
Suffice to say I'm still not putting the board in to production use yet - hopefully the next set of firmware fixes will put these niggles to bed.
I've now received the replacement board; it's identical to the board I originally bought (hardware revision 1.01, came with IPMI v1.50 and BIOS v1.50) so hopefully the fix for this error is purely in software.
One thing that has been fixed is the NIC bonding - eth0 and eth1 are now both available straight out of the box without me having to use the S_BMCMAC.EXE/BMCMAC util to rewrite the IPMI NIC MAC address. Hopefully this oversight was just a config defect with one of the earlier batches of boards.
I am currently testing the not-yet-released IPMI v1.90 and BIOS v3.25. This comes with PSP firmware 0.13.0.29 and microcode version 08701013 under my 3700X; this still shows up as AGESA "Combo-AM4 1.0.0.3", I'm not sure how to verify whether or not this might be ABB or ABBA.
One obvious regression I've run in to already with the current board and software combination (that definitely didn't happen on my previous board) is that whenever the box is rebooted/goes through POST, the IPMI OS will also restart - you'll see the "Waiting for BMC MAC Initializing" message, at which point the BMC loses connection and appears to reboot (web interface stops working, interface stops responding to pings), followed by five beeps from the speaker. Not a deal breaker for me at the moment but very annoying, and will likely stop most people entering the BIOS via the HTML5 GUI if there are bootable devices attached since you won't be able to press the keyboard button in time. I can confirm that unplugging all bootable devices and setting the BIOS option Server Mgmt > Wait for BMC to Disabled does NOT work as a workaround. Flashing back to ye olde BIOS v3.20 and IPMI v1.60 and the bug is still there.
Suffice to say I'm still not putting the board in to production use yet - hopefully the next set of firmware fixes will put these niggles to bed.
Attachments
I have had a similar experience, although I cannot confirm if the speaker beeps because I've not been in the same room with the host when the BMC crashes.
Do you have a monitor attached to the VGA port?
In my case, the solution was to unplug the monitor from the VGA port. I have not had any issues with the BMC since unplugging the screen. This host is racked and the iKVM works so well that I don't have a need for an attached screen so this is an acceptable workaround for me.
I'm not sure if this counts as a BMC crash, since I don't know if this is done on purpose or not, all I know is that every time the board POSTs, the BMC ends up rebooting.
And no, no monitor attached to the VGA port, I do pretty much everything via the iKVM.
And no, no monitor attached to the VGA port, I do pretty much everything via the iKVM.
I would love to know or hear of anyone's experience using 32GB sticks of ram (either ECC Unbuffered or non-ECC ram). Looking to make a build using this board but a little confused on what it can support. Would ultimately love to get 128GB of ram in this thing.
I'm running 128GB on the X470D4u using 4x M391A4G43MB1-CTDQ
If you go to order those modules they'll show up as M391A4G43MB1-CTD on a lot of sites.
I can't claim to have gone the full 128GB (I'm cheap!) but I've been using two of the same Samsung 32GBs myself. I don't think I've seen any other 32GB ECC UDIMMs available in the UK but these ones definitely work (As ASRR even added them to the QVL some time ago).
Thats great to hear! Just ordered the board and am contemplating on non ECC higher speed (3000) ram vs the ones you listed. It seems much cheaper but being this is going into a datacenter and running Freenas (as a VM) something tells me I should just use what you all listed.
Edit: going with the one's you all have. Price is right ~150 each and know it should work out of the box. Thanks all!
Edit: going with the one's you all have. Price is right ~150 each and know it should work out of the box. Thanks all!
Last edited:
I have TR 2950x chip and want to replace old Xeon server (running VM) with new 2U build. Please advice with chassis and fan.
2U Chassis with redundant PSU?
CPU FAN with good cooling? Noise is not important.
128Gb RAM: 4x M391A4G43MB1 or 8xM391A2K43BB1 or 8xCT16G4WFD8266
CPU: TR 2950x
MB: ASRock X399D8A-2T
Storage: MSI M.2-XPANDER-AERO x4 Samsung 970 Pro
Btw. I have WraithRipper cooler. Please recommend chassis if I decide to keep it and not to go 2U.
2U Chassis with redundant PSU?
CPU FAN with good cooling? Noise is not important.
128Gb RAM: 4x M391A4G43MB1 or 8xM391A2K43BB1 or 8xCT16G4WFD8266
CPU: TR 2950x
MB: ASRock X399D8A-2T
Storage: MSI M.2-XPANDER-AERO x4 Samsung 970 Pro
Btw. I have WraithRipper cooler. Please recommend chassis if I decide to keep it and not to go 2U.
Last edited:
I think it is still some time until they are available, but since some have asked about 3rd gen Ryzen, https://forums.servethehome.com/ind...3rd-gen-amd-ryzen-mitx-with-dual-10gbe.26506/
That looks pretty exciting, might be a good target for my next router to route 10gb across VLANs
I sure hope they make a regular ATX X570 board. The X470 board is full of compromises due to the lane constraints of AM4. For example, the first m.2 is x4 PCIe2 and the second is x2 PCIe3. So, only half bandwidth from each m.2. The X470 is connected to the CPU via an x4 PCIe3 link so you're limited to 4GB/s aggregate and thus you're only going to get half bandwidth when reading from both simultaneously. That's not terrible but, because of the lane layout, reading from one m.2 will be at half speed (2GB/s) even though there's still available bandwidth to the CPU. An X570 version of the board would permit full bandwidth since link to the CPU is 8GB/s (x4 PCIe4) and the X570 can offer full x4 PCIe4 links to both m.2s. Two PCIe4 m.2s would be a problem but that's not a common use case (yet).
There are enough lanes hanging off the X570 to be able to offer an additional x4 slot which would be very welcome.
The Threadripper minimum price of admission (now $1,400) makes that option unappealing for small server use cases. Epyc just doesn't clock high enough for low thread count applications. So, the only cheap option for getting more lanes is to go X570 on AM4.
There are enough lanes hanging off the X570 to be able to offer an additional x4 slot which would be very welcome.
The Threadripper minimum price of admission (now $1,400) makes that option unappealing for small server use cases. Epyc just doesn't clock high enough for low thread count applications. So, the only cheap option for getting more lanes is to go X570 on AM4.
Just as an FYI. I have this board, the X470D4U variant, and I have been having nothing but issues. I've done various configurations with 3 separate kits of RAM with different speed settings and 3 separate processors.
-The system was unstable and would crash periodically every 24 hours or so with 4*8= 32GB of DDR4 non-ecc 2400 single rank Samsung e die and a Ryzen 7 1700. This memory and cpu are now stable and have been running for about a week on an MSI X370 gaming Pro Carbon running my Hypervisor.
-The system was unstable and would crash with that same memory and a Ryzen 5 3600. This further confirms that the Ryzen 1700s weaker IMC is not to blame.
-The system was more unstable with 64GB dual ranked memory, even with the 3600 and even when running at 1866mhz memory clock.
-The above CPU and memory are now in my FreeNAS system which now has an ASRock Phantom Gaming 4 motherboard (which has two 221100 m.2 real pcie4 x4 slots)
-During this adventure I canabalized my main desktop trying to find a winning combination. More or less my old PC is now my VMware host referenced above. So I bought a Ryzen 5 2600 because I got it for $100. I put my GSkill FlareX b-die memory in the ASRock rack board and put it in that box.i am still having memory issues. Windows doesn't see both my sticks of memory right now for some reason, but the BIOS does
-the IPMi is great, that is the only good thing I can say about this board. It is very discouraging that consumer motherboards are more stable, even a nearly 3 year old one,
Ryzen seems to work pretty well in FreeNAS. Perfoanfe is great. However, I may swap the Ryzen 5 2600 with the Ryzen 5 3600 I'm using in FreeNAS now because even on FreeNAS 11.2 beta 1 the GUI doesn't seem to be showing temperatures for 3rd gen Ryzen.
I hope this two week adventure and hole in my pocket saves someone some time
-The system was unstable and would crash periodically every 24 hours or so with 4*8= 32GB of DDR4 non-ecc 2400 single rank Samsung e die and a Ryzen 7 1700. This memory and cpu are now stable and have been running for about a week on an MSI X370 gaming Pro Carbon running my Hypervisor.
-The system was unstable and would crash with that same memory and a Ryzen 5 3600. This further confirms that the Ryzen 1700s weaker IMC is not to blame.
-The system was more unstable with 64GB dual ranked memory, even with the 3600 and even when running at 1866mhz memory clock.
-The above CPU and memory are now in my FreeNAS system which now has an ASRock Phantom Gaming 4 motherboard (which has two 221100 m.2 real pcie4 x4 slots)
-During this adventure I canabalized my main desktop trying to find a winning combination. More or less my old PC is now my VMware host referenced above. So I bought a Ryzen 5 2600 because I got it for $100. I put my GSkill FlareX b-die memory in the ASRock rack board and put it in that box.i am still having memory issues. Windows doesn't see both my sticks of memory right now for some reason, but the BIOS does
-the IPMi is great, that is the only good thing I can say about this board. It is very discouraging that consumer motherboards are more stable, even a nearly 3 year old one,
Ryzen seems to work pretty well in FreeNAS. Perfoanfe is great. However, I may swap the Ryzen 5 2600 with the Ryzen 5 3600 I'm using in FreeNAS now because even on FreeNAS 11.2 beta 1 the GUI doesn't seem to be showing temperatures for 3rd gen Ryzen.
I hope this two week adventure and hole in my pocket saves someone some time
If they can come out with a X570 version that has more room for heatsinks (so you don't have to worry about ram being touched), rotates the heatsink mounts perpendicular (how practically every other motherboard is configured), a low noise/no chipset fan, and is able to fix the CPU_PROCHOT issues, I'd definitely be onboard. I'd really like it if they came out with a full ATX version too but just an upgraded mATX version would be appreciated.