Hey guys,
first of all, i want to say thanks to servethehome and hardwareluxx for theire reviews about this board. Also thanks to @Shadowsong for another thread about a x570d4u-2l2t-build (and of course thanks to all contributors).
I also want to share my build, which is my first Homeserver build since a very long time. When i started to think about a new build, i thought about a Threadripper- or Epyc-based build first, but then i saw this tiny powerfull board. I like that its uATX, while it comes with a management chip, a lot of 1G and 10G Intel-NICs and of course the fact, that AM4 is fast enough for my needs with appropriate power consumtion. The goal is to get a fast encrypted NAS system with enough resources to host different kind of VMs in a compact case. So here is the exact setup:
Operating System/ Storage Platform: ESXi >7.0
CPU: Ryzen 7 4750G Pro
RAM: Kingston Server Premier DIMM 32GB, DDR4-3200 (KSM32ED8/32ME)
Motherboard: ASRock Rack X570D4U-2L2T
dGPU/iGPU: Onboard Aspeed AST2500 Management Chip
Case, PSU and cooling:
Chassis: SilverStone Case Storage CS381
Power Supply: be quiet! SFX-L Power 500W SFX12V-L 3.3 (BN238)
CPU-Cooler: Noctua NH-L9a-AM4 chromax.black
Chassis Fans: 4x Noctua NF-S12A PWM chromax.black.swap, 120mm
Drives:
Drive to boot ESXi: 1x USB Stick
Drive for ESXi-VMFS: 1x Crucial MX500 500GB, M.2 (CT500MX500SSD4)
Drive for VM-ZFS-Pool over NFS as BASIC vDev: 1x Samsung SSD PM983 1.92TB, M.2 (MZ1LB1T9HALS-00007)
Drives for ZFS-NAS with yet unknown vDev-config: 5x (maybe more) Western Digital WD Red Plus 10TB, SATA 6Gb/s (WD100EFAX)
Usage Profile: All in one Server based on ESXi 7.0 to serve the stuff i need at home (opnsense, cloud services, ...) and a not yet defined NAS-System based on NFS.
Planned future upgrades:
Zen 2 CPU will be upgraded to Zen 3 Ryzen 7 or 9 CPU when available and lower in price
Maybe a SAS/SATA HBA
Maybe a Intel X710-T4L NIC (4-Port 10/5/2.5/1Gase-T)
You can find the current setup in my Geizhals-Whishlist.
I started buying the equipment two weeks ago and i am still testing the mainboard, ESXi and NAS-OS (TrueNAS / Napp-IT) before iam going productive.
Ive attached some fotos to get impressions of this build. Please note, that the fotos are made with the 3-pin-fans which are included in the CS381. Noctua NF-S12A-fans will arrive shortly. Note: With UEFI Version P1.20 its only possible to controle PWM-Fans, 3-Pin-fans will run at full speed.
Some may be interested in replacing the original cooler backplate, because you want to use a cooler with custom backplate. The original backplate is attached with an adhesive pad, which in my case was very very strong, so if you try to remove it without any tricks, you will fail or probably damage the board. It was possible to cut the adhesive pad with dental floss first (which is not possible over the full area, only along the edges). Then i used a hair dryer to warm up the adhesive pad. I was able to carefuly push the cooler backplate off the PCB.
Because of the relative high power consumption of 8-9Watt while the system is only connected to the power cord but not booted, i also checked the board with a thermographic camera. One of the electrical consumers (and i think its the biggest consumer in standby because its the hottest) is the x550 10G-NIC. Its heating up to 65°C after a while when the system is in stand by and none of the fans are activated. I am wondering why the x550 is active in standby, because the AST2500 management chip has its own Realtek NIC and port?
The other consumers in stand by mode are the Aspeed AST2500, its RTL8211E -NIC next to the 8-Pin ATX connector, the NUVOTON NCT6796D-R and of course some parts of the PSU.
Note: In the following picture, the AST2500 is the hottest spot. This is because the picture was taken shortly after i connected the power cord - the heatsink of the x550 was just not warm enough at this time. On the right side, you can see the NUVOTON-chip.
When the system is booted, the x570 and the Samsung SSD will heat up immediate. I think i will also install a little fan and mabe a M.2 heatsink like @Shadowsong did.
I think i also noticed a UEFI-bug (BIOS Version P1.20). Im not sure which of the UEFI-settings are responsible for this bug, but i think it must be some setting regarding USB, the AST2500 or SATA. I trapped into this bug twice and maybe ill try to figure out which of the setting is responsible for this bug later... The bug leads into a unresponsible system and i was only able to get out of this by removing the cmos battery for some seconds. When the yet unknown according UEFI-setting is configured, the system will boot, beep 5 time, stuck in 0xAB (DXE_SETUP_INPUT_WAIT), enters the bios automatically and will remain unresponsible for inputs, no mather if the input comes over USB-Keyboard or the webinteface. I made a video about this and uploaded it to youtube:
I am also missing the options to make the onboard SATA-Ports Hotplug-able. It seems, that ports can only be configured as normal AHCI-Ports or Raid-Ports. I installed Win10 to test Hotplug, but the test failed. Only drives which are attached since power on get initialized. It was possible to remove drives while win10 was running without any problems, but it was not possible to attache drives again or attache drives which were not connected at power on.
In my opinion, for a server-board, eSATA/Hotplug-option is mandatory. What do you think?
I just contacted ASRock Rack over a week ago because of this but i did not get any response till now...
I also noticed, that the system will stuck in POST-Screen with errorcode 0xA2 DXE_IDE_DETECT when you insert a M.2-SATA SSD but you have this M.2-slot still configured for M.2 PCIe SSD (which might be normal).
I dont know if its me or ASRock products. I am alway very impressed by the pure specs of ASRock mainboards, but the bios/uefi always feel kinda bugy with the need of the finishing touch. I had the same feeling last year with my ASRock x570 creator workstation mainboard.... Maybe its the same with other brands, i dont know.
Anyway. In general, the system is running very stable with ESXi 7.0, but remember: Ryzen and Threadripper is officially not supported by VMWare. From AMDs current plattforms, only Epyc is supported by VMWare.
In the following quote you will find the PCI table which i have extracted from the ESXi-Webinterface under Host -> Manage -> Hardware -> PCI Devices
Its a long time ago i worked with ESX Hypervisor, so i am not very experienced with current virtualisation-technics or ESX versions, but i tried the following ( and i am not sure if this should work at all ).
One of the main things im interested in, is to passthrough the onboard AMD SATA-controllers to my ZFS-NAS-System. So i tried to activate Passthrough for the AMD SATA controllers ( 0000:2b:00.0 and 0000:2c:00.0 ), which is DISABLED in default setting. After i click "Toggle Passthrough", DISABLED turned into "Enabled / Needs reboot". After a reboot, its still "Enabled / Needs reboot". So, whats the reason for this? I am not sure, but i think these controllers are the ones which are CPU-integrated, because these controllers are used by the M.2 SATA SSD.
I also tried the same with the AMD SATA controllers 0000:31:00.0 and 0000:31:00.1, which are the ones from the x570 chipset i think. It was possible to Activate Passthrough, but i wasnt able to u use these controllers inside of a VM so far.
By the way: All AMD SATA controllers identify to windows with venID 1022 (which is AMD) and devID 7901. Some of the controllers can be switched from devID 7901 to 7904 if you chose "AHCI by 7904" in bios. I think this has something todo with this KB-arcticle: https://www.amd.com/en/support/kb/faq/pa-250
So, i would be realy happy if one experienced user can help me to figure out how to passthrough the AMD SATA controllers to use it within an TrueNAS or Napp-IT VM.
For people who want to run a virtualised high end router with lot of hardware offloading, the following informations might be very interesting:
I can confirm, that its possible to ENABLE Passthrough for the I210 NICs (only activated it, not tested with a VM till now).
I can confirm, that its possible to ENABLE Passthrough for the x550 NICs (only activated it, not tested with a VM till now).
I can confirm, that its possible to Configure SR-IOV for the x550NICs. I configured SR-IOV with the maximum number of 63, which were listed in the PCI Device list immediate. (only activated it, not tested with a VM till now).
If you also have this board and tested this functions with a VM let me know. Also let me know, if you are interested in this function. In this case, i will test it for you, but at the moment, i am not focused to this function.
For now, i think thats enough information.... Please share your experience with this board and let me know if you need some information about it.
first of all, i want to say thanks to servethehome and hardwareluxx for theire reviews about this board. Also thanks to @Shadowsong for another thread about a x570d4u-2l2t-build (and of course thanks to all contributors).
I also want to share my build, which is my first Homeserver build since a very long time. When i started to think about a new build, i thought about a Threadripper- or Epyc-based build first, but then i saw this tiny powerfull board. I like that its uATX, while it comes with a management chip, a lot of 1G and 10G Intel-NICs and of course the fact, that AM4 is fast enough for my needs with appropriate power consumtion. The goal is to get a fast encrypted NAS system with enough resources to host different kind of VMs in a compact case. So here is the exact setup:
Operating System/ Storage Platform: ESXi >7.0
CPU: Ryzen 7 4750G Pro
RAM: Kingston Server Premier DIMM 32GB, DDR4-3200 (KSM32ED8/32ME)
Motherboard: ASRock Rack X570D4U-2L2T
dGPU/iGPU: Onboard Aspeed AST2500 Management Chip
Case, PSU and cooling:
Chassis: SilverStone Case Storage CS381
Power Supply: be quiet! SFX-L Power 500W SFX12V-L 3.3 (BN238)
CPU-Cooler: Noctua NH-L9a-AM4 chromax.black
Chassis Fans: 4x Noctua NF-S12A PWM chromax.black.swap, 120mm
Drives:
Drive to boot ESXi: 1x USB Stick
Drive for ESXi-VMFS: 1x Crucial MX500 500GB, M.2 (CT500MX500SSD4)
Drive for VM-ZFS-Pool over NFS as BASIC vDev: 1x Samsung SSD PM983 1.92TB, M.2 (MZ1LB1T9HALS-00007)
Drives for ZFS-NAS with yet unknown vDev-config: 5x (maybe more) Western Digital WD Red Plus 10TB, SATA 6Gb/s (WD100EFAX)
Usage Profile: All in one Server based on ESXi 7.0 to serve the stuff i need at home (opnsense, cloud services, ...) and a not yet defined NAS-System based on NFS.
Planned future upgrades:
Zen 2 CPU will be upgraded to Zen 3 Ryzen 7 or 9 CPU when available and lower in price
Maybe a SAS/SATA HBA
Maybe a Intel X710-T4L NIC (4-Port 10/5/2.5/1Gase-T)
You can find the current setup in my Geizhals-Whishlist.
I started buying the equipment two weeks ago and i am still testing the mainboard, ESXi and NAS-OS (TrueNAS / Napp-IT) before iam going productive.
Ive attached some fotos to get impressions of this build. Please note, that the fotos are made with the 3-pin-fans which are included in the CS381. Noctua NF-S12A-fans will arrive shortly. Note: With UEFI Version P1.20 its only possible to controle PWM-Fans, 3-Pin-fans will run at full speed.
Some may be interested in replacing the original cooler backplate, because you want to use a cooler with custom backplate. The original backplate is attached with an adhesive pad, which in my case was very very strong, so if you try to remove it without any tricks, you will fail or probably damage the board. It was possible to cut the adhesive pad with dental floss first (which is not possible over the full area, only along the edges). Then i used a hair dryer to warm up the adhesive pad. I was able to carefuly push the cooler backplate off the PCB.
Because of the relative high power consumption of 8-9Watt while the system is only connected to the power cord but not booted, i also checked the board with a thermographic camera. One of the electrical consumers (and i think its the biggest consumer in standby because its the hottest) is the x550 10G-NIC. Its heating up to 65°C after a while when the system is in stand by and none of the fans are activated. I am wondering why the x550 is active in standby, because the AST2500 management chip has its own Realtek NIC and port?
The other consumers in stand by mode are the Aspeed AST2500, its RTL8211E -NIC next to the 8-Pin ATX connector, the NUVOTON NCT6796D-R and of course some parts of the PSU.
Note: In the following picture, the AST2500 is the hottest spot. This is because the picture was taken shortly after i connected the power cord - the heatsink of the x550 was just not warm enough at this time. On the right side, you can see the NUVOTON-chip.
When the system is booted, the x570 and the Samsung SSD will heat up immediate. I think i will also install a little fan and mabe a M.2 heatsink like @Shadowsong did.
I think i also noticed a UEFI-bug (BIOS Version P1.20). Im not sure which of the UEFI-settings are responsible for this bug, but i think it must be some setting regarding USB, the AST2500 or SATA. I trapped into this bug twice and maybe ill try to figure out which of the setting is responsible for this bug later... The bug leads into a unresponsible system and i was only able to get out of this by removing the cmos battery for some seconds. When the yet unknown according UEFI-setting is configured, the system will boot, beep 5 time, stuck in 0xAB (DXE_SETUP_INPUT_WAIT), enters the bios automatically and will remain unresponsible for inputs, no mather if the input comes over USB-Keyboard or the webinteface. I made a video about this and uploaded it to youtube:
I am also missing the options to make the onboard SATA-Ports Hotplug-able. It seems, that ports can only be configured as normal AHCI-Ports or Raid-Ports. I installed Win10 to test Hotplug, but the test failed. Only drives which are attached since power on get initialized. It was possible to remove drives while win10 was running without any problems, but it was not possible to attache drives again or attache drives which were not connected at power on.
In my opinion, for a server-board, eSATA/Hotplug-option is mandatory. What do you think?
I just contacted ASRock Rack over a week ago because of this but i did not get any response till now...
I also noticed, that the system will stuck in POST-Screen with errorcode 0xA2 DXE_IDE_DETECT when you insert a M.2-SATA SSD but you have this M.2-slot still configured for M.2 PCIe SSD (which might be normal).
I dont know if its me or ASRock products. I am alway very impressed by the pure specs of ASRock mainboards, but the bios/uefi always feel kinda bugy with the need of the finishing touch. I had the same feeling last year with my ASRock x570 creator workstation mainboard.... Maybe its the same with other brands, i dont know.
Anyway. In general, the system is running very stable with ESXi 7.0, but remember: Ryzen and Threadripper is officially not supported by VMWare. From AMDs current plattforms, only Epyc is supported by VMWare.
In the following quote you will find the PCI table which i have extracted from the ESXi-Webinterface under Host -> Manage -> Hardware -> PCI Devices
You can also find a more detailed list of IOMMU group informations in this reddit.48 items
Address Description SR-IOV Passthrough
0000:00:00.0 Advanced Micro Devices, Inc. [AMD] Renoir Root Complex Not capable Not capable
0000:00:00.2 Advanced Micro Devices, Inc. [AMD] Renoir IOMMU Not capable Not capable
0000:00:01.0 Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge Not capable Not capable
0000:00:01.2 Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge Not capable Not capable
0000:20:00.0 Advanced Micro Devices Inc PCI bridge Not capable Not capable
0000:21:0a.0 Advanced Micro Devices Inc PCI bridge Not capable Not capable
0000:2c:00.0 Advanced Micro Devices Inc AMD FCH SATA Controller [AHCI Mode] Not capable Disabled
0000:21:09.0 Advanced Micro Devices Inc PCI bridge Not capable Not capable
0000:2b:00.0 Advanced Micro Devices Inc AMD FCH SATA Controller [AHCI Mode] Not capable Disabled
0000:21:08.0 Advanced Micro Devices Inc PCI bridge Not capable Not capable
0000:2a:00.3 Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller Not capable Disabled
0000:2a:00.1 Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller Not capable Disabled
0000:2a:00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP Not capable Disabled
0000:21:05.0 Advanced Micro Devices Inc PCI bridge Not capable Not capable
0000:27:00.0 ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge Not capable Not capable
0000:28:00.0 ASPEED Technology, Inc. ASPEED Graphics Family Not capable Disabled
0000:21:04.0 Advanced Micro Devices Inc PCI bridge Not capable Not capable
0000:26:00.0 Intel Corporation I210 Gigabit Network Connection Not capable Disabled
0000:21:03.0 Advanced Micro Devices Inc PCI bridge Not capable Not capable
0000:25:00.0 Intel Corporation I210 Gigabit Network Connection Not capable Disabled
0000:21:01.0 Advanced Micro Devices Inc PCI bridge Not capable Not capable
0000:23:00.1 Intel(R) Ethernet Controller 10G X550 Disabled Disabled
0000:23:00.0 Intel(R) Ethernet Controller 10G X550 Disabled Disabled
0000:00:02.0 Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge Not capable Not capable
0000:00:02.1 Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge Not capable Not capable
0000:2d:00.0 Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 Not capable Active
0000:00:08.0 Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge Not capable Not capable
0000:00:08.1 Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus Not capable Not capable
0000:30:00.6 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller Not capable Disabled
0000:30:00.5 Advanced Micro Devices, Inc. [AMD] Raven/Raven2/FireFlight/Renoir Audio Processor Not capable Disabled
0000:30:00.4 Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 Not capable Disabled
0000:30:00.3 Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 Not capable Disabled
0000:30:00.2 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor Not capable Disabled
0000:30:00.1 ATI Technologies Inc Audio device Not capable Disabled
0000:30:00.0 Advanced Micro Devices, Inc. [AMD/ATI] Renoir Not capable Disabled
0000:00:08.2 Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus Not capable Not capable
0000:31:00.1 Advanced Micro Devices Inc AMD FCH SATA Controller [AHCI Mode] Not capable Disabled
0000:31:00.0 Advanced Micro Devices Inc AMD FCH SATA Controller [AHCI Mode] Not capable Disabled
0000:00:14.0 Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller Not capable Not capable
0000:00:14.3 Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge Not capable Not capable
0000:00:18.0 Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 0 Not capable Not capable
0000:00:18.1 Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 1 Not capable Not capable
0000:00:18.2 Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 2 Not capable Not capable
0000:00:18.3 Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 3 Not capable Not capable
0000:00:18.4 Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 4 Not capable Not capable
0000:00:18.5 Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 5 Not capable Not capable
0000:00:18.6 Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 6 Not capable Not capable
0000:00:18.7 Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 7 Not capable Not capable
Its a long time ago i worked with ESX Hypervisor, so i am not very experienced with current virtualisation-technics or ESX versions, but i tried the following ( and i am not sure if this should work at all ).
One of the main things im interested in, is to passthrough the onboard AMD SATA-controllers to my ZFS-NAS-System. So i tried to activate Passthrough for the AMD SATA controllers ( 0000:2b:00.0 and 0000:2c:00.0 ), which is DISABLED in default setting. After i click "Toggle Passthrough", DISABLED turned into "Enabled / Needs reboot". After a reboot, its still "Enabled / Needs reboot". So, whats the reason for this? I am not sure, but i think these controllers are the ones which are CPU-integrated, because these controllers are used by the M.2 SATA SSD.
I also tried the same with the AMD SATA controllers 0000:31:00.0 and 0000:31:00.1, which are the ones from the x570 chipset i think. It was possible to Activate Passthrough, but i wasnt able to u use these controllers inside of a VM so far.
By the way: All AMD SATA controllers identify to windows with venID 1022 (which is AMD) and devID 7901. Some of the controllers can be switched from devID 7901 to 7904 if you chose "AHCI by 7904" in bios. I think this has something todo with this KB-arcticle: https://www.amd.com/en/support/kb/faq/pa-250
So, i would be realy happy if one experienced user can help me to figure out how to passthrough the AMD SATA controllers to use it within an TrueNAS or Napp-IT VM.
For people who want to run a virtualised high end router with lot of hardware offloading, the following informations might be very interesting:
I can confirm, that its possible to ENABLE Passthrough for the I210 NICs (only activated it, not tested with a VM till now).
I can confirm, that its possible to ENABLE Passthrough for the x550 NICs (only activated it, not tested with a VM till now).
I can confirm, that its possible to Configure SR-IOV for the x550NICs. I configured SR-IOV with the maximum number of 63, which were listed in the PCI Device list immediate. (only activated it, not tested with a VM till now).
If you also have this board and tested this functions with a VM let me know. Also let me know, if you are interested in this function. In this case, i will test it for you, but at the moment, i am not focused to this function.
For now, i think thats enough information.... Please share your experience with this board and let me know if you need some information about it.