Yet another ASRock Rack X570D4U-2L2T build ( for ESXi and NAS-usage )

Partofthesolution

New Member
Dec 3, 2020
28
8
3
Hey guys,

first of all, i want to say thanks to servethehome and hardwareluxx for theire reviews about this board. Also thanks to @Shadowsong for another thread about a x570d4u-2l2t-build (and of course thanks to all contributors). :)

I also want to share my build, which is my first Homeserver build since a very long time. When i started to think about a new build, i thought about a Threadripper- or Epyc-based build first, but then i saw this tiny powerfull board. I like that its uATX, while it comes with a management chip, a lot of 1G and 10G Intel-NICs and of course the fact, that AM4 is fast enough for my needs with appropriate power consumtion. The goal is to get a fast encrypted NAS system with enough resources to host different kind of VMs in a compact case. So here is the exact setup:

Operating System/ Storage Platform: ESXi >7.0

CPU: Ryzen 7 4750G Pro
RAM: Kingston Server Premier DIMM 32GB, DDR4-3200 (KSM32ED8/32ME)
Motherboard: ASRock Rack X570D4U-2L2T
dGPU/iGPU: Onboard Aspeed AST2500 Management Chip

Case, PSU and cooling:
Chassis:
SilverStone Case Storage CS381
Power Supply: be quiet! SFX-L Power 500W SFX12V-L 3.3 (BN238)
CPU-Cooler: Noctua NH-L9a-AM4 chromax.black
Chassis Fans: 4x Noctua NF-S12A PWM chromax.black.swap, 120mm

Drives:
Drive to boot ESXi:
1x USB Stick
Drive for ESXi-VMFS: 1x Crucial MX500 500GB, M.2 (CT500MX500SSD4)
Drive for VM-ZFS-Pool over NFS as BASIC vDev: 1x Samsung SSD PM983 1.92TB, M.2 (MZ1LB1T9HALS-00007)
Drives for ZFS-NAS with yet unknown vDev-config: 5x (maybe more) Western Digital WD Red Plus 10TB, SATA 6Gb/s (WD100EFAX)

Usage Profile: All in one Server based on ESXi 7.0 to serve the stuff i need at home (opnsense, cloud services, ...) and a not yet defined NAS-System based on NFS.

Planned future upgrades:
Zen 2 CPU will be upgraded to Zen 3 Ryzen 7 or 9 CPU when available and lower in price
Maybe a SAS/SATA HBA
Maybe a Intel X710-T4L NIC (4-Port 10/5/2.5/1Gase-T)


You can find the current setup in my Geizhals-Whishlist.


I started buying the equipment two weeks ago and i am still testing the mainboard, ESXi and NAS-OS (TrueNAS / Napp-IT) before iam going productive.


Ive attached some fotos to get impressions of this build. Please note, that the fotos are made with the 3-pin-fans which are included in the CS381. Noctua NF-S12A-fans will arrive shortly. Note: With UEFI Version P1.20 its only possible to controle PWM-Fans, 3-Pin-fans will run at full speed.

20201203_132422.jpg 20201203_132434.jpg SS_CS381_vs_X570D4U-2L2T_01.jpg SS_CS381_vs_X570D4U-2L2T_02.jpg SS_CS381_vs_X570D4U-2L2T_03.jpg
SS_CS381_vs_X570D4U-2L2T_04.jpg SS_CS381_vs_X570D4U-2L2T_05.jpg SS_CS381_vs_X570D4U-2L2T_06.jpg SS_CS381_vs_X570D4U-2L2T_07.jpg SS_CS381_vs_X570D4U-2L2T_08.jpg

Some may be interested in replacing the original cooler backplate, because you want to use a cooler with custom backplate. The original backplate is attached with an adhesive pad, which in my case was very very strong, so if you try to remove it without any tricks, you will fail or probably damage the board. It was possible to cut the adhesive pad with dental floss first (which is not possible over the full area, only along the edges). Then i used a hair dryer to warm up the adhesive pad. I was able to carefuly push the cooler backplate off the PCB.

ASRock Rack X570D4U-2L2T cooler backplate.png ASRock Rack X570D4U-2L2T cooler backplate removed.jpg


Because of the relative high power consumption of 8-9Watt while the system is only connected to the power cord but not booted, i also checked the board with a thermographic camera. One of the electrical consumers (and i think its the biggest consumer in standby because its the hottest) is the x550 10G-NIC. Its heating up to 65°C after a while when the system is in stand by and none of the fans are activated. I am wondering why the x550 is active in standby, because the AST2500 management chip has its own Realtek NIC and port?

The other consumers in stand by mode are the Aspeed AST2500, its RTL8211E -NIC next to the 8-Pin ATX connector, the NUVOTON NCT6796D-R and of course some parts of the PSU.

Note: In the following picture, the AST2500 is the hottest spot. This is because the picture was taken shortly after i connected the power cord - the heatsink of the x550 was just not warm enough at this time. On the right side, you can see the NUVOTON-chip.

x570D4U-2L2T-Standby.jpg

When the system is booted, the x570 and the Samsung SSD will heat up immediate. I think i will also install a little fan and mabe a M.2 heatsink like @Shadowsong did.

x570D4U-2L2T-Booted.jpg


I think i also noticed a UEFI-bug (BIOS Version P1.20). Im not sure which of the UEFI-settings are responsible for this bug, but i think it must be some setting regarding USB, the AST2500 or SATA. I trapped into this bug twice and maybe ill try to figure out which of the setting is responsible for this bug later... The bug leads into a unresponsible system and i was only able to get out of this by removing the cmos battery for some seconds. When the yet unknown according UEFI-setting is configured, the system will boot, beep 5 time, stuck in 0xAB (DXE_SETUP_INPUT_WAIT), enters the bios automatically and will remain unresponsible for inputs, no mather if the input comes over USB-Keyboard or the webinteface. I made a video about this and uploaded it to youtube:

I am also missing the options to make the onboard SATA-Ports Hotplug-able. It seems, that ports can only be configured as normal AHCI-Ports or Raid-Ports. I installed Win10 to test Hotplug, but the test failed. Only drives which are attached since power on get initialized. It was possible to remove drives while win10 was running without any problems, but it was not possible to attache drives again or attache drives which were not connected at power on.
In my opinion, for a server-board, eSATA/Hotplug-option is mandatory. What do you think?
I just contacted ASRock Rack over a week ago because of this but i did not get any response till now...

I also noticed, that the system will stuck in POST-Screen with errorcode 0xA2 DXE_IDE_DETECT when you insert a M.2-SATA SSD but you have this M.2-slot still configured for M.2 PCIe SSD (which might be normal).

I dont know if its me or ASRock products. I am alway very impressed by the pure specs of ASRock mainboards, but the bios/uefi always feel kinda bugy with the need of the finishing touch. I had the same feeling last year with my ASRock x570 creator workstation mainboard.... Maybe its the same with other brands, i dont know.



Anyway. In general, the system is running very stable with ESXi 7.0, but remember: Ryzen and Threadripper is officially not supported by VMWare. From AMDs current plattforms, only Epyc is supported by VMWare.



In the following quote you will find the PCI table which i have extracted from the ESXi-Webinterface under Host -> Manage -> Hardware -> PCI Devices

48 items

Address Description SR-IOV Passthrough

0000:00:00.0 Advanced Micro Devices, Inc. [AMD] Renoir Root Complex Not capable Not capable

0000:00:00.2 Advanced Micro Devices, Inc. [AMD] Renoir IOMMU Not capable Not capable

0000:00:01.0 Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge Not capable Not capable

0000:00:01.2 Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge Not capable Not capable

0000:20:00.0 Advanced Micro Devices Inc PCI bridge Not capable Not capable

0000:21:0a.0 Advanced Micro Devices Inc PCI bridge Not capable Not capable

0000:2c:00.0 Advanced Micro Devices Inc AMD FCH SATA Controller [AHCI Mode] Not capable Disabled

0000:21:09.0 Advanced Micro Devices Inc PCI bridge Not capable Not capable

0000:2b:00.0 Advanced Micro Devices Inc AMD FCH SATA Controller [AHCI Mode] Not capable Disabled

0000:21:08.0 Advanced Micro Devices Inc PCI bridge Not capable Not capable

0000:2a:00.3 Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller Not capable Disabled

0000:2a:00.1 Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller Not capable Disabled

0000:2a:00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP Not capable Disabled

0000:21:05.0 Advanced Micro Devices Inc PCI bridge Not capable Not capable

0000:27:00.0 ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge Not capable Not capable

0000:28:00.0 ASPEED Technology, Inc. ASPEED Graphics Family Not capable Disabled

0000:21:04.0 Advanced Micro Devices Inc PCI bridge Not capable Not capable

0000:26:00.0 Intel Corporation I210 Gigabit Network Connection Not capable Disabled

0000:21:03.0 Advanced Micro Devices Inc PCI bridge Not capable Not capable

0000:25:00.0 Intel Corporation I210 Gigabit Network Connection Not capable Disabled

0000:21:01.0 Advanced Micro Devices Inc PCI bridge Not capable Not capable

0000:23:00.1 Intel(R) Ethernet Controller 10G X550 Disabled Disabled

0000:23:00.0 Intel(R) Ethernet Controller 10G X550 Disabled Disabled

0000:00:02.0 Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge Not capable Not capable

0000:00:02.1 Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge Not capable Not capable

0000:2d:00.0 Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 Not capable Active

0000:00:08.0 Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge Not capable Not capable

0000:00:08.1 Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus Not capable Not capable

0000:30:00.6 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller Not capable Disabled

0000:30:00.5 Advanced Micro Devices, Inc. [AMD] Raven/Raven2/FireFlight/Renoir Audio Processor Not capable Disabled

0000:30:00.4 Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 Not capable Disabled

0000:30:00.3 Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 Not capable Disabled

0000:30:00.2 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor Not capable Disabled

0000:30:00.1 ATI Technologies Inc Audio device Not capable Disabled

0000:30:00.0 Advanced Micro Devices, Inc. [AMD/ATI] Renoir Not capable Disabled

0000:00:08.2 Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus Not capable Not capable

0000:31:00.1 Advanced Micro Devices Inc AMD FCH SATA Controller [AHCI Mode] Not capable Disabled

0000:31:00.0 Advanced Micro Devices Inc AMD FCH SATA Controller [AHCI Mode] Not capable Disabled

0000:00:14.0 Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller Not capable Not capable

0000:00:14.3 Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge Not capable Not capable

0000:00:18.0 Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 0 Not capable Not capable

0000:00:18.1 Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 1 Not capable Not capable

0000:00:18.2 Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 2 Not capable Not capable

0000:00:18.3 Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 3 Not capable Not capable

0000:00:18.4 Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 4 Not capable Not capable

0000:00:18.5 Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 5 Not capable Not capable

0000:00:18.6 Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 6 Not capable Not capable

0000:00:18.7 Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 7 Not capable Not capable
You can also find a more detailed list of IOMMU group informations in this reddit.


Its a long time ago i worked with ESX Hypervisor, so i am not very experienced with current virtualisation-technics or ESX versions, but i tried the following ( and i am not sure if this should work at all ).

One of the main things im interested in, is to passthrough the onboard AMD SATA-controllers to my ZFS-NAS-System. So i tried to activate Passthrough for the AMD SATA controllers ( 0000:2b:00.0 and 0000:2c:00.0 ), which is DISABLED in default setting. After i click "Toggle Passthrough", DISABLED turned into "Enabled / Needs reboot". After a reboot, its still "Enabled / Needs reboot". So, whats the reason for this? I am not sure, but i think these controllers are the ones which are CPU-integrated, because these controllers are used by the M.2 SATA SSD.

I also tried the same with the AMD SATA controllers 0000:31:00.0 and 0000:31:00.1, which are the ones from the x570 chipset i think. It was possible to Activate Passthrough, but i wasnt able to u use these controllers inside of a VM so far.

By the way: All AMD SATA controllers identify to windows with venID 1022 (which is AMD) and devID 7901. Some of the controllers can be switched from devID 7901 to 7904 if you chose "AHCI by 7904" in bios. I think this has something todo with this KB-arcticle: https://www.amd.com/en/support/kb/faq/pa-250

So, i would be realy happy if one experienced user can help me to figure out how to passthrough the AMD SATA controllers to use it within an TrueNAS or Napp-IT VM.


For people who want to run a virtualised high end router with lot of hardware offloading, the following informations might be very interesting:

I can confirm, that its possible to ENABLE Passthrough for the I210 NICs (only activated it, not tested with a VM till now).
I can confirm, that its possible to ENABLE Passthrough for the x550 NICs (only activated it, not tested with a VM till now).
I can confirm, that its possible to Configure SR-IOV for the x550NICs. I configured SR-IOV with the maximum number of 63, which were listed in the PCI Device list immediate. (only activated it, not tested with a VM till now).

If you also have this board and tested this functions with a VM let me know. Also let me know, if you are interested in this function. In this case, i will test it for you, but at the moment, i am not focused to this function.


For now, i think thats enough information.... Please share your experience with this board and let me know if you need some information about it.
 

ramblinreck47

Active Member
Aug 3, 2019
137
51
28
Great write up! I’ll be getting my Ryzen 5 5600X in a couple of weeks to pair with my X570D4U-2L2T. I already know that I’ll be on 1.25 Beta BIOS, so I’ll report any issues when I get it up and running.
 
  • Like
Reactions: Partofthesolution

PigLover

Moderator
Jan 26, 2011
3,071
1,375
113
The x550 is getting hot when “powered off” because it provides the shared-NIC access for the BMC (see the STH review of this board for more info). It is really odd that they provide shared-NIC for BMC on a 10gbe port but that is what it is...

I’d be really interested to see photos when you figure out how to put a fan on the chipset and m.2. Very curious about the solution you come up with.
 

Partofthesolution

New Member
Dec 3, 2020
28
8
3
You can see photos in Shadowsongs post i linked above. Thanks for the hint regarding the shared NIC for BMC. I didnt noticed that before.


Today i installed the four NF-S12A:

SS_CS381_with_4x_Noctua_NF-S12A (1).jpg SS_CS381_with_4x_Noctua_NF-S12A (2).jpg SS_CS381_with_4x_Noctua_NF-S12A (3).jpg SS_CS381_with_4x_Noctua_NF-S12A (4).jpg SS_CS381_with_4x_Noctua_NF-S12A (5).jpg SS_CS381_with_4x_Noctua_NF-S12A (7).jpg SS_CS381_with_4x_Noctua_NF-S12A (8).jpg


Whit this setup, i looks like i dont need an additional mini-fan on the x570 heatsink, the x570 stays at under 60°C. A passive M.2 heatsink (like EK Water Blocks EK-M.2) is still a future option i think about.

The system is very silent now. At the moment the four NF-S12A are fixed at 35% duty which means around 500rpm each and the CPU-Cooler is in default mode, which means its controlled by CPU-temperature.

With 21,7°C room temperature, idle load and without any 3,5" drives i get these temperatures:

1607098605302.png


I also stressed the CPU with a Win10 guest and prime 95. Here is what happens with the CPU temperature.

There is a first fast temperature rise from 26 to 52°C. From 52°C there is flat temperature ramp up to 66°C. From this point, a second temperature rise up to 95°C is rushing in. When i stop Prime 95, the temperature rapidly falls down back to 56°C. After that, temp is falling with a flat temperature ramp back down to 26°C.

I have never seen this before. This reaction was also not observed in any review of the Noctua NH-L9a. Whats the reason for this temperature profile?

My guess at the moment is:
1. The temperature sensor is defective around 66°C
2. The Noctua NH-L9a is defective. Maybe there is not enough fluid in the heat pipes and around 66°C the heatpipes just collaps.

What do you think about that?

1607099939513.png


While stressing the CPU with Prime 95, the rest of the system keeps relative cool, but i think i can optimize the temperatures under load by tuning the fan mappings a bit more.

1607102607556.png
 

Partofthesolution

New Member
Dec 3, 2020
28
8
3
OK; Just found out that i selected the wrong profile (Large FFTs) in Prime95, which is stressing not only the CPU, but also the RAM at the beginning of the torture test. Thats the reason for what i noticed in the prevous post.

When i select Small FFTs, ony the CPU get stressed with 100% load and without any delay. In this case, there is of course no delay in temperature rise.


1607289764135.png

Needless to say, that this is still not normal.


I read on the Noctua website, that this cooler is designed for CPUs up to 65Watt TDP without any limitations, but it also can be used with cpus up to 95Watt TDP and good case ventilation. You can also use it with 105Watt TDP, but the cpu might limit its core clock under continius heavy load.

I just wrote a email to the Noctua-support.. Lets see what they say.
 
  • Like
Reactions: Patrick

Partofthesolution

New Member
Dec 3, 2020
28
8
3
I just spend some more hours to fight the temperature problem under full load.

In short: The Ryzen 7 Pro 4750G is some kind of a boiler or something like that. Cant belive that this CPU is specified with 65W TDP, because the Noctua NH-L9a is made for 65W TDP CPUs.

I installed a wattmeter to observe the electrical power consumption of the whole system during my tests in front of the PSU. Note: There was no 3,5" HDD-drive attached during the test. Only the Mainboard, CPU, RAM, two M.2-SSDs and CPU-/Case-fans were connected (see full system specs in the Geizhals-link in the first post and the test report at the end of this post).

Test No. 1 is with default UEFI settings. As already stated, CPU-temp rises high under full load. The total power consumption for the whole system is 140W during this conditions.

Test No. 2-5 are tests with different predefined cTDP profiles, which are labeled as "System Configuration AM4" in the UEFI. While Test No. 1 was performed with AUTO-setting, these tests are with profiles from 65W down to 35W. Interesting thing: There are two 65W-profiles. One is labeld as "65W consumer configuration", which needs 20W more electrical power than the other one, which is labeled as "65W commercial configuration". Due to switching the cTDP profile from AUTO or 65W consumer-Profile down to the 35W profile, its possible to reduce the electronic power consumption from 140W to 102W or in other words from 92°C down to 70°C CPU-temp under full load with Prime 95 Small FFTs.

Test 6 is with modified "Precicion Boost Overdrive"-option disabled in the UEFI menu. I think this must be a part of "Precicion Boost 2" technology. The 4750G is specified with 3600Mhz base clock and 4400Mhz boost clock for all cores. If i am right, this option is to automatically overclock cores which are good enough to clock higher than the overall boost clock of 4,4Ghz, because disabling this option just reduced the electrical power consumption by 4Watts.

In Test 7, i disabled "Core Performance Boost" (CPB) option. With this option, its possible to disable any kind of boost clock, so the CPU is running nailed down with 3,6Ghz over all cores. This option reduced the electrical power consumption a alot from 140Watt to 82Watt or in other words from 92°C down to 59°C under full load.

In Test No. 8, i combined the 35W cTDP-setting from Test No. 5 with with the disabled CPB-option from Test No. 7, but it wasnt possible to see any further advantages.

Maybe i will make some further tests if its possible to reduce boost clock from 4,4Ghz to a clock rate which is a good compromise between heat and better performance. Maybe there are some other tweaks to optimize the power consumption, but i think non of theme will be more effective than disabling any kind of boost clock.

For now and the near future, i will let CPB disabled, which will fix the clock of all cores to 3,6Gghz. Of course, this will reduce the compute power (i dont know how much), but 3,6Ghz is enough for my current requirements and will also safe a lot power costs, which isnt that bad i think. When i switch to Zen3 in the future, i will see how to deal with the heat. Maybe i will switch to a CPU-watercooler or i will disable CPB again and deal with the higher compute power per clock/watt and maybe more cores.

If one is interested in the test results, here is the test report.

1607474010569.png
 

Attachments

Magic8Ball

Member
Nov 27, 2019
53
16
8
I wonder if this is just an airflow issue in your case? My understanding is that air prefers the path of least resistance so you will get the bulk of the flow across the top of your case because the front intake and rear/side fans are all positioned high up. That means the volume of air actually getting down to the bottom of the case where the motherboard and heatsinks are might not quite be enough to handle the stress tests. It might be worth a quick test where you rig up an airflow shroud (aka piece of paper or thin plastic) to push the airflow from the side fans directly down onto the board and/or try other ways to move more air across the very bottom? It's hard to tell from the pictures if you can squeeze in another fan (40mm?) somewhere low down.
 

Partofthesolution

New Member
Dec 3, 2020
28
8
3
Thats a good point!

Indeed, this case is far away from being an airflowchampion. And thats the reason why i turned the four 120mm fans up to 100% / 1300rpm earch during the test series. There are two for intake on the left, two for blow out on the rear. None in the front. Plus the PSU-fan is also used to blow the air out of the case.

I am pretty sure, that there was no fundamental airflow issue during the testseries, because i checked all the other tempsensores meanwhile. For example, the x570 chipset was below 40°C and also the DDR4 modules, MB-sensor 33°C, Onboard-LAN-sensor 30°C, cardside-temp-sensor below 30°C. I also touched the end of the heatpipes but they were kinda cold.

Some moments ago, i verified this one more time with test settings from Test No. 7. Because i dont have the material at the moment to build a shroud, i placed a room fan at full speed beside the case to completely flush it with fresh air, then i remove the casecovers on the left, right an upper side. This is what happened:

1607518612354.png

I did this test three times and its absolutely reproducible.


CPU-temp dropped instantly from 60° to 59° degree, but not more. Also DDR4-temps and cardside (i think its a temp sensor for M.2?) temp started to drop instantly up to -3K. Interesting: Temps of onboard-LAN, MB and x570 raised around +4-5K.

It seems like that there is a general airflow on the mainboardsurface while the case is closed, but also a very decent airlock around the CPU/M.2 area.

I opened a thread on hardwareluxx before i started the test series. It turned out, that other peoples have the same temperature issues with 4750G and even smaller renoir APUs like 4650G. They also disabled CPB, undervolted the CPU or reduced the boost clock to reduce maximum CPU-temp.

Anyway, i think guiding plates will be a good idea for lower fan speeds. For example, i could shroud the PSU to use its fan to suck the air from below the 3,5"-bays.
 
Last edited:

Magic8Ball

Member
Nov 27, 2019
53
16
8
You could well be right about having enough airflow already (nice testing by the way), and perhaps it's a genuine ryzen characteristic, but here's a couple of extra thoughts:

- Perhaps you can benefit from a fan setup that gives you significant positive air pressure inside the case, as opposed to neutral or negative pressure. That should ensure air gets forced around the chassis and between the heatsink fins etc. Right now you've got two input and three exhaust fans (including the PSU) so perhaps it's a more neutral setup (hard to tell without comparing the speed and static pressure curves of all fans combined). Make sure you test with the case closed though and not with an open chassis otherwise it won't be representative.

- It looks like the side rear fan blows directly over the rear left fan, so at 100% surely air is just going directly in and out of the case and not anywhere else? You definitely need a shroud, but it looks fairly easy to stick a flat piece of card to the side of the drive cage and block off a direct airflow to the rear fans. You'll also need to block off the pci-e grates so air is forced down onto the motherboard and along to the exhaust fans, rather than escaping any other way.
Screenshot 2020-12-09 142858.png Screenshot 2020-12-09 142858-2.png
 

Magic8Ball

Member
Nov 27, 2019
53
16
8
Excelent idea! Here is the result, starting from idle temperatures (30°C CPU-temp). The case fans are not made for preasure, only for (silent) airflow and they are still fixed to only 500rpm so i think i can tweak them a bit more.
So the CPU is at 65°C with the fans only at 500rpm? Is that under stress with the case fully closed?
 

Partofthesolution

New Member
Dec 3, 2020
28
8
3
Yes, thats right. Case fans runnig at fixed rate (500rpm), cpu-fan at 1900rpm (because of torture run) and boost clock is disabled, which means the CPU is running at its fixed base clock at 3,6Ghz.
 
Last edited:

Magic8Ball

Member
Nov 27, 2019
53
16
8
Sweet! Looks like that's solved the overheating problem then :)

Now you just need to figure out a permanent shroud solution and you're done. When you do, perhaps make sure it's at a slight angle so it naturally pushes the air down. Right now the fans are blowing directly into a flat (vertical) wall which is fairly close by, and this may start to impact their performance if there's lots of back pressure and turbulence. The more you can help the fans the better.
 

Partofthesolution

New Member
Dec 3, 2020
28
8
3
Yes i will optimize this in the near future.

I told Noctua that i solved the overheating problem by disabling the boost clock and i also sent them the link to this thread. As a result of this, they thanked and updated theire 4750G-compatibility-list from "NH-L9a not compatible" to "NH-L9a compatible without Turbo/overclocking headroom" because they had no 4750G in their own lab to test it till now.

Next topic, same project:

Before i started this project, i thought about to passthrough the onboard SATA-controllers which are integrated in the x570 to present them to the NAS VM. Before i ordered hardware, i searched the internet for indications, but it looked that nobody tried this before or could say that its not possible. The only thing i found before i ordered the hardware, was a PCIe/IOMMU-list on reddit, were the SATA-controllers were listed. When i had finished to build the server, this was the first things i tested. Sad to say, i wasnt able to present the SATA controllers of the CPU or the x570 via passthrough in any guests, no matter which OS i used. It was possible to activate passthrough in ESXi for two of four controllers, but nothing more. I cant tell if it is not possible in general, but i did not get it to work till now. I opened an issue at the AMD support about this topic, but they told me "its to mainboard specific, please contact the vendor of the mainboard". I also mailed ASRock Rack, but as usual, no response...

At that point, i started to think about an additional HBA. The LSI3008 chipset was the only chipset, which fullfilled most of my requirements. So i bought a Supermicro AOC-S3008L-L8i adapter. Here you can find the spec sheet for this chipset.

Today the controller arrived and of course i immediately installed it.

First step was to update the firmware to enable IT-mode. IT-mode is neccessary for ZFS and other software based raids. The newer firmware also activates SR-IOV. SR-IOV is used to passthrough not only the whole controllercard to one guest, but to passthrough / virtualize every single SAS/SATA-channel, so i can eight drives to eight differend Guests with full hardwareacceleration (i hope so).

Because the X570D4U-2L2T is a UEFI-based mainboard, lsi2flash.exe (which is DOS-based) will is not working. As i tried, “Failed to initialize PAL"-error-message occured. I had to use the UEFI-version of this tool. The filename of this tool is: "sas3flash.efi". Maybe, you will run into the same problem with this or other controllers when using this board, so here is a little how to.

I know that there are tons out of HowTos out there, but this is to complete the documentation of this project and of course for myself if i want to flash again, because i am lazy.. ;-)

HOW TO to update controller cards with UEFI-based flash tools on X570D4U-2L2T (and probably other UEFI-based mainboards):

1. What you need


- A Mainboard with built-in UEFI Shell (i thinks its also possible to boot into a UEFI Shell with some kind of USB-UEFI-bootstick, but i dont know how to make one because UEFI Shell is implemented in my mainboard native)
- USB Stick
- UEFI-based Flashprogramm (filextension should be .efi) and firmware files for your controller. In this case, for the LSI3008 you need "lsi3flash.efi". If you want to know more about this tool, search for sas3Flash_quickRefGuide_rev1-0.pdf to get a documentation for this programm (or take a look at the end of this page).

For the SuperMicro 3008 cards, you will find the latest version here: www.supermicro.com - /wftp/Driver/SAS/LSI/3008/Firmware/

For LSI3008-based controllers, also take a look at this thread and download "SAS 3008 Firmwares.zip" at the end of post no1 if you want a good selection of different brandings. Thanks to @Sleyk for this collection! Please report latest firmware versions to Sleyk or in this thread.

2. Prepare yourself

- For LSI- based controller cards, note the SAS-Adresse which is labeled onto the PBC. You might need it later.

- Insert your card into your PC/Server
- Verify your card is working and activated. For example, attache drives to your controller, boot your PC, check if POST-screen of your card occurs and if you drive(s) will get detected. If you want, also boot a OS (maybe a live Linux) to verify that the cards gets detected by an OS. Disattache all drives after verification.

- Format your USB-Stick with Fat32 (dont know if ExtFat or Fat16 will work). Maybe you can also make a .iso file and
- copy the flashprogramm and the firmware files onto the USB-stick. In this case, i saved it into the folder LSI3008 (but you can also place the files into root)

- Not for my LSI3008 based card, but other cards: Some (like IBM/Lenovo M1215) need a special preperation to get flashed. For example, this card must get jumpered.


3. Flash your controller

- Insert the USB stick into an onboard USB-Port. Maybe you can moun
- Boot your Mainboard into UEFI Shell. In case of the X570D4U-2L2T, press F11 during Post screen

1607705853495.png


- Select "UEFI: Built-in EFI Shell" and press ENTER.


1607705731627.png


- UEFI Shell booted successfully

1607724959166.png




- Type "fs0:" and press ENTER to select your USB drive
- Verify fs0 is your USB-stick by typing "ls" and take a look at the root folder. If this is the wrong drive, try "fsi1:" + "ls"
and so on until you found the correct drive.

- Type "cd LSI3008" and press ENTER to switch into folder LSI3008
- Type "ls" and press ENTER again to see whats inside the folder


1607724986823.png




- Type "sas3flash.efi -list" and press ENTER to see details of your controller. As you can see, the SAS address of my controller is not set. Thats not a problem, because its labeled on the card. Make a screenshot and save the picture for you interest.

1607725006060.png




- Type "sas2flash.efi -o -e 7" and press ENTER to erase firmware and rom

1607725027722.png





- Type "sas3flash.efi -list" and press ENTER again to verify that your controller is cleared

- Press ENTER one more time to exit the programm without flashing the controller

1607725055956.png




- Type "sas3flash.efi -o -f 3008IT16.rom" and press ENTER to flash FIRMWARE. Wait till "Finished Processing Commands Successfully." appears.
- Type "sas3flash.efi -o -b mptsas3.rom" and press ENTER to flash BIOS. Wait till "Finished Processing Commands Successfully." appears.
- Type "sas3flash.efi -o -b mpt3x64.rom" and press ENTER to flash UEFI BSD. Wait till "Finished Processing Commands Successfully." appears.

- Type " sas3flash.efi -o -sasadd abcdefghijklmnop" to set your SAS address, while abcdefghijklmnop is your 16 digets SAS address. Wait till "Finished Processing Commands Successfully." appears.

- Type "sas3flash.efi -list" to verify that Firmware Version, BIOS Version, UEFI BSD Version and also the SAS address is correct.


1607725082518.png




- Power off your server and attach datacables and drives

- Power on your server and check if your controller shows you some presented drives

1607730402794.png



Have fun with your controller.
 
Last edited:

llowrey

Active Member
Feb 26, 2018
143
114
43
Before i started this project, i thought about to passthrough the onboard SATA-controllers which are integrated in the x570 to present them to the NAS VM. Before i ordered hardware, i searched the internet for indications, but it looked that nobody tried this before or could say that its not possible. The only thing i found before i ordered the hardware, was a PCIe/IOMMU-list on reddit, were the SATA-controllers were listed. When i had finished to build the server, this was the first things i tested. Sad to say, i wasnt able to present the SATA controllers of the CPU or the x570 via passthrough in any guests, no matter which OS i used. It was possible to activate passthrough in ESXi for two of four controllers, but nothing more. I cant tell if it is not possible in general, but i did not get it to work till now. I opened an issue at the AMD support about this topic, but they told me "its to mainboard specific, please contact the vendor of the mainboard". I also mailed ASRock Rack, but as usual, no response...
I had this problem, but with an Epyc. This is, apparently, an uncommon use case since I didn't get any replies to my post linked below.

https://forums.servethehome.com/ind...-pass-through-sata-ports-to-vm-flr-bug.28592/

AMD seems to have buggy Function Level Reset (FLR) implementations. I gave up on it but I also didn't invest much time searching for a workaround.
 

Partofthesolution

New Member
Dec 3, 2020
28
8
3
Just to address some more bugs here:

- SR-IOV is working with intial BIOS release v1.20 under ESXi7. With 1.28beta, SR-IOV is not working anymore (and also not with unreleased v1.30)

Dont know if this bug is also present in the public available BIOS version 1.20 or 1.28beta (i have access to the upcoming unreleased BIOS v1.30):

- Its not possible to configure tagged VLAN for onboard IPMI Management-Interface with BIOS v1.30. IPMI can only be used in untagged LAN. I will not test this issue with public 1.20 or 1.28beta.