Tyan S8030GM2NE

hmw

Active Member
Apr 29, 2019
236
77
28
(UPDATE: see below on FAN control)
(UPDATE#2: see section on GPU passthrough)
(UPDATE#3: SlimSAS 8i to 2 * U.2 cable now available from Tyan)


I've started on my home-lab, hyperconverged build to replace my old NAS server, gaming PC and workstation. I decided to go AMD and initially go for Ryzen or Threadripper - but thanks to several helpful forum members e.g @TXAG26, @zer0sum and others, I was able to snag parts for cheap on eBay and other retailers - and go with EPYC and a proper server build with IPMI

Here's my build:

  • Motherboard: Tyan S8030GM2NE
  • CPU: 16 core EPYC 7302P (Using the HP Upgrade Kit from Provantage)
  • RAM: 128GB using 8 * 16GB DDR4-3200 RDIMMs (Samsung M393A2K43DB3)
  • COOLER: Supermicro 4U AMD heatsink (SNK-P0064AP4)
  • GPU: 8GB GTX1080 Mini (from Gigabyte eGPU Gaming Box )
  • SSD: Samsung 970 EVO 1 TB
  • NET: HP 544/Mellanox ConnectX-3 ConnectX-4 LX
  • HBA: LSI 2308
  • PSU: Corsair AX1200i Seasonic Focus SSR-850PX
  • CASE: RPC-4308 (It's a short depth 4U chassis with removable HDDs RPC-4308 )

The ESXi hosts my NAS, which is a Windows server with 4 x 4TB HDDs (using the LSI2308 HBA), as well as a Windows 10 VM that uses the GTX 1080 in passthrough mode for some gaming sessions. And of course, my workstation - which was formerly a quad core Intel NUC with 32GB - becomes just an Ubuntu VM on the server.

The Tyan motherboard:

mobo_bare.jpg

The S8030GM2NE is probably the cheapest PCIe 4.0 EPYC motherboard around. It doesn't have the dual 10GbE LAN adapters of its sibling (S8030GM4TNE) and has only 5 PCIe slots. However it does have SAS Mini-HD and SlimSAS connectors (Instead of OCulink) and IPMI powered by the usual ASPEED 2500 chip. It also has two M.2 NVMe slots. However, if you use a video card with an oversized fan (the Gigabyte GPU has a 130mm fan and barely clears any components next to it) one of those M.2 slots is effectively useless

The Tyan BIOS is also basic. The IPMI is similarly basic and allows you to mount virtual images from a network - provided you use the Java KVM utility. The HTML5 KVM webapp doesn't have this functionality. The sensors it has are also basic - although I don't know if that is a Tyan thing or an EPYC thing. If you're used to Dell's BIOS and IPMI or even Supermicro ... Tyan is just a lot more basic. For example here's all the temp sensors the board can display:

mobo_sensors.png

The good thing is that the board layout means the MOSFETs are relatively cool. I tested the board and RAM with Memtest86. The memory MOSFETs didn't cross 52c. Nor did the CPU exceed 50c.

tyan_memtest_hammer.jpeg

Memtest reveals the memory bandwidth to be 13.98 GB/sec and the latency to be 42.835ns. I also set the cTDP and the Package Power Limit control to 180W, as well as setting Determinism to Power

tyan_ctdp.jpeg

For installing ESXi, I made sure that the IOMMU was enabled

tyan_iommu.jpeg

The board does support PCIe bifurcation on all slots, if that is something that is needed

tyan_pci_bifur.jpeg

The most disappointing aspect of the board is the fan control - it can be set to one of two values - MANUAL and FULL SPEED. That's it. Right now all the temps you see above were with the fan duty cycle set to the default of 30%. I am going to create a support ticket and ask Tyan how to get the fan to vary with CPU or SYS temps

tyan_fan_control.jpeg

UPDATE: thanks to @jpmomo pointing out that the fan control works - in a fashion - I set the duty cycle to 15 and load tested the motherboard. The winner here is clearly the Supermicro SNK-P0064AP4 heatsink. At the default 30% duty cycle, it does 1600RPM and is super quiet. The heatsink also ensures that the CPU doesn't cross 68C under load. That is an amazing result. Pushing the duty cycle down to 15% makes the CPU fan spin at 1300RPM but it will spin up - once the CPU crosses 75C. I can't believe how good this heatsink is. Of course the 7302P is "only" a 180W processor so there's that :)


The other disappointment was that supposedly one can access the BIOS via the IPMI (without the KVM) - there's a BIOS icon in the WebUI but all it does is show a blank page, even after the proper credentials have been entered

mobo_bios_redfish.png

Here's the build (not yet in the case)

mobo_cards.jpg

I tried to flash the LSI2308 card from IR mode to IT mode, but booting on the board with FreeDOS and using the DOS sas2 flasher didn't work. Had to use sas2flash.efi and also an older EFI shell that would support the sas2flash utility.

Another problem was that installing ESXi 7.0 didn't work. I'm going to test with more settings but I got 'decompression' errors despite using a USB drive with the image - and not the KVM virtual image. Luckily ESXi 6.7U3 installed like a champ and I was able to create a Win10 and a Server2019 VM.

For passing the GPU and the LSI controller thru to the VM, I was able to set that in the ESXi console

esxi_passthru.png

and the following values

In the Win10 vm's VMX file:

Code:
hypervisor.cpuid.v0='FALSE'
pciPassthru1.msiEnabled='FALSE'
UPDATE: also added this line, it's now rock solid:
Code:
pciPassthru.use64bitMMIO="TRUE"
In the /etc/vmware/passthru.map
Code:
# <ven_id> <dev_id> <reset_method> <setting>
10de  10f0  d3d0  false
The GPU is actually two devices passed thru - one is the GPU itself and the other is the HDMI audio controller (edit: dev_id 10f0 is the HDMI Audio device) that is part of the GPU. I simply set pciPassthruX.msiEnabled=FALSE for every device associated with the GPU except the GPU itself. And then repeated it in the passthru.map - after commenting out the default NVIDIA entry (It says '10de ffff bridge false'')

With these modifications I was able to start the Windows VM, install Windows 10 2004 and the GPU drivers for NVidia. After (re)booting I lost the ability to view the guest console in the ESXi WebUI, but VNC and Remote Desktop work fine. Graphics acceleration works fine and I can run 3DMark without problems (well ... almost)

3DMark has a problem with Ryzen and EPYC sensors and to prevent it from hanging when it starts collecting system info, you need to install an OLDER version of FutureMark system info (ver 5.2 link here) - once that was installed, 3DMark completed the benchmarks successfully, and gave me a TimeSpy score of 7158 for graphics and 7670 for the CPU. I was able to install the Aorus Engine app from Gigabyte and OC the graphics card

timespy_bench01.PNGaorus_settings.PNG

For the Windows Server 2019 VM I was able to pass through the LSI controller without problems. I attached a SATA SSD (ADATA) to it and ran CrystalDiskMark just for fun - it gave me the expected SATA speeds.

Capture1.PNG

That's all I have time for today, but will be updating this post with more benchmark results - especially memory benchmarks both native and in the VMs, more installation experiences (will be installing Ubuntu 20.04 LTS, Percona and Jupyter for sure). I will also get a power meter and measure the actual watts consumed. And I need to mod some PSU cables and connectors so that the motherboard fits nicely into the chassis and doesn't have PCIe and EPS cables all over the place. Never modded PSU cables before - so would love input!

I also noticed that the SAS controller runs hot! Hence I will be trying to shoe horn three fans into the case so that there's airflow over the SAS controller, the Mellanox adapter and NVMe drive. If you look at the RPC-4308, it already has two fans at the back of the case to extract air outwards - wondering if that is enough to ensure airflow over the DIMMs . The SNK-P0064AP4 heatsink and five 80mm Arctic Cooling fans serve to keep the DIMMs and CPU running cool. CPU temps are 32c idle and DIMMs are 33c to 37C

Update 3#

The SlimSAS 8i to 2 x U.2/NVMe cable is now available - I got one for $75. It's good quality and comes with a Molex connector and not the usual SATA power connector. Now to find a decent low-TDP NVMe drive

IMG_1312.jpeg
 
Last edited:

Wolvez

New Member
Apr 24, 2020
18
4
3
Thanks for posting. I was thinking about getting a S8030GM4TNE for my Epyc build since its available now. Based on your review I think I will keep waiting for the Asrock RomeD8-2t.
 

Kev

Active Member
Feb 16, 2015
439
84
28
37
I'm expecting my board on Wednesday from AVA Direct. Also have the 7302P from pro provantage waiting. I only have 128GB of 2666 DDR4 but I was hoping it could overclock to something a bit higher to keep the latency down? Something about 1:1 with the internal fabric. Any memory overclock options in the BIOS?
 

hmw

Active Member
Apr 29, 2019
236
77
28
I'm expecting my board on Wednesday from AVA Direct. Also have the 7302P from pro provantage waiting. I only have 128GB of 2666 DDR4 but I was hoping it could overclock to something a bit higher to keep the latency down? Something about 1:1 with the internal fabric. Any memory overclock options in the BIOS?
You can set the following DDR frequencies:

2666
2993
3200

tyan_bios_mem.jpeg

I have 3200 memory and set it to 2933 (so that latency is reduced) and it seems to work fine. But I didnt see any options to set CL, TRCD, TRP or TRAS. I guess it just reads the SPD ROM info. In this case, if you have CL at 17 for DDR4-2666 and you overclock to DDR4-2933 AND you need relaxed timings - I don't know what would happen. Perhaps you can see what all timings are there in your memory SPD ROM?
 

jpmomo

Member
Aug 12, 2018
99
31
18
I also have that mb. I think the manual fan control will allow the fans to automatically spin up as needed. Ex. I set to 30 and run a stress test. I can see the fans increase their speed automatically. It seems to work in a similar way as supermicro's optimal mode.
 
  • Like
Reactions: hmw

hmw

Active Member
Apr 29, 2019
236
77
28
I also have that mb. I think the manual fan control will allow the fans to automatically spin up as needed. Ex. I set to 30 and run a stress test. I can see the fans increase their speed automatically. It seems to work in a similar way as supermicro's optimal mode.
@jpmomo - can you confirm this? I will run a stress test next week (now that my VMs are up and running - I don't want to disturb them just yet) I actually ran a memtest run and the fans didn't speed up, is there a way to specify what the curve is or what are the temps where the fans speed up? Or is this automatic?
 

jpmomo

Member
Aug 12, 2018
99
31
18
I can confirm that the fans speed up (at least the fans that are pwm). I tried this with both w10 and benchmark sw and VMware with some proprietary sw. I didn't see any way to set a proper curve but will check tomorrow. In the bios, I remember being able to set the starting percentage.
 

jpmomo

Member
Aug 12, 2018
99
31
18
@jpmomo - can you confirm this? I will run a stress test next week (now that my VMs are up and running - I don't want to disturb them just yet) I actually ran a memtest run and the fans didn't speed up, is there a way to specify what the curve is or what are the temps where the fans speed up? Or is this automatic?
please see my post in the other thread "are tyan boards any good"
 

Kev

Active Member
Feb 16, 2015
439
84
28
37
I also got my S8030 board today. Does anyone have PMBUS connected from PSU? I enabled it because my SM PSU has PMBUS but in the sensors, all the PSU sensors are "disabled" Can't see how to enable it. Also, the BIOS menu seems to be password protected and its not the IPMI password. Any clues?
 

hmw

Active Member
Apr 29, 2019
236
77
28
I also got my S8030 board today. Does anyone have PMBUS connected from PSU? I enabled it because my SM PSU has PMBUS but in the sensors, all the PSU sensors are "disabled" Can't see how to enable it. Also, the BIOS menu seems to be password protected and its not the IPMI password. Any clues?
The BIOS password is Administrator/superuser - let me know if you get it to work though. All I can see is a blank screen. Do your fans work (i.e. change speed and ramp up and down?)
 

Kev

Active Member
Feb 16, 2015
439
84
28
37
The BIOS password is Administrator/superuser - let me know if you get it to work though. All I can see is a blank screen. Do your fans work (i.e. change speed and ramp up and down?)
Yes, I got in but doesn't load anything. Unfinished I guess. I am running memtest86 on my new build and i got a burst of memory errors on core 9. I went directly to 2993 from 2666 ECC RDIMM so let me try to bump that back down and run memtest 5.01. Also memtest is just reporting x64 and not x72 or ECC?
 

hmw

Active Member
Apr 29, 2019
236
77
28
Yes, I got in but doesn't load anything. Unfinished I guess. I am running memtest86 on my new build and i got a burst of memory errors on core 9. I went directly to 2993 from 2666 ECC RDIMM so let me try to bump that back down and run memtest 5.01. Also memtest is just reporting x64 and not x72 or ECC?
Yeah it's a blank screen - jeez I guess we're beta testing for Tyan

Memtest says for my RAM: PC4-25600 DDR4 ECC 3200MHz / 22-22-22-52
 

hmw

Active Member
Apr 29, 2019
236
77
28
Finally discovered what was going on with the FAN control. It won't kick in unless the CPU is at 76C to 78C. And with the SNK-P0064AP4 @ 30% duty cycle, the CPU just wasn't getting hot enough. This is one amazing heatsink. Either that or I have a CPU that just does 3GHz on all cores at super low temps :p

In any case, I can say that the Tyan fan control works - in a fashion. I put the fan duty cycle to 15% to increase temps and re-tested with Prime95 and here's what happened:

Screen Shot 2020-06-02 at 11.27.11 PM.png

CPU went from 41c to 76c - the fan kicked in from 1300rpm to 1900rpm (the max speed is 4300 rpm) and cpu temps went from 76c to 68c!

Kudos to AMD for an amazing CPU. And to Supermicro for an amazing heatsink !
 

hmw

Active Member
Apr 29, 2019
236
77
28
Indeed. It’s a good problem to have.
I suspect that the CPU might have started life as a 7702P that had damaged CCDs. AMD would then fuse off the other CCDs and sell it as a 7302P. But it would have been binned much higher to start with ...

Coming from the Intel world where all the Xeons are power hogs and dissipate oodles of watts when pushed - this is a refreshing experience
 

Kev

Active Member
Feb 16, 2015
439
84
28
37
I can't figure out how to get the microSD card to read. I had some super duper microSD cards here and I wanted to install ESXI on them. Also, seems like IPMI mounting of ISO is super slow. At least the ESXI boot from IPMI was sooooo slow. I don't remember supermicro with the same image having this problem.
 

hmw

Active Member
Apr 29, 2019
236
77
28
I can't figure out how to get the microSD card to read. I had some super duper microSD cards here and I wanted to install ESXI on them. Also, seems like IPMI mounting of ISO is super slow. At least the ESXI boot from IPMI was sooooo slow. I don't remember supermicro with the same image having this problem.
The microSD card on the motherboard? Isn't that for IPMI logs?
 

Kev

Active Member
Feb 16, 2015
439
84
28
37
I assumed the microSD card was a usb bootable device but I see no options to save logs or anything to the SD card either.
 

hmw

Active Member
Apr 29, 2019
236
77
28
I assumed the microSD card was a usb bootable device but I see no options to save logs or anything to the SD card either.
If you look at the motherboard diagram - the MicroSD is linked to the ASPEED BMC, it could still be a boot device (and that would be awesome) but I doubt it ...

Screen Shot 2020-06-03 at 1.07.47 AM.png
 

Kev

Active Member
Feb 16, 2015
439
84
28
37
You are absolutely right. I didn’t think to look at the block diagram. Now I’m kinda bummed this board does not have sata dom power or internal usb.