Cheap 16-core build with IPMI: what's possible currently?

vangoose

Active Member
May 21, 2019
268
70
28
Canada
what are the cooling options for these chips? any water cooling available?
I use Supermicro heatsink, the cpu runs around 40C.
I'm using this: Supermicro SNK-P0064AP4 4U Active CPU Heat Sink For AMD EPYC OLGA4094 SP3 Socket. It is oriented the correct way so blows out of the case and not onto the DIMM slots like the Noctua

Waiting for PSU cables - that's the only thing missing now. I will update this thread with build details, temps, ESXi install + passthrough experience etc
I use the same heatsink on my SM board, CPU runs around 40C idle, around 50C under typical load to run a few storage VMs with pci passthrough.
 

SINN78

Active Member
Apr 3, 2016
152
30
28
41
Plenty of AIO or Custom water, you just need the right case with rad/pump space for it.
not sure what case i'll be going with i have a norco rpc-470 my current system is in a lian li pc-p80


I'm using this: Supermicro SNK-P0064AP4 4U Active CPU Heat Sink For AMD EPYC OLGA4094 SP3 Socket. It is oriented the correct way so blows out of the case and not onto the DIMM slots like the Noctua

Waiting for PSU cables - that's the only thing missing now. I will update this thread with build details, temps, ESXi install + passthrough experience etc
Nice please let me know how loud it is i have my server in my bedroom :(
 

hmw

Active Member
Apr 29, 2019
236
77
28
not sure what case i'll be going with i have a norco rpc-470 my current system is in a lian li pc-p80

Nice please let me know how loud it is i have my server in my bedroom :(
I plan to put this in a Norco RPC-4308. It's a short depth chassis but accommodates ATX motherboards. I'll need to replace the front USB with something like this though: Amazon.com: Toptekits 2 Port USB 3.0 A Female Screw Panel Mount to Motherboard 20 Pin Flat Cable Cord by Toptekits 1.6ft/0.5m: Computers & Accessories

Even though it is in the basement, I'm using Artic Cooling P8 fans for exhaust - so it should be quite quiet
 
  • Like
Reactions: SINN78

nasi

Member
Feb 25, 2020
32
8
8
Anybody knows how much power the combination of ASRock Rack X470D4U and 3950X versus Epyc 7302P and fitting Tyan/SM/... Board in comparable configuration draws?
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,269
428
83
  • Like
Reactions: nasi

zer0sum

Active Member
Mar 8, 2013
389
144
43
@hmw - did you manage to get this up and running with ESXi yet?

I'm really curious how all of the hardware works with ESXi 7.0 and what all of the device pass through options end up looking like :D
 

vangoose

Active Member
May 21, 2019
268
70
28
Canada
My 7302P server power consumption.
powercon.JPG

Device fully populated running ESXi 6.7U3, should have no problem to run 7.0.
- 7302 P
- 4*32G DDR4-3200
- 8*10TB HGST SAS disks
- Intel X550-T2
- QLogic QL41212 dual 25gb port
- Mellanox MCX4121A-ACAT dual 25gb port
- 2*6.4TB HGST SN260 nvme AIC (25w)
- Optane 905P
- Samsung 970 pro
- 2*64G SM Satadom
- SAS backplane in the case.


Workloads
- 1 Domain controller
- 1 Windows VM to backup CIFS
- 2 vCenter appliance
- 1 NSX-T Manager
- 1 FreeNAS Storage VM for NAS with 10TB disks
- 1 FreeNAS Storage VM for NVME disk for VMs iSCSI
- 1 Linux Storage VM for iSER or Solaris 11 testing
 
  • Like
Reactions: nasi

hmw

Active Member
Apr 29, 2019
236
77
28
@hmw - did you manage to get this up and running with ESXi yet?

I'm really curious how all of the hardware works with ESXi 7.0 and what all of the device pass through options end up looking like :D
I wondering whether to create a new thread or post it all in this thread :D

But what I can tell you quickly is:

- Using EPYC 7302P from the HP DL385 Upgrade kit
- Using 8 x Samsung 16GB DDR4-3200 RDIMMs (M393A2K43DB3-CWE | RDIMM | Samsung Module | Semiconductor)
- Network card is a Mellanox ConnectX-3
- Passthrough IO controller is LSI2308
- Passthrough GPU is GTX 1080 Mini

All the above is working with ESXi 6.7u3. Managed to get a TimeSpy benchmark run on a Windows 2004 VM, it was 5% higher than a what I got with the card attached as an eGPU to my 2.8 GHz quad core MacBook Pro. Also managed to install a Windows 2019 Server VM with the LSI2308 passed through. Still need to attach a drive or two and test out whether the 2308 is stable.

Sadly installing ESXi 7.0 was a no-go. It gave 'decompression' errors and I checked the MD5 & SHA checksums to make sure the ISO was not corrupted.

First thoughts on Tyan and their customer service: Weird

- I asked them where I could get the SlimSAS-8i to 2xNVMe cable and they responded with: 'it's a non-standard cable, what would you possibly want with this?' :rolleyes:

- I also asked them to correct an error on their website stating support for Dual Rank Samsung 16GB RDIMMs - M393A2K40DB2-CWE. Except that it is a SINGLE rank RDIMM and not dual rank

Temps: I'm seeing 31c, ESXi doesn't show me anything other than CPU_Tctl_value. I have a Supermicro 4U heatsink and fan on it. The FAN controls on the Tyan motherboard are close to non-existent (more on that later)

Over the weekend I will test out the Windows VMs in greater detail (ordered a dummy HDMI plug to keep the GPU happy) but also need to work on modding PSU cables to fit in my RPC-4308 server case. At the time, the AX1200i seemed like a good deal- now I am afraid I might have to get another PSU if that doesn't fit :(

I've written down what I did to make passthrough work - so will put that up as well ...

IMG_1227.jpg
 
  • Like
Reactions: SINN78

hmw

Active Member
Apr 29, 2019
236
77
28
Oh and a couple of gotchas:

If you want to flash any SAS/LSI controllers - do yourself a favor and get them flashed before you get the motherboard :)

And when you login to the BMC/IPMI for the first time with the default username and password - it will ask you to create a new password. Do yourself another big favor and DO NOT select a long complex password. If you're lucky it will reject it. If you're unlucky, it might truncate it and you're left with a BMC you cannot access because the password doesn't work. Luckily I had a VGA monitor and was able to enter BIOS to change the BMC password
 
Last edited:

hmw

Active Member
Apr 29, 2019
236
77
28
looks very nice thanks for all the tips how loud is the cooler?
Cooler is very quiet. However - this is the fan control for the Tyan


tyan_fan_control.jpeg

There's just two settings: Manual and Full Speed

Manual defaults to 30% and I would say it is less noticeable than my MacBook Pro at full speed. But setting the duty cycle to 40% causes it to be much more noticeable

What I really don't like is that the fan doesn't seem to speed up - I will do some load testing and check what's going on ...
 
  • Like
Reactions: SINN78

dandanio

Active Member
Oct 10, 2017
147
50
28
Just fyi, I went the AMD route and here is what I built:
  1. Fractal Design Core 500 Black
  2. EVGA SuperNOVA 750 GA, 220-GA-0750-X1
  3. X570D4I-2T, purchased on Newegg
  4. AMD Ryzen 9 3950X
  5. 64GB Kit 2 x 32GB DDR4-2933 PC4-23400 ECC Sodimm: 9SIA7S6BAF9355 (Newegg) (with space for 64 GB more)
  6. be quiet! BK009 PURE ROCK CPU Cooler - 150W TDP
  7. WD Black 250GB High-Performance NVMe PCIe Internal - M.2 2280, 8 Gb/s - WDS250G2X0C
    and:
  8. Corsair Processor Cooler Mounting Kit Cooling CW-8960010
My favorite features:
2x 10Gbe RJ-45
IPMI
16 cores, 32 threads
UP TO 128GB of ECC RAM
All for under $2000 bnib.

It has been chillin' in my basement for a few days now. So far, I love it.
 
  • Like
Reactions: hmw

Tom S

New Member
Jan 31, 2017
13
5
3
34
I really like that Mini-ITX X570 server board. Being able to put 16 cores and 128GB of ECC RAM into a small form factor case would be a fun build.

If I didn't have to fit six hard drives into my case, I would have used the X570D4I-2T and a small case like the NCASE M1 for the ultimate tiny server build.
 

dandanio

Active Member
Oct 10, 2017
147
50
28
I really like that Mini-ITX X570 server board. Being able to put 16 cores and 128GB of ECC RAM into a small form factor case would be a fun build.

If I didn't have to fit six hard drives into my case, I would have used the X570D4I-2T and a small case like the NCASE M1 for the ultimate tiny server build.
With 2 oculink ports you can marry 8 hard drives with this board. :) (One breakout cable is included).
 

Tom S

New Member
Jan 31, 2017
13
5
3
34
With 2 oculink ports you can marry 8 hard drives with this board. :) (One breakout cable is included).
Fitting 6-8 hard drives into a decent Mini ITX case is the real problem.

The only reason I'd choose the X570 board over the X470 board at this point is for the onboard 10G ports or if I wanted to fit it into a Mini ITX case. With 6+ hard drives, you're looking at a bigger case anyway, the size advantage is wasted. The onboard 10G ports are nice, but I prefer SFP+ to keep power usage down so I run add-in cards anyway.

The X470 chipset is also less power hungry and runs cooler than the X570 chipset. Plus the X470 board is significantly cheaper.
 
  • Like
Reactions: dandanio and JGC

hmw

Active Member
Apr 29, 2019
236
77
28
Just fyi, I went the AMD route and here is what I built:
  1. Fractal Design Core 500 Black
  2. EVGA SuperNOVA 750 GA, 220-GA-0750-X1
  3. X570D4I-2T, purchased on Newegg
  4. AMD Ryzen 9 3950X
  5. 64GB Kit 2 x 32GB DDR4-2933 PC4-23400 ECC Sodimm: 9SIA7S6BAF9355 (Newegg) (with space for 64 GB more)
  6. be quiet! BK009 PURE ROCK CPU Cooler - 150W TDP
  7. WD Black 250GB High-Performance NVMe PCIe Internal - M.2 2280, 8 Gb/s - WDS250G2X0C
    and:
  8. Corsair Processor Cooler Mounting Kit Cooling CW-8960010
My favorite features:
2x 10Gbe RJ-45
IPMI
16 cores, 32 threads
UP TO 128GB of ECC RAM
All for under $2000 bnib.

It has been chillin' in my basement for a few days now. So far, I love it.
I went with the Tyan S8030GM2NE & 7302P simply because I got some amazing deals on both. The 7302P was $680 and the S8030 was $380. The 3950x is definitely much faster than the 7302P for single core/single threaded tasks, but the EPYC is faster for multi-core.
 
  • Like
Reactions: dandanio

dandanio

Active Member
Oct 10, 2017
147
50
28
I went with the Tyan S8030GM2NE & 7302P simply because I got some amazing deals on both. The 7302P was $680 and the S8030 was $380. The 3950x is definitely much faster than the 7302P for single core/single threaded tasks, but the EPYC is faster for multi-core.
I like your build, but it is far from mitx I was going for. :)
 

dandanio

Active Member
Oct 10, 2017
147
50
28
Fitting 6-8 hard drives into a decent Mini ITX case is the real problem.
I love the mitx form factor, my gaming PC, my esxi server(s), my file server and a few others are just that. With that said, the DS380 is the case to have. :) Highly recommended!
 

hmw

Active Member
Apr 29, 2019
236
77
28
I like your build, but it is far from mitx I was going for. :)
Actually I was ONLY looking at Mini-ITX motherboards because I have a short depth rack and my case is ~ 15" (it's a Norco RPC 4308).

But after measuring (like 20 times) and taking a (huge) gamble I went with the ATX sized Tyan. It *almost* didn't work out as I had to ditch my AX1200i PSU, use a GTX1080 Mini instead of a full sized card, make do with only one NVMe drive ... and mount the CPU fan backwards. The chassis has a HDD cage that made life difficult.

case_top.jpg

At this point, if I had to do it all over again, I would ditch the cage, replace the 5 x 4TB HDDs I have with 4 x 7.68TB SSDs - these don't need to be in a cage because, how many times do you replace an SSD?

But since the bottom part of the server front is empty, I mounted three 80mm fans, so now the CPU is 32c idle and the whole thing is super quiet.

Not that I care as it's all down in the basement. The good thing is that I now have my hyperconverged machine: running ESXi with one Windows 10 VM for gaming, another VM for Linux & ML, and finally a Windows Server VM with a HBA passed through to DrivePool.

I use Moonlight to stream the games from the gaming VM to my Mac or the TV - the GTX1080 is plenty fast for now ...

You can see the rack below

rack_front.jpg

At some point when Zen3/Milan comes out, I would be tempted to replace the 7302P with either a 7402P (assuming it is cheaper) or a Milan based CPU - assuming it is faster/consumes less power

How's your experience with AsRock IPMI? Tyan's IPMI is barebones but the thing is exceptionally stable.
 
  • Like
Reactions: Marsh and dandanio

TLN

Active Member
Feb 26, 2016
403
55
28
30
  1. X570D4I-2T, purchased on Newegg
  2. AMD Ryzen 9 3950X
  3. 64GB Kit 2 x 32GB DDR4-2933 PC4-23400 ECC Sodimm: 9SIA7S6BAF9355 (Newegg) (with space for 64 GB more)
Hi

What OS are you running? Any issues so far? I'm eyeing this mobo for workstation build, but as everything workstation it might be repurposed with ESXi later. Really wanted 10Gb and mITX with Ryzen and this seems to be the only solution so far.