Cheap 16-core build with IPMI: what's possible currently?

hmw

Active Member
Apr 29, 2019
236
77
28
Trying to build a modern (TRX40, x570, PCIe 4.0 etc) 16-core rackmounted server for my home lab and network and would welcome input. The setup would run ESXi and/or Hyper-V & Storage Spaces, several Linux VMs - the bonus being if it is able to run a Windows 10 VM with a GPU for passthrough (to stream games). IPMI is a must as the rack is in the basement. Also 10GbE or NBASE-T is another must-have since it is hooked up to a multi-gig switch and several nodes are connected at 5 gigabit.

Initially I had considered the Ryzen 3950x and Xeon-W - what I found was that:
  • Ryzen motherboards with IPMI just aren't that common and neither are ThreadRipper ones
  • Intel Xeon-W and other chips are just too expensive per-core
  • Going for EPYC ones seems the logical choice considering a good x570 motherboard costs just 25% less - to approximately the same as a server motherboard
  • ECC enabled UDIMMs are costlier than if not the same price as RDIMMs
There's the AsRock x570 which doesn't seem to be available for sale anywhere and I am guessing would cost $500-$600 https://www.asrockrack.com/general/productdetail.asp?Model=X570D4I-2T (UPDATE - will cost $490 and be generally available some time in June 2020)

AsRock also has the Rome motherboard ASRock Rack > ROMED8-2T - same problems, not available anywhere

Then we come to Supermicro and TYAN. Supermicro has H12 motherboards at last - H12SSL-NT | Motherboards | Super Micro Computer, Inc., but it's "coming soon"

Tyan has the S8030 range which has IPMI and looks quite decent actually without the oculink stuff that Supermicro wanted to put on all its new motherboards - https://www.tyan.com/Motherboards_S8030_S8030GM4NE-2T

CPU: I'm considering the AMD EPYC 7282. It has only half the memory bandwidth BUT costs $700 retail. It actually is the cheapest $/core amongst EPYC processors - thanks to this brilliant analysis https://www.servethehome.com/amd-epyc-7002-sku-list-and-value-analysis/. The equivalent single processor 7302P is $900 retail !

MOTHERBOARD: Considering the Tyan S8030 because I don't know how long the SuperMicro will take to show up in the channel https://www.tyan.com/Motherboards_S8030_S8030GM4NE-2T. I'd expect this to be $550

RAM: would start off with 4 x 16GB DDR4-3200 ECC RDIMMs - or 4 x 32GB ones - the EPYC 7278 is 1/2 the bandwidth so will it benefit from populating all 8 channels?

CASE: NorcoTek RPC-4308. Eight drive bays, supports ATX and an ATX PSU (side mounted). Also has a 2 x SFF-8087 backplane for the eight drives

PSU: SilverStone 850W Platinum

Storage: Reusing my 6 x 4TB array, at some point want to put some NVMe drives and convert to tiered Storage Spaces


Questions - what is everyone's opinion on TYAN vs Supermicro? Is their IPMI as good or worse than Supermicro? What about BIOS quality and support? Running Hyper-V or ESXi on consumer grade motherboards is never fun but it gets progressively worse on server grade motherboards if the manufacturers don't expose the correct BIOS settings ...

Would welcome everyone's input !
 
Last edited:

TLN

Active Member
Feb 26, 2016
403
55
28
30
I'm waiting that that mobo as well. Everything else is ready for 10g.
If you need more performance, you can look at:
X399D8A-2T: ASRock Rack > X399D8A-2T
Threadripper with 10Gb ports and lots pf PCIE slots.

I'd check if you need more then 128Gb of memory (I think that's Ryzen limit). If you don't there're lots of options. Also, you may consider two boxes: one on older Xeon E5- v1/v2 CPU with lots of memory for lab, and some faster machine for gaming VM and other CPU-intensive tasks.
 

hmw

Active Member
Apr 29, 2019
236
77
28
I'm waiting that that mobo as well. Everything else is ready for 10g.
If you need more performance, you can look at:
X399D8A-2T: ASRock Rack > X399D8A-2T
Threadripper with 10Gb ports and lots pf PCIE slots.
The X399 is ~ $580 and a generation older.

I just found out that the X570 Asrock is out June 2020 and will cost $490. The problem is that [a] I don't know how well ESXi and Hyper-V will handle Ryzen & x570 - and the damn OCulink is a mystery. I have a SFF 8087 backplane and can't seem to find a OCulink (Host) to SFF8087 (Target) cable
 

TLN

Active Member
Feb 26, 2016
403
55
28
30
The X399 is ~ $580 and a generation older.

I just found out that the X570 Asrock is out June 2020 and will cost $490. The problem is that [a] I don't know how well ESXi and Hyper-V will handle Ryzen & x570 - and the damn OCulink is a mystery. I have a SFF 8087 backplane and can't seem to find a OCulink (Host) to SFF8087 (Target) cable

1. X399d8a-2t is $450 on amazon right now.
2. Mind you asking where did u get info about x570d4i-2t? I remember reading april 2020 and haven't seen price yet.
3. Agree on the rest. Id use ryzen as workstation, but take intel for server. With no support id rater take proven platform.

Ps. Local store have trx40 mobo for $250-260
 

zer0sum

Active Member
Mar 8, 2013
389
144
43
I really want to love AMD, and especially that EPYC 7282, but the motherboard choices suck so hard!!
And it's been like that for a long time unfortunately :(

I wasn't really looking to upgrade my X9 setups, but I just jumped on a deal for the SuperMicro X10SRH-CF for $135 off Ebay.
It only has 4 x PCI-E 3.0, and 2 x PCIE 2.0, but has a LOT of stuff onboard in an ATX form factor including dual 10Gbps Intel NIC's, and an LSI 3008 SAS3 controller with 8 x SAS3 ports :)

Might be worth considering if you can live with a Xeon v3/v4 :D

 
Last edited:

TXAG26

Active Member
Aug 2, 2016
274
81
28
That's a nice board, I have a similar one. Do note, it has the Intel i350 network controller and it is not 10Gbt. The max the i350 can do is 1Gbt. Supermicro makes an inexpensive Intel X550 adapter that works well with this board. It comes in 1 and 2 port versions if you're looking for ethernet. Various fiber options work as well.
 

zer0sum

Active Member
Mar 8, 2013
389
144
43
That's a nice board, I have a similar one. Do note, it has the Intel i350 network controller and it is not 10Gbt. The max the i350 can do is 1Gbt. Supermicro makes an inexpensive Intel X550 adapter that works well with this board. It comes in 1 and 2 port versions if you're looking for ethernet. Various fiber options work as well.
My reading skills suck today.

I run Mellanox CX3 cards so I'm fine with that anyways :)
$25 for a HP649281-B21 that you can reflash back to standard Mellanox firmware
 
Last edited:
  • Like
Reactions: hmw

Spartacus

Well-Known Member
May 27, 2019
688
265
63
Austin, TX
^ I have the quad port version of the above board for my unraid setup (can flash the sas controller to it mode) its fantastic paired with a 2660 v3 relatively high base clocks 10c/20t with a decent turbo and costs <100$ regularly for the cpu. I also use the CX3 cards for my 10g connection.

If you need more 16x slots (16x16x0x8 or 16x8x8x8), my backup board is the X10SRA-F which is only lacking SAS (easily fixed by a H310 addon)

Both of the X10SRH boards have bifurcation support as well for nvme cards (grabbed one of the 2x cards from supermicro yesterday for $35 to replace my raid10 vm datastore Supermicro aoc-slg3-2m2 pcie adapter for nvme ssds | eBay) .
 

hmw

Active Member
Apr 29, 2019
236
77
28
I really want to love AMD, and especially that EPYC 7282, but the motherboard choices suck so hard!!
And it's been like that for a long time unfortunately :(

I wasn't really looking to upgrade my X9 setups, but I just jumped on a deal for the SuperMicro X10SRH-CF for $135 off Ebay.
It only has 4 x PCI-E 3.0, and 2 x PCIE 2.0, but has a LOT of stuff onboard in an ATX form factor including dual 10Gbps Intel NIC's, and an LSI 3008 SAS3 controller with 8 x SAS3 ports :)

Might be worth considering if you can live with a Xeon v3/v4 :D
That is something to consider - I generally don't buy off eBay since I've had some bad luck in the past - but look at ebay, the Xeon E5-2690v4 is around $725 to $850, the X10SRH-CF is $200 and X550-T2 10GbE adapters cost $200 - $300. The CPU + Motherboard combo therefore costs $1150 to $1350

I'm contrasting that with the Tyan S8030/EPYC 7282 combo that costs $1250 new @ retail - and supports DDR4-3200. Note that the Tyan doesnt have the SAS 3008 card but the Supermicro one will.

So choices appear to be

1. Take a chance on eBay and get a Xeon E5 V4
2. Take a chance on Tyan and get the EPYC 7282
3. Wait for Supermicro to release H12 in June 2020 (although there's a fair bit of price gouging during the first month)
 

zer0sum

Active Member
Mar 8, 2013
389
144
43
The whole point of going E5 v3/v4 is really to get a bargain :)
Otherwise I'd go for the new EPYC setup

X10SRH-CF has been bought on offer for $115-135
E5-2680 v4 - 14C/28T @ 2400-3300Mhz for $298
Mellanox CX3 10/40/56Gbps dual port NIC (HP649281-B21) - $30-50

I'm not sure what your workloads are, but I never notice much appreciable difference with DDR speeds.
 
Last edited:

hmw

Active Member
Apr 29, 2019
236
77
28
The 7302p can be had for a lot less than $900+.

PROVANTAGE: HPE P16667-B21 DL325 GEN10 AMD Epyc 7302P Upgkit

$678+tax new at Provantage via an HPE processor upgrade kit. A couple of us on the board have ordered these and they work great.
Damn this is why I love this board. Thanks TXAG26, ordered one immediately - they're out of stock, so hoping they will honor the price and ship one

Looks like I've decided to jump on the EPYC train :)

So Tyan or SuperMicro it is ...
 

hmw

Active Member
Apr 29, 2019
236
77
28
The whole point of going E5 v3/v4 is really to get a bargain :)
X10SRH-CF has been bought on offer for $115-135
E5-2680 v4 - 14C/16T @ 2400-3300Mhz for $298
Still cannot find the X10SRH-CF but that's an awesome price for the E5-2680 v4 ... might just buy 2-4 to upgrade my servers @ work

Mellanox CX3 10/40/56Gbps dual port NIC (HP649281-B21) - $30-50
That is super cheap ! Found a MC2207130 cable on eBay for $20 as well

I'm not sure what your workloads are, but I never notice much appreciable difference with DDR speeds.
Workloads are varied: home automation, media server (SMB and no transcoding) and Linux VMs running Python & Java code for numerical analysis ... I don't have very large data sets, so 128GB should be sufficient

OTOH, lately, I haven't seen much difference between DD4 2666,2993 and 3200 memory. I believe 16GB ECC RDIMMs are ~ $95 for all speeds
 

Spartacus

Well-Known Member
May 27, 2019
688
265
63
Austin, TX
That is something to consider - I generally don't buy off eBay since I've had some bad luck in the past - but look at ebay, the Xeon E5-2690v4 is around $725 to $850, the X10SRH-CF is $200 and X550-T2 10GbE adapters cost $200 - $300. The CPU + Motherboard combo therefore costs $1150 to $1350
Why a V4 instead of a V3? the E5-2690 V3 is only $200 be it may the epyc is about 2x the compute.
Best of luck with it!
 

hmw

Active Member
Apr 29, 2019
236
77
28
My reading skills suck today.

I run Mellanox CX3 cards so I'm fine with that anyways :)
$25 for a HP649281-B21 that you can reflash back to standard Mellanox firmware
Thanks zer0sum, that's $35 for the card with full and half height brackets + $20 for a Mellanox MAM1Q00A-QSA converter. I have a MS510TX so works perfectly with the switch's SFP+ port

This is a lot cheaper than buying onboard 10GbE which serves to increase the motherboard price by $150 -$200 over ones with standard GbE
 
Last edited:

Kev

Active Member
Feb 16, 2015
439
84
28
37
Thinking about getting the tyan board for virtualization and we wondering if you can set cTDP to something lower than 120W.