Cheap 16-core build with IPMI: what's possible currently?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Marsh

Moderator
May 12, 2013
2,644
1,496
113
But since the bottom part of the server front is empty, I mounted three 80mm fans, so now the CPU is 32c idle and the whole thing is super quiet.

I have the same Norco RPC4308 Chassis
Do you have a picture of the 3 front 80mm FAN mount?
Could you describe how you mounted the 3 fans?

I like to mod the chassis like yours.

Thanks
 

hmw

Active Member
Apr 29, 2019
570
226
43
I have the same Norco RPC4308 Chassis
Do you have a picture of the 3 front 80mm FAN mount?
Could you describe how you mounted the 3 fans?

I like to mod the chassis like yours.

Thanks
I had planned to make a proper 3D printed fan wall cage - but having zero experience and knowledge of 3D printing, I figured it would take a lot of time for me to do this - and was looking for a temp solution in the meantime.

I simply used some thick PE foam - cut into a rectangular shape with cutouts for the fans and power/fan cables. The foam was sturdy enough (and rigid enough) to stay put. The foam width is almost the same width as the chassis, there's a 1" gap for all the cables i.e. most of the cabling runs out between the mesh and fans and then back to the power supply etc. This ensures that there's almost no cabling between the fans and motherboard.

As for mounting the the fans - they simply went into the cutouts. I used 2" thick foam, there's some double sided tape at the bottom and also at the bottom of each fan cutout - to make sure it and the fans doesn't move

Here's a pic from the top. And if you look at the server you can see the fans from the front

The fans are Arctic Cooling P8 PWM CO - this specific model because you can link the three fans together and have only one connection to the motherboard FAN connector

I'll post better pics over the weekend with the top HDD cage removed
 
  • Like
Reactions: Marsh

Billy1

New Member
Jun 21, 2020
14
5
3
Thanks zer0sum, that's $35 for the card with full and half height brackets + $20 for a Mellanox MAM1Q00A-QSA converter. I have a MS510TX so works perfectly with the switch's SFP+ port

This is a lot cheaper than buying onboard 10GbE which serves to increase the motherboard price by $150 -$200 over ones with standard GbE

Good afternoon quick question are you guys talking about the S8030 boards?? That you can add 10GbE to the non 10GbE board? I'm new with networking builds linux but i"m trying learn.. I would like to buy one but the price for the 10g board is out of my budget.. Thank You!!
 

hmw

Active Member
Apr 29, 2019
570
226
43
Good afternoon quick question are you guys talking about the S8030 boards?? That you can add 10GbE to the non 10GbE board? I'm new with networking builds linux but i"m trying learn.. I would like to buy one but the price for the 10g board is out of my budget.. Thank You!!

Just get some cheap Mellanox cards - HP 544QSFP MCX354A-FCBT 649281-B21 656089-001 VPI FDR 40GbE Mellanox OEMFirmware | eBay and you can connect them via QSFP to your switch. You can also use SFP 10Gbase-T modules if your switch uses 10Gb-T

I got the one linked to above for $35 off eBay. It’ll do 2 x 10GbE or 1 x 40 GbE
 

Billy1

New Member
Jun 21, 2020
14
5
3
Just get some cheap Mellanox cards - HP 544QSFP MCX354A-FCBT 649281-B21 656089-001 VPI FDR 40GbE Mellanox OEMFirmware | eBay and you can connect them via QSFP to your switch. You can also use SFP 10Gbase-T modules if your switch uses 10Gb-T

I got the one linked to above for $35 off eBay. It’ll do 2 x 10GbE or 1 x 40 GbE
Thanks for getting back to me so quick!! I'm going to order that board tomo!! I didnt have almost 600 for the 10g one.. So with adding these cards i will have the same performance as the 10g board?? lastly can i use the AMD 3900x in this board until I'm able to upgrade to 7302p upgrade? I already have the 3900x? As far as ram ECC UNBUFFERED What would you recommend?? Thank You!!
 

hmw

Active Member
Apr 29, 2019
570
226
43
Thanks for getting back to me so quick!! I'm going to order that board tomo!! I didnt have almost 600 for the 10g one.. So with adding these cards i will have the same performance as the 10g board?? lastly can i use the AMD 3900x in this board until I'm able to upgrade to 7302p upgrade? I already have the 3900x? As far as ram ECC UNBUFFERED What would you recommend?? Thank You!!
I'd recommend you read my thread about this build: https://forums.servethehome.com/index.php?threads/tyan-s8030gm2ne.28914/. The S8030 motherboard is a server oriented EPYC motherboard. It won't take Threadripper and AMD 39xx CPUs, only the AMD EPYC 7XX1 and 7XX2 CPUs. And these CPUs support eight channel memory and large amounts of RAM. But it has to be ECC RDIMM/LRDIMM, neither the board nor the CPUs accept unbuffered ECC DIMMs

If you already have the 3900x, you're better off with something like an AsrockRack X570D4I-2T. Newegg has it for $430. The 3900X, 3950x and Threadrippers will be faster than their corresponding core-count equivalent EPYC chips. What they wont have is support and validation when you're installing ESXi for example. If that's important or the number of PCIe slots are important - EPYC is the way to go. If those things don't matter to your use case, there's no real reason to get EPYC

Because the EPYC motherboards have 5 to 7 PCIe 4.0 slots, it is easy to add a dual 10G network card. One can get Intel X550-T2 cards off eBay for $150 to $200 and there's loads of other NICs (Mellanox, Solarflare, Broadcom, Chelsio) that will do 10G and are under $100 or even $50, it's all down to which OS you're deploying and what features you want support for
 
Last edited:

dandanio

Active Member
Oct 10, 2017
182
70
28
Hi

What OS are you running? Any issues so far? I'm eyeing this mobo for workstation build, but as everything workstation it might be repurposed with ESXi later. Really wanted 10Gb and mITX with Ryzen and this seems to be the only solution so far.
vanilla esxi7/vcenter7 vmug advantage. Absolutely no issues. I love this board.
 

Billy1

New Member
Jun 21, 2020
14
5
3
I'd recommend you read my thread about this build: https://forums.servethehome.com/index.php?threads/tyan-s8030gm2ne.28914/. The S8030 motherboard is a server oriented EPYC motherboard. It won't take Threadripper and AMD 39xx CPUs, only the AMD EPYC 7XX1 and 7XX2 CPUs. And these CPUs support eight channel memory and large amounts of RAM. But it has to be ECC RDIMM/LRDIMM, neither the board nor the CPUs accept unbuffered ECC DIMMs

If you already have the 3900x, you're better off with something like an AsrockRack X570D4I-2T. Newegg has it for $430. The 3900X, 3950x and Threadrippers will be faster than their corresponding core-count equivalent EPYC chips. What they wont have is support and validation when you're installing ESXi for example. If that's important or the number of PCIe slots are important - EPYC is the way to go. If those things don't matter to your use case, there's no real reason to get EPYC

Because the EPYC motherboards have 5 to 7 PCIe 4.0 slots, it is easy to add a dual 10G network card. One can get Intel X550-T2 cards off eBay for $150 to $200 and there's loads of other NICs (Mellanox, Solarflare, Broadcom, Chelsio) that will do 10G and are under $100 or even $50, it's all down to which OS you're deploying and what features you want support for

Thank you for the info!! That really helped me a lot!! I'm building a server to run a Cardano node.. I'm going to run ubuntu 20.04 I will look into the Mellanox cards.. Again Thank you I'm going to order the AsrockAack X570D4i-2t today and will keep you posted on the build!! Thanks Billy
 

hmw

Active Member
Apr 29, 2019
570
226
43
Thank you for the info!! That really helped me a lot!! I'm building a server to run a Cardano node.. I'm going to run ubuntu 20.04 I will look into the Mellanox cards.. Again Thank you I'm going to order the AsrockAack X570D4i-2t today and will keep you posted on the build!! Thanks Billy
@Billy1 - Provantage has the X570D4U-2L2T for $389 PROVANTAGE: ASRock Inc X570D4U-2L2T Asrock MB X570D4U-2L2T AMD Ryzen MAX128GB DDR4 PCIE uAtx. This is the mATX version. You're limited to 128GB of RAM but you get more PCIe slots and you don't get the 10GbE network.

This one has 1*PCIe x16 + 1*PCIe x1 OR 2*PCIe x8 + 1*PCIe x1
 

hmw

Active Member
Apr 29, 2019
570
226
43
I have the same Norco RPC4308 Chassis
Do you have a picture of the 3 front 80mm FAN mount?
Could you describe how you mounted the 3 fans?

I like to mod the chassis like yours.

Thanks
IMG_1318.jpeg

@Marsh - here's how it looks with now with the PE foam mockup. The height is the same as the front grill (with the top HDD rack removed) minus 1mm. There's cutouts for cables and the whole thing is mounted as near the motherboard as possible. Right now, I'm looking to see how to 3D print this thing - if it is too expensive, I might just go in for another case as I've decided to shift the storage to 2.5" SSDs as soon as feasible - so moving slowly to 8GB SATA SSDs when they become available

But with this mod, I can keep the HBA, the Mellanox card and the RAM - cooler by 8c-10c
 
  • Like
Reactions: Marsh

hmw

Active Member
Apr 29, 2019
570
226
43
I love what you did with the fans.

Do you think 92mm fan would fit?
A 92mm fan will fit - the case is 7"/178mm and the lower part is ~ 136mm. However my foam mockup wouldn't work with 92mm, since I didn't have the correct rigid, high density foam. I was thinking of cutting some wood to size and mocking it up that way - so it would look like the Norco fan wall cage.

Another addition I was planning was using a 120mm fan shroud on the front of the entire server like this guy. Especially since I switched to Server 2019 and my NAS HDDs don't spin down anymore (I use Drivepool and I've enabled all the settings on the 2308 HBA driver). But I would use 120mm x 15mm fans instead of the usual 25mm
 
  • Like
Reactions: TXAG26 and Marsh

TLN

Active Member
Feb 26, 2016
523
84
28
34
Provantage has the X570D4U-2L2T for $389 PROVANTAGE: ASRock Inc X570D4U-2L2T Asrock MB X570D4U-2L2T AMD Ryzen MAX128GB DDR4 PCIE uAtx. This is the mATX version. You're limited to 128GB of RAM but you get more PCIe slots and you don't get the 10GbE network.
Actually you do, hence -2T in the name.
I'm debating between those two boards for workstation. mATX switches to x8/x8 if 3rd slot is occupied, so it doesn't do much for me. If that was x16/x8 it would be no brainer.
 
  • Like
Reactions: hmw

hmw

Active Member
Apr 29, 2019
570
226
43
Actually you do, hence -2T in the name.
I'm debating between those two boards for workstation. mATX switches to x8/x8 if 3rd slot is occupied, so it doesn't do much for me. If that was x16/x8 it would be no brainer.
ah good catch. For $389 this board is almost perfect - paired with a 3950x. If this had been out I wouldn’t have built the EPYC based server ...
 

Spartacus

Well-Known Member
May 27, 2019
788
328
63
Austin, TX
Moreover, If you want current tech, the “cheapest” 16-core CPU available now is the Ryzen 9 3950X. 16 cores and 32 threads, retail for around € 780 (945.5 USD).
Yeah, not exactly what I’d call “cheap”… :)
Pretty sure the 5950x is 16c/32t and is 200$ less (750$ US) unless you get either used, that said I expect a new AMD cpu announcement soon so might be a new option soonish too.
 

TLN

Active Member
Feb 26, 2016
523
84
28
34
I saw a Xeon E5-2698 v3 for 700 Australian dollars, that's 540 US dollars. That's a 16 core / 32 thread CPU from 2014. When it came out it cost over 3,000 US dollars. Today it's not super great anymore, on Passmark it's ranked 145, equal to an I7-10700k. So still good but not brilliant.
Moreover, If you want current tech, the “cheapest” 16-core CPU available now is the Ryzen 9 3950X. 16 cores and 32 threads, retail for around € 780 (945.5 USD).
Yeah, not exactly what I’d call “cheap”… :)
I got mine (2698v3) for $200 or $250 here. Works great. Using with asrock mobo with integrated 10gbe/sas as homelab server.
 

nukke

New Member
Aug 5, 2021
14
0
1
Pretty sure the 5950x is 16c/32t and is 200$ less (750$ US) unless you get either used, that said I expect a new AMD cpu announcement soon so might be a new option soonish too.
I can't find a 3950X in stock on any of the big retailers (Amazon, Newegg, B&H, Walmart, Office Depot, Staples, etc). They're going for ~$500 USD used on eBay. That's not a bad deal for a 16c/32t CPU. I'm also finding X470D4U-2T boards for ~$240 on eBay.

The most expensive part of this setup is by far RAM. Unfortunately RDIMMs are not supported by the board (or is it the Ryzen chips themselves?) and ECC UDIMMs are almost double the price of RDIMMs.
 

hmw

Active Member
Apr 29, 2019
570
226
43
I saw a Xeon E5-2698 v3 for 700 Australian dollars, that's 540 US dollars. That's a 16 core / 32 thread CPU from 2014. When it came out it cost over 3,000 US dollars. Today it's not super great anymore, on Passmark it's ranked 145, equal to an I7-10700k. So still good but not brilliant.
Moreover, If you want current tech, the “cheapest” 16-core CPU available now is the Ryzen 9 3950X. 16 cores and 32 threads, retail for around € 780 (945.5 USD).
Yeah, not exactly what I’d call “cheap”… :)

The post you're replying to was in April 2020. Thanks to folks on the forum - I managed to snag a 7302P for ~ $680 before the prices went up [ My post on the build is here if you're interested: https://forums.servethehome.com/index.php?threads/tyan-s8030gm2ne.28914/]

Right now prices are insane due to the chip shortage and low to zero inventory on many things like motherboards, SSDs, memory etc. I do regret not buying 256GB of RAM, but like Spartacus said - AMD will come out with the newer CPUs soonish so waiting to build another system on that
 

Tom S

Member
Jan 31, 2017
42
20
8
38
I can't find a 3950X in stock on any of the big retailers (Amazon, Newegg, B&H, Walmart, Office Depot, Staples, etc). They're going for ~$500 USD used on eBay. That's not a bad deal for a 16c/32t CPU. I'm also finding X470D4U-2T boards for ~$240 on eBay.

The most expensive part of this setup is by far RAM. Unfortunately RDIMMs are not supported by the board (or is it the Ryzen chips themselves?) and ECC UDIMMs are almost double the price of RDIMMs.
3950X has been superseded by the 5950X so I don't know if 3950X is still being produced.

If someone is looking for a budget build I would start with an X470D4U and a $280 Ryzen 5600X. The 5600X is only 6-core, but it performs similarly to the E5 2698v3 CPU listed previously. Upgrade to a 16-core 5950X later when they're cheap.

The hidden advantage of these Ryzen server builds is the high turbo clock speeds. The 5950X's clock speed allows it to outperform some of the more expensive Threadripper Pro and EPYC parts depending on the benchmarks: AMD Threadripper Pro Review: An Upgrade Over Regular Threadripper?

For my purposes (mostly software development, not really limited by memory bandwidth) these Ryzen Zen 3 servers are an incredible value.