Cheap 16-core build with IPMI: what's possible currently?

Marsh

Moderator
May 12, 2013
2,294
1,109
113
But since the bottom part of the server front is empty, I mounted three 80mm fans, so now the CPU is 32c idle and the whole thing is super quiet.

I have the same Norco RPC4308 Chassis
Do you have a picture of the 3 front 80mm FAN mount?
Could you describe how you mounted the 3 fans?

I like to mod the chassis like yours.

Thanks
 

hmw

Active Member
Apr 29, 2019
236
77
28
I have the same Norco RPC4308 Chassis
Do you have a picture of the 3 front 80mm FAN mount?
Could you describe how you mounted the 3 fans?

I like to mod the chassis like yours.

Thanks
I had planned to make a proper 3D printed fan wall cage - but having zero experience and knowledge of 3D printing, I figured it would take a lot of time for me to do this - and was looking for a temp solution in the meantime.

I simply used some thick PE foam - cut into a rectangular shape with cutouts for the fans and power/fan cables. The foam was sturdy enough (and rigid enough) to stay put. The foam width is almost the same width as the chassis, there's a 1" gap for all the cables i.e. most of the cabling runs out between the mesh and fans and then back to the power supply etc. This ensures that there's almost no cabling between the fans and motherboard.

As for mounting the the fans - they simply went into the cutouts. I used 2" thick foam, there's some double sided tape at the bottom and also at the bottom of each fan cutout - to make sure it and the fans doesn't move

Here's a pic from the top. And if you look at the server you can see the fans from the front

The fans are Arctic Cooling P8 PWM CO - this specific model because you can link the three fans together and have only one connection to the motherboard FAN connector

I'll post better pics over the weekend with the top HDD cage removed
 
  • Like
Reactions: Marsh

Billy1

New Member
Jun 21, 2020
9
2
3
Thanks zer0sum, that's $35 for the card with full and half height brackets + $20 for a Mellanox MAM1Q00A-QSA converter. I have a MS510TX so works perfectly with the switch's SFP+ port

This is a lot cheaper than buying onboard 10GbE which serves to increase the motherboard price by $150 -$200 over ones with standard GbE

Good afternoon quick question are you guys talking about the S8030 boards?? That you can add 10GbE to the non 10GbE board? I'm new with networking builds linux but i"m trying learn.. I would like to buy one but the price for the 10g board is out of my budget.. Thank You!!
 

hmw

Active Member
Apr 29, 2019
236
77
28
Good afternoon quick question are you guys talking about the S8030 boards?? That you can add 10GbE to the non 10GbE board? I'm new with networking builds linux but i"m trying learn.. I would like to buy one but the price for the 10g board is out of my budget.. Thank You!!

Just get some cheap Mellanox cards - HP 544QSFP MCX354A-FCBT 649281-B21 656089-001 VPI FDR 40GbE Mellanox OEMFirmware | eBay and you can connect them via QSFP to your switch. You can also use SFP 10Gbase-T modules if your switch uses 10Gb-T

I got the one linked to above for $35 off eBay. It’ll do 2 x 10GbE or 1 x 40 GbE
 

Billy1

New Member
Jun 21, 2020
9
2
3
Just get some cheap Mellanox cards - HP 544QSFP MCX354A-FCBT 649281-B21 656089-001 VPI FDR 40GbE Mellanox OEMFirmware | eBay and you can connect them via QSFP to your switch. You can also use SFP 10Gbase-T modules if your switch uses 10Gb-T

I got the one linked to above for $35 off eBay. It’ll do 2 x 10GbE or 1 x 40 GbE
Thanks for getting back to me so quick!! I'm going to order that board tomo!! I didnt have almost 600 for the 10g one.. So with adding these cards i will have the same performance as the 10g board?? lastly can i use the AMD 3900x in this board until I'm able to upgrade to 7302p upgrade? I already have the 3900x? As far as ram ECC UNBUFFERED What would you recommend?? Thank You!!
 

hmw

Active Member
Apr 29, 2019
236
77
28
Thanks for getting back to me so quick!! I'm going to order that board tomo!! I didnt have almost 600 for the 10g one.. So with adding these cards i will have the same performance as the 10g board?? lastly can i use the AMD 3900x in this board until I'm able to upgrade to 7302p upgrade? I already have the 3900x? As far as ram ECC UNBUFFERED What would you recommend?? Thank You!!
I'd recommend you read my thread about this build: https://forums.servethehome.com/index.php?threads/tyan-s8030gm2ne.28914/. The S8030 motherboard is a server oriented EPYC motherboard. It won't take Threadripper and AMD 39xx CPUs, only the AMD EPYC 7XX1 and 7XX2 CPUs. And these CPUs support eight channel memory and large amounts of RAM. But it has to be ECC RDIMM/LRDIMM, neither the board nor the CPUs accept unbuffered ECC DIMMs

If you already have the 3900x, you're better off with something like an AsrockRack X570D4I-2T. Newegg has it for $430. The 3900X, 3950x and Threadrippers will be faster than their corresponding core-count equivalent EPYC chips. What they wont have is support and validation when you're installing ESXi for example. If that's important or the number of PCIe slots are important - EPYC is the way to go. If those things don't matter to your use case, there's no real reason to get EPYC

Because the EPYC motherboards have 5 to 7 PCIe 4.0 slots, it is easy to add a dual 10G network card. One can get Intel X550-T2 cards off eBay for $150 to $200 and there's loads of other NICs (Mellanox, Solarflare, Broadcom, Chelsio) that will do 10G and are under $100 or even $50, it's all down to which OS you're deploying and what features you want support for
 
Last edited:

dandanio

Active Member
Oct 10, 2017
147
50
28
Hi

What OS are you running? Any issues so far? I'm eyeing this mobo for workstation build, but as everything workstation it might be repurposed with ESXi later. Really wanted 10Gb and mITX with Ryzen and this seems to be the only solution so far.
vanilla esxi7/vcenter7 vmug advantage. Absolutely no issues. I love this board.
 

Billy1

New Member
Jun 21, 2020
9
2
3
I'd recommend you read my thread about this build: https://forums.servethehome.com/index.php?threads/tyan-s8030gm2ne.28914/. The S8030 motherboard is a server oriented EPYC motherboard. It won't take Threadripper and AMD 39xx CPUs, only the AMD EPYC 7XX1 and 7XX2 CPUs. And these CPUs support eight channel memory and large amounts of RAM. But it has to be ECC RDIMM/LRDIMM, neither the board nor the CPUs accept unbuffered ECC DIMMs

If you already have the 3900x, you're better off with something like an AsrockRack X570D4I-2T. Newegg has it for $430. The 3900X, 3950x and Threadrippers will be faster than their corresponding core-count equivalent EPYC chips. What they wont have is support and validation when you're installing ESXi for example. If that's important or the number of PCIe slots are important - EPYC is the way to go. If those things don't matter to your use case, there's no real reason to get EPYC

Because the EPYC motherboards have 5 to 7 PCIe 4.0 slots, it is easy to add a dual 10G network card. One can get Intel X550-T2 cards off eBay for $150 to $200 and there's loads of other NICs (Mellanox, Solarflare, Broadcom, Chelsio) that will do 10G and are under $100 or even $50, it's all down to which OS you're deploying and what features you want support for

Thank you for the info!! That really helped me a lot!! I'm building a server to run a Cardano node.. I'm going to run ubuntu 20.04 I will look into the Mellanox cards.. Again Thank you I'm going to order the AsrockAack X570D4i-2t today and will keep you posted on the build!! Thanks Billy
 

hmw

Active Member
Apr 29, 2019
236
77
28
Thank you for the info!! That really helped me a lot!! I'm building a server to run a Cardano node.. I'm going to run ubuntu 20.04 I will look into the Mellanox cards.. Again Thank you I'm going to order the AsrockAack X570D4i-2t today and will keep you posted on the build!! Thanks Billy
@Billy1 - Provantage has the X570D4U-2L2T for $389 PROVANTAGE: ASRock Inc X570D4U-2L2T Asrock MB X570D4U-2L2T AMD Ryzen MAX128GB DDR4 PCIE uAtx. This is the mATX version. You're limited to 128GB of RAM but you get more PCIe slots and you don't get the 10GbE network.

This one has 1*PCIe x16 + 1*PCIe x1 OR 2*PCIe x8 + 1*PCIe x1
 

hmw

Active Member
Apr 29, 2019
236
77
28
I have the same Norco RPC4308 Chassis
Do you have a picture of the 3 front 80mm FAN mount?
Could you describe how you mounted the 3 fans?

I like to mod the chassis like yours.

Thanks
IMG_1318.jpeg

@Marsh - here's how it looks with now with the PE foam mockup. The height is the same as the front grill (with the top HDD rack removed) minus 1mm. There's cutouts for cables and the whole thing is mounted as near the motherboard as possible. Right now, I'm looking to see how to 3D print this thing - if it is too expensive, I might just go in for another case as I've decided to shift the storage to 2.5" SSDs as soon as feasible - so moving slowly to 8GB SATA SSDs when they become available

But with this mod, I can keep the HBA, the Mellanox card and the RAM - cooler by 8c-10c
 
  • Like
Reactions: Marsh

hmw

Active Member
Apr 29, 2019
236
77
28
I love what you did with the fans.

Do you think 92mm fan would fit?
A 92mm fan will fit - the case is 7"/178mm and the lower part is ~ 136mm. However my foam mockup wouldn't work with 92mm, since I didn't have the correct rigid, high density foam. I was thinking of cutting some wood to size and mocking it up that way - so it would look like the Norco fan wall cage.

Another addition I was planning was using a 120mm fan shroud on the front of the entire server like this guy. Especially since I switched to Server 2019 and my NAS HDDs don't spin down anymore (I use Drivepool and I've enabled all the settings on the 2308 HBA driver). But I would use 120mm x 15mm fans instead of the usual 25mm
 
  • Like
Reactions: TXAG26 and Marsh

TLN

Active Member
Feb 26, 2016
403
55
28
30
Provantage has the X570D4U-2L2T for $389 PROVANTAGE: ASRock Inc X570D4U-2L2T Asrock MB X570D4U-2L2T AMD Ryzen MAX128GB DDR4 PCIE uAtx. This is the mATX version. You're limited to 128GB of RAM but you get more PCIe slots and you don't get the 10GbE network.
Actually you do, hence -2T in the name.
I'm debating between those two boards for workstation. mATX switches to x8/x8 if 3rd slot is occupied, so it doesn't do much for me. If that was x16/x8 it would be no brainer.
 
  • Like
Reactions: hmw

hmw

Active Member
Apr 29, 2019
236
77
28
Actually you do, hence -2T in the name.
I'm debating between those two boards for workstation. mATX switches to x8/x8 if 3rd slot is occupied, so it doesn't do much for me. If that was x16/x8 it would be no brainer.
ah good catch. For $389 this board is almost perfect - paired with a 3950x. If this had been out I wouldn’t have built the EPYC based server ...