AMD Epyc build for fluid simulation (now with pics)

alex_stief

Active Member
May 31, 2016
749
238
43
36
Build’s Name: AMD is back (Noctua fanboy edition) :cool:
Operating System/ Storage Platform: OpenSuse (Tumbleweed)
CPU: 2x AMD Epyc 7301
Motherboard: Supermicro H11DSi
Chassis: Nanoxia Deep Silence 2
Drives: Crucual MX100 512GB, 4x Intel S3500 800GB (RAID0), HDDs undecided, probably 4x WD red RAID5
RAM: 16x16GB Samsung 2Rx4 DDR4-2133 reg ECC (still hoping to hit 2400 MT/s)
Add-in Cards: Nvidia GTX 960 4GB
Power Supply: Seasonic Focus Platinum 850W
Other Bits: 2x Noctua NH-U14S TR4-SP3 CPU coolers, lots of fans. I am a big fan of air cooling.

Usage Profile: Mostly fluid simulation stuff and code development. All my solvers are memory bandwidth limited, hence AMD Epyc. The RAID0 will serve as a scratch drive to save and work with large transient result files when creating animations.

I know that faster memory would have been better. But most of the workstation consists of parts I already had or could source very cheap. Memory might be upgraded once the prices normalize.
Getting the parts (specifically CPUs and mainboard) was quite an ordeal. Would have started this build much earlier if they had been available. But now they are all here and I am trying to get it to run. Looking good so far.

Edit1: first issues
1) I was aware of this from previous builds, but still worth noting: Supermicro refuses to use standard ATX mounting holes even for their workstation boards they label as "E-ATX". So a few holes needed to be added to the motherboard tray of the case.
2) Cables of the power supply (24pin ATX and 8pin EPS) are too short for proper cable management behind the board. Already ordered some extension cables.
3) Again Supermicro: the motherboard has issues with these slow-spinning Noctua fans. They idle at around 300 rpm which causes the motherboard to flash a red LED and rev up all fans to maximum speed in an interval of ~10s. When I disconnect the fans and run the coolers passively, everything works fine and temperatures are ok. Will contact SM for a bios update request that adds "ignore fan failure" options or allows for slow-spinning fans.
4) Poor performance under windows 10 pro. I like to use a windows test installation while setting up new machines to have a different OS for cross-checks. Cinebench yields ~4000 points, I would have expected somewhere above 5000 points for these CPUs. Will check if windows server 2016 runs better.
5) Does not post with memory speed set to DDR4-2400. Will have to check again when new bios versions are released.

Edit2: lessons learned
So fan control issues could be resolved, however the lack of fan control through bios options still baffles me. If I had any other choice Supermicro would not have made the cut. Their support was not particularly helpful either. The motherboard has a few design flaws. The m.2 slot only gets 2 PCIe lanes, no USB3 header, only two PCIe x16 slots...
I will consider changing the motherboard once ASRock rack releases their dual SP3 lineup. They mentioned availability in Q2 2018.
The case is cheap, and it shows. OK, you can't expect much for 80€. but the material is paper-thin, parts barely fit and hold together due to loose tolerances and even mounting the side panels is a challenge unless you have 4 hands or lay the case flat on its side. I already had a similar case from this brand with similar issues. However, there are not many alternatives on the market that have a similar small footprint and the same expansion options. I feel like there is a compact case missing on the market with no window side panel, E-ATX compatibility, no wasted space for optical drives and room for ~8-10 3.5"/2.5" drives. For a less compact case I would recommend Fractal Design Define XL. It is of much higher quality.
Apart from these issues I am pretty pleased with the comeback that AMD has made with these Epyc processors, they deliver exactly what I need at a very competitive price. Too bad availability is still an issue.
Until the system is put to its intended use in Q1 2018, time to mine some Aeon to offset the price.
 
Last edited:

alex_stief

Active Member
May 31, 2016
749
238
43
36
I got them from Alternate, one of the largest hardware retailers here in Germany. There was a very brief period of time when these CPUs were listed as available. I guess I just got lucky. Now they are out of stock again. You would have to pay the premium for a 7351 if you need it right now, these are available from multiple sources.

Need pictures!
There will be pictures. But right now it is too messy to show anything, I am still in the process of setting it up both in terms of hardware and software.
Can you comment on the poor performance running windows 10?
 
Last edited:

Aestr

Well-Known Member
Oct 22, 2014
932
329
63
Los Angeles
3) Again Supermicro: the motherboard has issues with these slow-spinning Noctua fans. They idle at around 300 rpm which causes the motherboard to flash a red LED and rev up all fans to maximum speed in an interval of ~10s. When I disconnect the fans and run the coolers passively, everything works fine and temperatures are ok. Will contact SM for a bios update request that adds "ignore fan failure" options or allows for slow-spinning fans.
This is a common problem, but more or less working as designed. I don't have a thread handy, but there are a bunch on here about changing the thresholds for the fan monitors using IPMIutil.
 

alex_stief

Active Member
May 31, 2016
749
238
43
36
Thanks Aestr. That is the same response I got from Supermicro support. Since I am quite a noob in some (most) areas: how can I access IPMI? I connect a laptop to the IPMI port via ethernet cable, open a browser... and what do I do next?
 

alex_stief

Active Member
May 31, 2016
749
238
43
36
I am fairly confident that I can try to figure something out once I get access to IPMI with some GUI tool.
I installed IPMI View from supermicro website, connected the second LAN port to the IPMI port and started IPMI View. Here I am lost. How do I find my own IPMI System? Or is IPMI View the wrong tool? Do I have to configure IPMI in the bios first? It is switched on and I changed none of the default settings.
 

MiniKnight

Well-Known Member
Mar 30, 2012
3,013
922
113
NYC
ipmitool is what you are going to want for this.

You want IPMI port attached to your LAN. It will get an IP address and that is how you log in and manage remotely.
 

alex_stief

Active Member
May 31, 2016
749
238
43
36
Yeah I tried that. Both hooking up a laptop to the IPMI port and one of the LAN ports of the motherboard.
I set "update IPMI LAN Configuration" to yes (configured as DHCP) and rebooted the machine. Unfortunately, IPMI module did not get a valid IP address this way.
 

alex_stief

Active Member
May 31, 2016
749
238
43
36

squidman

Member
Jul 8, 2017
92
4
8
50
Is this for VFX/feature film or scientific development? Curious as a Maya user (and looking at Houdini of course!). Home development lab or work use? Always wondered what peps with racks in their homes do with their vm's etc (lab use)...etc.
 

alex_stief

Active Member
May 31, 2016
749
238
43
36
Scientific/engineering simulation, not the CGI kind.
For one thing I was just curious if it would perform as I expected from the specs, which it does. And I like to tinker with hardware.
Of course this was not the main reason to spend several thousand on hardware. It will be used for some actual work. Can't give any details yet, sorry.
 

squidman

Member
Jul 8, 2017
92
4
8
50
Question because of this guy, who just obviously likes servers! Taking it to the extreme! (ok think he renders a bit in Max)..
 

squidman

Member
Jul 8, 2017
92
4
8
50
Cannot have too much horsepower in simulating fluids tho! My early attempts in RealFlow show that! so yeah...but was more VFX kind of thing...

Ditto rendering..still a huge rack in the home like this guy..hmm...maybe not! Maybe amazon cloud or similar?

Edit he prbly is a hoarder of old servers...but very cool that he got it all working! Very cool youtube channel, iff not very practical vis-a-vis power costs and actual sim/rendering effectiveness. But a monster server basement!
 
Last edited:

alex_stief

Active Member
May 31, 2016
749
238
43
36
I like to distinguish between the machine types. For me this is a workstation, not a server. Servers are boring, workstations are not ;)
Anyway, here are some pictures. It is still missing hard drives, but apart from that the build is final. Cable management is crappy as usual, but I don't care as long as side panel can be closed.





 
Last edited:

alex_stief

Active Member
May 31, 2016
749
238
43
36
Updated to bios version 1.0b which includes AGESA version 1.0.0.3.
Still no post with DDR4-2400 :(
 

TomUK

Member
Aug 30, 2017
67
12
8
35
Updated to bios version 1.0b which includes AGESA version 1.0.0.3.
Still no post with DDR4-2400 :(
Do you know how much the RAM speed will impact your particular workload? If its a major hit you could sell the memory and get ECC 2667? Expensive though!

Sent from my STV100-4 using Tapatalk
 

alex_stief

Active Member
May 31, 2016
749
238
43
36
My workloads scale very well with memory performance. But I won't spend 3000€ on RAM right now knowing that it was less than half of that ~18 months ago and will be again in a not so distant future.
And i am still not convinced that single-rank DDR4-2666 is that much faster.