AMD Epyc dual 7281 home build

[Flybird]

Build’s Name: to be determinated
Operating System/ Storage Platform: to be determinated (win server 12 essentials or win 10 pro)
CPU: 2x AMD Epyc 7281 (Box)
Motherboard: Supermicro H11DSi-O
Chassis: Phanteks Enthoo Pro Tempered Glass
Drives: SSD Evo 840 256gb + 3x4TB WD red (RAID5) + Wd Red 8TB (external distant location backup)
RAM: 2x16GB DDR4 2133 ECC-Reg (Kingston KVR21R15D4/16)
Add-in GPU Cards: No GPU yet but intending to get a Tesla or Titan for the near future (deep learning, Fp32 vs FP64? => another debate)
Add-in network Cards: 2x Mellanox ConnectX10Gbit (1 card for the server and 1 card for my workstation. Direct link. Really cheap solution!)
Add-in case fans: starting with the stock Panteks fans. May add some noctuas. Will depend on the temps
Power Supply: Seasonic 850W Prime Gold
Other Bits: 2x Noctua NH-U14S TR4-SP3 CPU coolers

I already have some components. The others are already ordered and coming from various sellers. I will probably post some pictures when I will build the server (end of march because had issues with the order of the supermicro motherboard).

Usage Profile:

I'm an french entreprenor, working for myself. I'm in the field of trading/fund management. Already have an upgraded home build workstation. As I needed more core power for my computations (parallel computing for increasing backtests speed), and also a computer to do other various stuff on an 24/24 basis (real time streaming, data storage), I decided to go with AMD Epyc, espacially regarding the fantastic performance/price ratio.

Here are the main usages:

1) Financial computations: backtesting using multiple cores (in fact, this server will be part of a cluster of up to arround 86 cores (90% max cores usage =>more efficient) close to my main workstation (overclocked i9 7960x))
2) Real time operations: streaming financial data (tick data) + making live financial transactions
3) NAS/data storage/local backup for several OS (server/workstation/laptop)

Issues
1) Memory: already have it. I know that this will not be enough. Will for sure upgrade to 64gb minimum. It is difficult to know right now how much ram I will really need. Wait for it. Did not find the compatibility on the supermicro website but according to their support it should work. What do you think? What are your advice guys regarding the possible upgrade? Any model suggestion?
2) OS: I previsouly had another home made server (E5 2686 v4) running on win R2 2012 essentials. I'm not a win server specialist. Had some issues with it. Regarding my usage win 10 may be enough (compatibility issues, easy to manage...) but for sure, may not be the optimal solution (espacially for stability...). VM may be a good choice on win server 12. What do you think guys, should I go for win server again or win 10 should do the job regarding my usages? I'm OK to forgot lets say, 10% performance, for using win 10 PRO instead of win 12 server, but no less.
3) Fans: I saw on other posts that there are some issues with Noctua CPU fans on supermicro motherboards. But it seems that it can be fixed. https://forums.servethehome.com/ind...ild-for-fluid-simulation-now-with-pics.17703/. Any additionnal feedback on it is welcome!
4) SSD: not really a good solution but already own it. Will change it in the near future. On a short term horizon, it may not impact performances
5) RAID: I did not already ordered the disks. I'm thinking of the RAID 5 with 3 disk + 1 distant 8TB disk external backup (optic fiber connection on both site). I'm really new to RAID. I think that 8TB will be enough for the start. For the external backup I may use incremental backuping but not really know something on it.. What do you think about it? What be the optimal configuration?

Thanks to all!

NB: I discovered this STH some weeks ago. Very useful and friendly place. Nice to meet you guys and thanks for the job!

 

[alex_stief]

1) Memory: already have it. I know that this will not be enough. Will for sure upgrade to 64gb minimum. It is difficult to know right now how much ram I will really need. Wait for it. Did not find the compatibility on the supermicro website but according to their support it should work. What do you think? What are your advice guys regarding the possible upgrade? Any model suggestion?

You got their support to admit that memory not listed on their tested memory list might actually work? That is a first for me
Apart from the total amount of memory you might need, the minimum requirement for two AMD Epyc processors are 8 DIMMs. The reason is the unusual CPU architecture with 4 NUMA nodes per CPU. A node without memory directly attached to it will suffer from high memory latency and low bandwidth. So upgrading to 8x8GB asap is a must. In terms of what to buy: keep it homogeneous by adding similar DIMMs to the ones you already have, especially concerning the number of ranks per DIMM.
BTW: according to the model number you posted (Kingston KVR21R15D4/16) you have 16GB DIMMs.

2) OS: I previsouly had another home made server (E5 2686 v4) running on win R2 2012 essentials. I'm not a win server specialist. Had some issues with it. Regarding my usage win 10 may be enough (compatibility issues, easy to manage...) but for sure, may not be the optimal solution (espacially for stability...). VM may be a good choice on win server 12. What do you think guys, should I go for win server again or win 10 should do the job regarding my usages? I'm OK to forgot lets say, 10% performance, for using win 10 PRO instead of win 12 server, but no less.

I personally had pretty bad results with windows 10 and windows server. Another guy I know had serious trouble getting a dual-socket Epyc to run properly on Windows 10. My guess is that it has to do with windows 10 officially supporting only 2 CPUs. While these are two physical CPUs, they form 8 NUMA nodes which might be too much. No way to confirm this though, just my wild guess. There sure must be a way to use the full potential of these CPUs under windows, but did not dig too deep since I am using linux.

3) Fans: I saw on other posts that there are some issues with Noctua CPU fans on supermicro motherboards. But it seems that it can be fixed. https://forums.servethehome.com/ind...ild-for-fluid-simulation-now-with-pics.17703/. Any additionnal feedback on it is welcome!

No issues so far, the workaround works like a charm.
I switched to a Phanteks Enthoo Pro recently. The fans that come with it are pretty shitty, but the case is ok. The Noctua coolers fit perfectly thanks to their variable mounting position.

 

[Flybird]

Thanks for your reply Alex.

Regarding the memory, if adding 6x16 Ram 2133 may 1. Cost far to high 2. Probably be more than necessary for my usage and 3. May not be optimal in terms of frequency (2133 only). I think I'm gonna sell it and search for affordable 8x8 2666 memory.

You got their support to admit that memory not listed on their tested memory list might actually work? That is a first for me

Here is the message of Supermicro:

"We don’t test or validate Kingston memory, so we cannot give you any guarantee. Technically however the memory should be fine.
Just to note, the Epyc CPU’s can give you quite a bit more performance when using higher speed memory. This as the 4 dies on the CPU are connected by a bus running at the memory clock speed."

I personally had pretty bad results with windows 10 and windows server. Another guy I know had serious trouble getting a dual-socket Epyc to run properly on Windows 10. My guess is that it has to do with windows 10 officially supporting only 2 CPUs. While these are two physical CPUs, they form 8 NUMA nodes which might be too much. No way to confirm this though, just my wild guess. There sure must be a way to use the full potential of these CPUs under windows, but did not dig too deep since I am using linux.

I think I'm gonna test win 10 then win server next. On linux of course I know that I will probably get the best performance but unfortunately, for compatibility issues I may me forced to stay windows...

Thanks again++

 

[Flybird]

Hello guys!

Finally finished up with my build:




Have some issues, some as excepted and some not:

Solved issues:

1. OS: I first try with win10 but it rapidly shows unstability (mouse and keyboard hangs out a few minutes after light usage, forced to reset...). So I decided to install win server 12 and now it is perfectly stable
2. Fans: as others I got issues because I have noctua fans and they are running very low at idle. I fixed the issue first with the noise reductors (including with noctua fans) but finally came up using ipmitool to set lower thresholds. This is globally working like a charm now

Work in progress issues:

3. CPU speed:

- As others, I got pretty disapointing results at first sight regarding various benchmarks and computations. CPU's were not able to run at the boost speed of 2.7ghz (whether single or multicore processing). I finally found the solution thanks to Internet by setting the option "high performance" on the control panel (default was "balanced"). I also set the minimum CPU on 5% (like in the balanced mode) instead of 100% (default for high performance). This lead to not run the cpu's at full speed anytime...
- I still have an issue: using parallel computing (basic matrix multiplication via Matlab) at 87.5% max load (56 cores/"workers" on 64, close to optimum in terms of productivity because going higher will increase temp and decrease efficiency), I noticed a kind of decreasing marginal efficiency. That is to say, at start it runs at 2.7ghz (all workers), and then decrease progressively until 2.4ghz (all workers close to). I try to set fans at full speed (leading to a stunning decrease in temps from 51°C to 34°C) in order to analyze if there is any correlation with this decreasing power effect but for the moment I see no changes... Moreover, I noticed that CPu's voltages decreased progressively (non linear fashion). There is possibility a link (consequence or cause?). Really don't know if the culprit is the motherboard or the Epyc.. Last but not least, I noticed that win server shows 100% CPU usage on anycore whereas I only use 87.5% of them (I have almost no others apps running so...). Any ideas guys on this issue? I never have such an issue with Xeon V4 or even i7 7700k (stock)

Edit: maybe a link with this?
- AMD's Future in Servers: New 7000-Series CPUs Launched and EPYC Analysis
- https://images.anandtech.com/doci/1...r_press_and_analysts__06_19_2017-page-009.jpg

4. Benchmark results:

- Cinebench R15: At the beginning I expected to get at least 3500... I got very unstable results instead. Such a wide range of scores: from 1500 to 2900! Please note that when running, it is 100% usage (as expected) at 2.7ghz. I suspect the memory playing a role in this mess. I only use 2 DIMMS and moreover they are only running at 2133. I guess this is the culprit. What do you think? Instability plus not able to reach the minimum score it should..
- Geekbench: same analysis. Got 2657 on single (pretty good results compare to the others) and only 10791 on multicore. Please note that when running (when at max), it is 100% usage (as expected) at 2.7ghz. Please have a sight: 7281 - Geekbench Search - Geekbench Browser. Still the memory? The OS maybe playing a role also? (I heard that benchs used to be better on linux)
- Matlab: I got what I expected before setting up the build => 3.5x computing power increase compare to my i7 (again 87.5% max cores mobilisation). This is the only good news lol..For such computations, I know it did not use a lot of memory. This could explain why Matlab benchmark is ok whereas the 2 previous are not. Furtherwore, I have to say that I get 970 on cinebench r15 with my i7 7700k (stock). So it should translate to at least 3500 with the Epyc, don't you think?

One last thing, according to CPUZ, max TDP 155 (don't know if CPUZ is reliable). We know that max TDP is 170w on theses Epyc 7281. Any link?

My best,

 

[alex_stief]

Epyc 7281 has a default TDP of 155W and can be increased to 170W https://www.amd.com/system/files/2017-06/AMD-EPYC-Data-Sheet.pdf
I believe that with "auto" settings the TDP is configured based on the memory frequency. Don't remember where I read that, but from what I recall: DDR4-2666 -> 170W; everything below -> 155W
My H11DSi allows me to override the TDP value, but I haven't noticed any changes linked to this value. But since you are having throttling issues you might want to give it a try setting TDP to 170W manually.

Concerning Cinebench R15, my dual 7301 got to just over 4000 points using Windows server 2012. Still way below its theoretical potential when we look at how Ryzen CPUs score here. This is one of the things that led me to believe windows does not like these CPUs. I am using Linux anyway where I get the expected results for workloads relevant to me, so I did not look further into this.
But in your case, memory configuration definitely plays a role in this. NUMA-modes without their own memory will introduce a huge performance bottleneck, even for workloads that are not typically considered to be memory bandwidth limited.

 

[Flybird]

Epyc 7281 has a default TDP of 155W and can be increased to 170W https://www.amd.com/system/files/2017-06/AMD-EPYC-Data-Sheet.pdf
I believe that with "auto" settings the TDP is configured based on the memory frequency. Don't remember where I read that, but from what I recall: DDR4-2666 -> 170W; everything below -> 155W
My H11DSi allows me to override the TDP value, but I haven't noticed any changes linked to this value. But since you are having throttling issues you might want to give it a try setting TDP to 170W manually.

Concerning Cinebench R15, my dual 7301 got to just over 4000 points using Windows server 2012. Still way below its theoretical potential when we look at how Ryzen CPUs score here. This is one of the things that led me to believe windows does not like these CPUs. I am using Linux anyway where I get the expected results for workloads relevant to me, so I did not look further into this.
But in your case, memory configuration definitely plays a role in this. NUMA-modes without their own memory will introduce a huge performance bottleneck, even for workloads that are not typically considered to be memory bandwidth limited.

Think I'm gonna get sooner as expected 8x8 memory. We will see next. Thanks Alex.

 

[MiniKnight]

If you're doing deep learning you'll want Ubuntu. NVIDIA uses it heavily in their documentation. It'll also do ZFS out of the box for your storage needs.

 

[Flybird]

If you're doing deep learning you'll want Ubuntu. NVIDIA uses it heavily in their documentation. It'll also do ZFS out of the box for your storage needs.

Well for the moment I'm not making any GPU computing. I'm only concerned by parallel computing. This is done first by various R&D on Matlab and then going on production using c# or c++. My guess is that at this level the gap should be short between win server and Linux. If anybody prove me the contrary (regarding my usage) I'm very impatient to learn from her/him!

Thanks!

 

[TLN]

- Cinebench R15: At the beginning I expected to get at least 3500... I got very unstable results instead. Such a wide range of scores: from 1500 to 2900! Please note that when running, it is 100% usage (as expected) at 2.7ghz. I suspect the memory playing a role in this mess. I only use 2 DIMMS and moreover they are only running at 2133. I guess this is the culprit. What do you think? Instability plus not able to reach the minimum score it should..

I'm getting aroung 4k poings with dual E5-2683v3 CPUs with BIOS hack. It's been rock stable for me, even though it's QS CPU.

 

[beaker7]

Cinebench starts to get unreliable above about 4000 points. It runs too quickly to give repeatable results in a lot of cases due to how quickly or not the cpu's turbo up, scaling, etc. My dual Xeon 6154 will give anywhere from 5400 to 6450 in that bench.