Epyc 7302P & H11SSL-NC Workstation Build: Follow-Up Report

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

balnazzar

Active Member
Mar 6, 2019
221
30
28
Thank you. I'm on the verge of ordering an H12SSL-i. I asked that to supermicro by e-mail, but they didn't answer...
 

TXAG26

Active Member
Aug 2, 2016
397
120
43
In the latest version of Windows 10, I don't see the option to "hibernate" or "sleep". When I click on the start menu power button, it just shows "Restart" or "Shutdown". I do remember doing some registry tweaks and some other things when I first built this system as it was not turbo boosting up as high as it should. I also recall maybe disabling some of the S3/S4/S6 states in the bios.

Wish I could be more help. What is your reason for needing to sleep the machine vs just shutting it down? Once you disable the various add-on bioses, these systems can generally be set up to boot really fast with modern hardware. At the moment, I have the LSI 3008 bios disabled and not having to initialize that at boot up also helped quicken boot time.
 
  • Like
Reactions: balnazzar

TXAG26

Active Member
Aug 2, 2016
397
120
43
Performance & Stability Update:
I wanted to post an update on the performance and stability of this build. In two words, it has been rock-solid! Since COVID-19 reared its ugly head in early March of 2020, I've been running Folding@Home on this machine on both the AMD 7302P and the Nvidia GTX 1660 Super. This machine with CPU and GPU folding does a little over 1.1 million F@H points per day.

It's been running F@H full tilt 24/7/365 since early March and has remained snappy when functioning as my daily workstation. Since March, the CPU has run constantly at 100% @ 3.27 GHz and stays pegged at a boring 61 degrees C using the 4U Supermicro active heatsink. The 4x 80mm fans in the fan wall operating in the "Standard" fan mode spin at a quiet 2,100 rpm and keep everything cool and quiet. These fans go down to around 1,100 rpm on "Optimal" setting, but I wanted a little more air flow for the GTX 1660 Super since I am also folding on it. Even so, the GTX 1660 Super stays at 66 degrees C folding 24/7/365 with the current airflow setup.

I have the dual port Supermicro X550-T2 10GBe network card installed in the lowest PCI slot and did add a quiet 40mm fan to its heatsink just to make sure it stays cool as the bottom back corner of most server cases do not get great air flow without internal ducts. The AOC NIC is reporting a 49 degree C temperature in IPMI. It was in the upper 70's before adding the small 40mm fan.

Bottom line, this build just works and has been super solid for what I do. An exciting milestone is also right around the corner for this build - it will complete 200 million points at F@H by early next week!
 
  • Like
Reactions: balnazzar and hmw

balnazzar

Active Member
Mar 6, 2019
221
30
28
What is your reason for needing to sleep the machine vs just shutting it down?
Essentially, it's about having your work ready at the point you left it.
It's a deep learning workstation. A lot of terminals open upon the right paths. A lot of conda envs which would require to be re-initialized after a reboot. Jupyter notebooks to be re-instantiated, etc etc.
Leaving the machine always on is a non-option for me. Too much power draw & heat, given it has to host 3-4 GPUs.
 

TXAG26

Active Member
Aug 2, 2016
397
120
43
Honestly, I’m not sure how well the programs you described will handle being put to sleep. If there’s a way to unload the GPU’s when you’re finished for the day, that may be viable as most GPU’s use very little power when not being used. Is there a way to test your current workload on a machine that does sleep to see how it handles waking back up?
 

balnazzar

Active Member
Mar 6, 2019
221
30
28
Honestly, I’m not sure how well the programs you described will handle being put to sleep. If there’s a way to unload the GPU’s when you’re finished for the day, that may be viable as most GPU’s use very little power when not being used. Is there a way to test your current workload on a machine that does sleep to see how it handles waking back up?
Yes, I use (and had used) such practice daily upon diverse mainboards (mainly C612 with e5v3/v4, but a ryzen 3700X too).The status of the machine is resumed flawlessly, as it was never been put at sleep. You can even put them asleep during intense computations.. Both the cpu and the gpus resume such computations without any hassle.

It is important foe me, since electricity in Europe costs more than it costs in the US. Consider that nvidia-smi says that each 2060 Super draws 16W at idle (which is over 60W for 4 of them). The Epyc machine I built with the EPYCD8 and the 7282 idled at ~160W with four GPUs installed, which is remarkable, but still to high for me to afford leaving it on 24/7.

Another user with the ROMED8 confirms that it doesn't sleep in Windows 10. Bios settings? Maybe, but I cannot afford the risk.
Too many people are confirming that epyc boards don't sleep, and the xeon w-22XX have a very bad price/performance/power draw ratio. I think I'll go for a desktop processor (maybe the 3950x) with 128gb of non-ecc ram, hoping it won't cause filesystem corruption in the long run... No, I won't buy ecc udimms, they cost too much and are still impossible to find in 32gb size.
 

james23

Active Member
Nov 18, 2014
441
122
43
52
Thanks again for all the info you have provided (and super detail!) , i really like this build! (and am looking to upgrade my i7-4930x setup).

2x quick questions please,

1- with the SM mb you have, does the MSI GeForce GTX 1660 card block one of the pcie slots? (i will be running 2x gpus, so this is a concern) - my assumption is that with 2x of your GPUs and the same MB, i would only be left with 2x free/open PCIe slots.

(ie in my rough image below , with 2x of your GPUs, i would only be left with the 2x slots i have big red check marks on).
1600020222427.png

2- am i correct in that even on this supermicro board, the ONLY fan control you have is the normal 3x or 4x supermicro ipmi fan profiles- (ie- Standard, optimal, heavy IO, full-on), ie you have no ability to lower the fans rpms "slower' than the optimal sm fan profile, right?

(easily solved with an external fan controller though)

thanks
 

TXAG26

Active Member
Aug 2, 2016
397
120
43
Thanks again for all the info you have provided (and super detail!) , i really like this build! (and am looking to upgrade my i7-4930x setup).

2x quick questions please,

1- with the SM mb you have, does the MSI GeForce GTX 1660 card block one of the pcie slots? (i will be running 2x gpus, so this is a concern) - my assumption is that with 2x of your GPUs and the same MB, i would only be left with 2x free/open PCIe slots.

(ie in my rough image below , with 2x of your GPUs, i would only be left with the 2x slots i have big red check marks on).
View attachment 15724

2- am i correct in that even on this supermicro board, the ONLY fan control you have is the normal 3x or 4x supermicro ipmi fan profiles- (ie- Standard, optimal, heavy IO, full-on), ie you have no ability to lower the fans rpms "slower' than the optimal sm fan profile, right?

(easily solved with an external fan controller though)

thanks
I think my GTX1660 Super is a 2 slot card, but with the side fans, a card next to it could cause airflow issues. I would look very closely at the heatsink on the GPU you select if you need to be able to put 2 of them side by side as you indicated. I only have the fan default profiles setup by SM. No way to change the speeds unless you use external controller.
 

james23

Active Member
Nov 18, 2014
441
122
43
52
thanks, that is good to keep in mind (and i had forgot about slamming 2x 2-slot cards right next to each other). I actually will test some low end , single slot options first (ie quadro p1000) , as i dont do any gaming, nor any creative stuff that requires GPU (i just run 8x 2k/4k monitors for network engineering/admin work, so will need 2+ physical gpus).

Forgive me if this is a dumb question- but as im totally new to AMD builds, why can i not find many of the gen2 amd cpus easily (or in quantity) either on ebay or at reasonable prices/stock from retailers (new or used). I can find them, but its def. not what im used to with intel parts (im referring to ebay for the most part). Even when i look at ebay prior sales, there arent a ton sold.

is it a high demand issue? or because the new gen3 cpus are about to come out? ( like you, i dont want pcie 4.0). Or have they never been manufactured in high supply/numbers?
tks
 

TXAG26

Active Member
Aug 2, 2016
397
120
43
They’re still relatively new and I doubt many are on eBay since most with a Rome system likely recently acquired it. I think production has kept up with CPU demand and likely not exceeded it.
 

james23

Active Member
Nov 18, 2014
441
122
43
52
TXAG- how did you deal with the CPU install in regards to the install guide/specs calling for a Torque adjustable Torx T20 driver @ 16.1 kgf/cm? (did you already have one, or did you just feel it out with a standard non-Torque adjustable T20 screwdriver?)

I dont have one, and would rather not spend the 50-80$ on one for just this one use. (but im not clear on how necessary it is to use one, considering ive, carefully, installed 100s of xeon cpus over the years wo bending any pins)

i ask as i just ordered-
a H11SSL-NC Rev. v2.0 and a epyc 7262. (FWIW, from provantage for $440 (mb) / $605 (cpu)).
I was also able to get 4x 32gb 3200mhz ECC DDR4 sticks for $120 ea (best offer accepted to a ebay seller who had qty 4x of them - the ram is on the SM QVL for our particular board).
thanks again!
 

TXAG26

Active Member
Aug 2, 2016
397
120
43
I had a torque wrench that I could adjust to that spec. I have not installed one without it. I think you can get one of the pre-set wrenches that comes with the Ryzen CPU’s on eBay for a reasonable price.
 

hmw

Active Member
Apr 29, 2019
570
226
43
I dont have one, and would rather not spend the 50-80$ on one for just this one use. (but im not clear on how necessary it is to use one, considering ive, carefully, installed 100s of xeon cpus over the years wo bending any pins)
These are server class CPUs and like the modern Xeons, they need to be tightened with the *exact* amount of torque. I would advise you to buy an AMD tool like this one - Torx Screwdriver for AMD Ryzen Threadripper Processor, Star Tool Only | eBay - and not risk a $700 cpu over a $50 tool. If you get the EPYC CPU as retail, it will have the AMD tool. Else just grab the tool from eBay
 

james23

Active Member
Nov 18, 2014
441
122
43
52
i agree, i bought one of the 10$ threadripper orange AMD tools from ebay. In terms of your xeon comment though, i have socket'd 100s of xeons over the years with a standard screwdriver (or most often with one of those cheap 1-10 "torque adjustable" small power screw drivers, on the lowest settings). i always use the criss-cross method for the screws.
Xeon is different in that the only screws are for the heat sink, not for the CPU -> Socket (like with amd cpus)
.
(im sure you know all of this above already, but just wanted to be clear- i do agree on AMD this is a much more important torque "need" than on xeon installs). thanks
 

james23

Active Member
Nov 18, 2014
441
122
43
52
Regarding the installation of Windows 10, I found it necessary to do the following:
1) Download the latest Windows 10 Media Creation Tool from Microsoft and create a bootable USB thumbdrive with the latest version of Windows 10 Pro.
2) Before installing Windows 10, disable IOMMU and Hyper-threading in the bios. See FAQ Entry | Online Support | Support - Super Micro Computer, Inc. for some discussion about IOMMU. I read in another article that disabling Hyper-Threading is also helpful, so I did that too.
3) Make sure that one of the Intel i210 networking ports on the H11SSL has internet access as the Windows 10 installer will grab some needed drivers during the installation.
Thanks again for this thread! it helped me pick parts for my new daily PC (as prior to this specific thread i was going to go intel 2nd gen silver + SM mb ect..) Its replacing my current daily- i7-4930x w 64gb ddr3 (nonECC)

I now have all the parts (will list below), and will be OS testing / stressing it for the next week or so. I can add this now though-
While i had your 3x points RE: changes needed to install win10 , pasted into my notes:
i can confirm that i didnt have to do *any* of those steps to get a working win10 install running on almost the exact same HW as you.
Im assuming that msft fixed the issues you point to, in the latest win10 ISO. (i used the msft media creation tool, to create/save a ISO of Win10-64b-pro, today- Sept 26 2020 , and then used Rufus to burn it to usb).

I havent changed anything in the stock SM bios (latest, v2.1 that came on board already), except i changed these minor BIOS items:
+ disabled all Serial (com) ports
+Set SATA to Enabled (from auto) (i have not used any sata ports yet though)
+set boot mode to UEFI (from BOTH - + disabled all network boot functions, such as PXE , http ect)
+changed the boot order to first try USB sticks (vs HDDs first).

Then used ipmi/vKVM to install win10 from usb stick to a p4800 u.2 nvme drive (connected via one of the 2x NVMe ports on the MB). It went really fast, auto-rebooted 2x times, then i was at the normal win10 "choose your region" setup prompts (and eventually win10 desktop)..

HW i ended up getting (with prices i paid);

CPU
: amd 7262- $605 (provantage, cpu wo fan - said 3x in-stock but took 8 days to ship out)
Motherboard: - H11SSL-NC Rev. v2.0 - $436 (provantage - shipped out same day as order)
RAM: 4x 32gb ECC- HYNIX HMAA4GR7AJR8N-XN - $120 ea = $524 w tax for 128gb (ebay) (exact sticks are on SM mb QVL)
Disk: optane u.2 p4800x 375gb ($387 w tax - ebay) - for boot/apps, my storage is all network attached here
GPU: 2x Quadro p1000 ($246 each = $491 w tax - ebay) - i dont game nor need GPU power, i just need to run 8x monitors, many of which are 4k

Networking: 10gbit SFP+ pcie card (i have many already, so will be one of those cards)

Chassis: existing consumer chassis i already have (MB is E-ATX fyi)
Power/psu: existing consumer 1400w psu i have

Audio- Crap! i dont have this covered! i had been looking at consumer supermicro MBs and didnt realize this was a server SM mb (so no audio), will figure something out, either USB to 3.5mm or a small x1 pcie audio card. I dont need quality, just some audio and it to be stable.
so far so good!
thanks again
 
  • Like
Reactions: TXAG26

TXAG26

Active Member
Aug 2, 2016
397
120
43
I’ve been happy with the Soundblaster USB audio dongle listed in my post. It just works with Win10. I just use it to listen to music/Youtube, nothing crazy, but it’s always worked with no fussing. I’m sure any of the $5-$40 USB audio adapter would work fine for what you’re using it for.
 
  • Like
Reactions: james23

james23

Active Member
Nov 18, 2014
441
122
43
52
I’ve been happy with the Soundblaster USB audio dongle listed in my post. It just works with Win10. I just use it to listen to music/Youtube, nothing crazy, but it’s always worked with no fussing. I’m sure any of the $5-$40 USB audio adapter would work fine for what you’re using it for.
Thanks, i will probably end up getting that or similar.

btw, did you install any drivers on windows 10? (im referring to motherboard or cpu related , not gpu or other add-ons). I see for our board SM does not have win10 drivers (its not an "officially" supported OS), so i may try their win server 2019 drivers just to test. (this is not my final win10 OS install, just a test install for the next week or so).

These are the only devices i show as missing drivers (and i have a feeling those may be the onboard LSI 3008 controller, which i wont be using).
1601172014582.png
i havent seen any problems, and the performance is excellent! (i did have an issue where it seemed when win10 would try to sleep the monitor (ie after 10m of no activity), the entire machine would crash + reboot , and log an ipmi event like these in yellow):
1601172099021.png

But i think a win10 update i just did may have fixed it ( + i checked off an optional win10 driver for AMD it was showing). since then i have slept the monitor 3x times with no isssue. (btw, i am still using the crap on MB vga graphics, not my GPUs yet, so im not super concerned about this anyway - the onboard graphics does not have a driver as is usually the case with server MBs and win consumer oses)

thanks
quick numbers from my test setup (of the 375gb p4800x , is the boot drive here though):

1601172366800.png 1601172399827.png 1601172748697.png
 

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
60 to 90 watts idle is really good, the 7002 systems seems to be a winner in that regard.