Epyc 7302P & H11SSL-NC Workstation Build: Follow-Up Report

TXAG26

Active Member
Aug 2, 2016
274
81
28
I just completed an AMD Epyc 7302P workstation build using a Supermicro H11SSL-NC board with the following components:

CPU: HPE AMD Epyc 7302P 3.0/3.3GHz Boost; 16-core; 128MB L3; HPE No. P16667-B21
Heatsink: Supermicro SP3 AMD Epyc Heatsink SNK-P0064AP4
RAM: Supermicro 4x 32GB 3200MHz DDR4 ECC; 128GB total; MEM-DR432L-SL02-ER32
Motherboard: Supermicro H11SSL-NC Rev. v2.0 w/LSI 3008 SAS3 + 2x NVME + 1x M.2 + IPMI
Networking: Supermicro AOC-STGS-i2T-O; 2-Port 10GbE w/ Intel X550-AT2
Chassis: Supermicro CSE-743AC-1200B-SQ 4U Tower Chassis; 8-port SAS3 Backplane; Whisper-Quiet Series
Power: Supermicro PWS-1K25P-PQ Power Supply w/PMBus;
Fans: 2x extra CSE-743 80mm Middle Case Fans (FAN-0104L4) 2,800 rpm; 32.9 CFM; 24 dBA
Graphics: MSI GeForce GTX 1660 Super Gaming X 6GB GDDR6 (G166SGX) w/Nvidia Studio Drivers
OS: Windows 10 Pro

This machine with all of the parts, above, and 1 SSD idles around 86w in Windows 10 and with Prime95 running 32 threads at max heat/stress setting, it stays at 3.1 GHz and does not go above 60 C with "Optimal" fan profile selected in IPMI.

Cinebench R20 score DDR4 @ 2933MHz : 7826 (32 threads, runs @ 3.27GHz and 56 C with "Optimal" fan profile.
Cinebench R20 score DDR4 @ 3200MHz : 7822 (32 threads, runs @ 3.27GHz and 56 C with "Optimal" fan profile.

I am running the DDR4 ram at 2,933 MHz so that it runs at the same speed as the AMD infinity fabric, which reduces system latency a bit and makes things more responsive.
Using 4 channels of DDR4, the difference in memory bandwidth between 3200MHz and 2933MHz is only 102.4 GB/s vs 93.8 GB/s, which isn't much. For me at least, IMHO, the lower latency seems to be more important and helps system responsiveness. I have not noticed a significant difference either way with various benchmarks.
https://forums.servethehome.com/index.php?threads/epyc-rome-ifop-limited-to-1467mhz.27113/

Regarding the installation of Windows 10, I found it necessary to do the following:
1) Download the latest Windows 10 Media Creation Tool from Microsoft and create a bootable USB thumbdrive with the latest version of Windows 10 Pro.
2) Before installing Windows 10, disable IOMMU and Hyper-threading in the bios. See FAQ Entry | Online Support | Support - Super Micro Computer, Inc. for some discussion about IOMMU. I read in another article that disabling Hyper-Threading is also helpful, so I did that too.
3) Make sure that one of the Intel i210 networking ports on the H11SSL has internet access as the Windows 10 installer will grab some needed drivers during the installation.

Why I decided on this particular build:
This current build replaces my previous workstation, built in 2013, which was an Intel Xeon E3-1245 v3 (4C/8T @ 3.4GHz) w/32GB DDR3 ECC. I do a lot of camera raw processing in Adobe Bridge and Photoshop for travel & outdoor photography and have huge piles of images to sort and process after each outing. With 30+MP images, the 4-core Xeon just wasn't cutting it. This new system chews through a stack of 100 raw images 4.5x faster than the prior system.

I researched various options including:

Intel Core Series -
Positives: Great CPU prices and decent performance.
Negatives: No reasonably priced larger capacity DDR4, hefty price premium for 32GB stick UDIMM's. Limited options for upgrades past 128GB. Only dual channel DDR4. No ECC support and I was not happy with the "RGB gamer style" motherboard options currently available. Limited expandability and limited PCIe lanes & slots.

Intel W-2200 Xeon -
Positives: ECC RDIMM and larger DDR4 capacities, workstation-class motherboard options.
Negatives: Price per core & price per unit of performance still much higher than AMD, lingering security concerns, still 14nm. Only quad-channel DDR4. Huge price premium to jump up to W-3200 series with 6-channel DDR4 and more PCIe lanes. Limited expandability and limited PCIe lanes/slots. A W2200 build was my backup option if the 7302P didn't materialize.

AMD Threadripper -
Positives: High clock rates and core counts, 7nm.
Negatives: Only quad-channel DDR4, still no ECC RDIMM support, more expensive UDIMM ram required, expensive $1,300 entry-level price for 24 cores was not in the budget, higher power consumption, no reasonable workstation-class motherboard options, available motherboards are expensive and looked gimmicky with all the fans, heatpipes, covers and RGB lighting.

AMD Eypc 7302P -
Positives: 16 cores/32 threads for $665 (via HPE deal), great value, 7nm, latest CPU architecture and security, infinity fabric, 1P NUMA, very snappy and responsive in Windows 10 Pro, power efficient for the amount of processing power (16 cores @ 155w/180w TDP), large amount of PCIe lanes, ECC RDIMM support, server/workstation class motherboards. Will transition well into a home file server role when this system is outgrown.
Negatives: 3.0GHz/3.3GHz clock speeds could be faster, especially since 4-5GHz is more common these days, but 16 cores @ 3.3GHz boost speed helps offset this issue. Currently, limited options for PCIe v4.0 motherboards. I ultimately decided to go ahead with a PCIe v3.0 board as I do not foresee needing the bandwidth of PCIe 4.0 devices within the next couple of years. PCIe 4.0 board also use more power and are more expensive to manufacture, so going with a PCIe 3.0 board actually has some benefits for some use cases.

All in all, this build went together very smoothly, no major hiccups. So far, I've been extremely pleased with this build and its a huge step up from what I was using previously.
 
Last edited:

balnazzar

Member
Mar 6, 2019
105
7
18
Hi, thanks for your detailed report. I have an Epyc 7282 (which is the 120W counterpart, 2.8 GHz base clock), with an Asrock Rack EPYCD8-2T and a Noctua cooler (https://noctua.at/en/nh-u9-tr4-sp3). EVGA 1000W T2 Titanium PSU. Silverstone MM01 case (I wanted the HEPA filtration system).

I was more interested in power consumption than clock, and made considerations similar to yours. I also needed more than 48 lanes, since mine is a deep learning workstation with four GPUs.
That also motivated the choice of the motherboard. I like Supermicro, but their SP3 boards just had three 16X slots and one M.2 port.
W.r.t. an Intel scalable I just regret AVX-512 and MKL performance.

Power consumption results are indeed wonderful. With a single RTX 2060 Super card (which accounts for 16W according to nvidia-smi), four fans (two Silverstone AP182 and Two Noctua), and two NVMe SSDs the system idles at ~60W with Ubuntu.
Same conditions, but loaded with 32 threads via the `stress` unix utility, it barely reaches 175W and never goes above 54/55C, while the (two) cpu fans remain to their lowest possible rotation rate, 600rpm. This is really something.

I didn't perform performance benchmarks.

Allow me to bother you with a couple questions.

1. I am just a machine learning engineer with little knowledge of microprocessors. more specifically I do not know what exactly are IOMMU, Ifop and similar. May you provide a link where such stuff is clearly explained? Googling provided low quality results.

2. I imagine supermicro has an IPMI module too with a dedicated NIC. My problem is that mine *does* work inside my LAN (I can access the ipmi interface from other pc over my LAN) but it doesn't work over the internet. Now I have other services running and listening over the main X550 NICs, and all of them are reachable from the internet, so it is not something related to the router or the port forwarding.. I think I did not set the management NIC correctly, but at the same time I didn't find any clue in the manual.

Thanks!
 
Last edited:

IamSpartacus

Well-Known Member
Mar 14, 2016
2,301
553
113
2. I imagine supermicro has an IPMI module too with a dedicated NIC. My problem is that mine *does* work inside my LAN (I can access the ipmi interface from other pc over my LAN) but it doesn't work over the internet. Now I have other services running and listening over the main X550 NICs, and all of them are reachable from the internet, so it is not something related to the router or the port forwarding.. I think I did not set the management NIC correctly, but at the same time I didn't find any clue in the manual.

Thanks!
How are you trying to access it over the internet? You'll need a VPN into your network for that. You wouldn't want it any other way.
 
Last edited:
  • Like
Reactions: balnazzar

KarelG

Member
Jan 29, 2020
30
8
8
[...]

1. I am just a machine learning engineer with little knowledge of microprocessors. more specifically I do not know what exactly are IOMMU, Ifop and similar. May you provide a link where such stuff is clearly explained? Googling provided low quality results.
[...]
IOMMU is like MMU but for devices, that means using IOMMU your OS may remap high RAM into the 0-4GB range where your device may pick it up and work on the data (save to drive/send to network/whatever). Without IOMMU your OS needs to make sure (usually!) that IO data are copied from high RAM to low 0-4GB RAM range to initiate/perform IO. That's this way since usually devices supports only 32bit DMA and from 32bit you get this narrow 4GB IO window.
 
  • Like
Reactions: balnazzar

balnazzar

Member
Mar 6, 2019
105
7
18
IOMMU is like MMU but for devices, that means using IOMMU your OS may remap high RAM into the 0-4GB range where your device may pick it up and work on the data (save to drive/send to network/whatever). Without IOMMU your OS needs to make sure (usually!) that IO data are copied from high RAM to low 0-4GB RAM range to initiate/perform IO. That's this way since usually devices supports only 32bit DMA and from 32bit you get this narrow 4GB IO window.
Thanks. Why one hears about it in reference to AMD processors (and not, afaik, to Intel ones)?
 

KarelG

Member
Jan 29, 2020
30
8
8
Thanks. Why one hears about it in reference to AMD processors (and not, afaik, to Intel ones)?
Probably due to less AMD market segmentation (e.g. intel reluctantly pushed VT-d down the CPU market pipe IIRC) and probably since AMD got it first.

Intel uses VT-d. AMD uses IOMMU.
Not so easy. IOMMU is more or less general term and since Intel marketing was more innovative and came with VT-d term, AMD can't be behind and replaced their usage of "IOMMU" with "AMD-Vi" term. Otherwise IOMMU is available also on IBM POWER, Sun SPARC and ARM (v8).
 
  • Like
Reactions: balnazzar

vangoose

Active Member
May 21, 2019
268
70
28
Canada
I ordered one similar to this one except memory will be Micron and no GPU since it's a server.

Can't wait to get it.

My old storage server is showing it's age, X5-2620v1 can't catch up with modem NVME and multiple 10GbE.
 
  • Like
Reactions: TXAG26

TXAG26

Active Member
Aug 2, 2016
274
81
28
You should like it! It will be a great reasonably priced replacement for an E5-2620v1.
 

james23

Active Member
Nov 18, 2014
419
80
28
48
This machine with all of the parts, above, and 1 SSD idles around 86w in Windows 10 and with Prime95 running 32 threads at max heat/stress setting, it stays at 3.1 GHz and does not go above 60 C with "Optimal" fan profile selected in IPMI.
what is the watts draw when running p95 max, on all 32 cores, please?

tks
 

TXAG26

Active Member
Aug 2, 2016
274
81
28
My baseline system, as of today, idling at the desktop in Windows 10 with a 10GBE adapter, 1x SSD and Nvidia 1660 Super video card is 98w, which jumps up to 232w total when I fire off the default settings in Prime95. The wattage values are reported in IPMI from SMLink in the Supermicro 1200w PSU. It idles around 86w at the Windows log-in screen when the video card isn't being used.

My baseline system load also consists of a 4x 80mm fan wall in a CSE-736AC case, plus a 92mm rear exhaust fan, a 92mm CPU heatsink fan, two fans on the video card, 1x 40mm fan on the 10GBE adapter and 1 fan in the PSU. This is a Supermicro "SuperQuiet" setup, so most of the fans are only running between 1600-2400 rpm. It's very quiet.
 
  • Like
Reactions: james23

TXAG26

Active Member
Aug 2, 2016
274
81
28
BTW, I have a 16-core CPU (7302P), with 32 threads. The 232w - 98w = 134w number is running Prime95 w/32 threads.
 
  • Like
Reactions: james23

lpallard

Member
Aug 17, 2013
216
6
18
@TXAG26

I am in the process of spec'ing a similar build (although with a combination of gaming & pro GPU's) for a daily linux based driver with frequent windows use via KVM. Host OS will probably end up being Linux Mint or a similar easy to use linux distro for day to day use and some casual gaming, but I need to be able to run a Windows 10 VM baremetal like performance for CAD / Engineering applications (FEA, 2&3D CAD, 3D printing, etc).

Dual booting with Linux & Windows sounds like a possible solution, but since I will use Windows much less than Linux, and I hate rebooting every time you want to switch OS, I'd rather flip a VM on, use it simultaneously with the host OS, then shut it down when not in use.

I'm gonna go AMD for sure, but I still hesitate between EPYC and Threadripper. I was gonna initially go for Threadripper because of the higher clock speeds which I think are more important than core count for my needs (games & CAD programs use only a few cores at best) but I dont have a huge desire for gaming stuff and all of the Threadripper mobo's are for extreme gamers or some stuff like that ...

Another thing to consider with me: each computer I build, I keep until forever. Current one is 10 years old but started showing its age and I'm expecting a failure within 6 to 8 months, and it cant do what I need to do so.... This is why I'd rather spend money on quality and stability than gamer features since I'm not gonna replace it ion 4 or 5 years anyways...

RAM: I always wondered about Supermicro RAM. Why did you buy Supermicro RAM instead of, lets say Kingston or Samsung or ....?

Chassis: How do you like that Supermicro tower case? Expensive AF but interesting. I believe the PSU comes with the case, and SAS backplane... Am I right? Are the PSU's redundant?

GPU: Why did you buy a gaming GPU instead of a professional one? (Quadro, Tesla or Radeon Pro)?
 

TXAG26

Active Member
Aug 2, 2016
274
81
28
RAM: For best compatibility, its always recommended to use ram that has been tested and verified by Supermicro to work with your board. The specific Supermicro ram chips I have are actually made by Samsung, and they include Supermicro's warranty with them as well. Usually there is no price premium for getting Supermicro ram vs something else, so I always go with Supermicro ram when it is available.

Chassis: The chassis works well for what I wanted it for - primarily the PSU with the SMBus/PMBus aware power supply (it shows power consumption and stats in the Supermicro IPMI interface which is pretty slick). I've been running the 7302P & 1660 Super at fill blast with Folding@Home to fight COVID-19, so power consumption has been up a bit this month! I also wanted it for the 8x SAS3 backplane, which directly connects to the LSI 3008 chipset on the H11SSL-NC board. I have 4 of the SAS3 trays hooked up to the SATA ports on the motherboard, and 4 hooked up to the LSI 3008 chipset, which makes swapping SSD's in and out nice. Instead of using VM's, I have a couple of different OS's that I occasionally need installed on different SSD's and I pop them in when I need to boot to them. This case is probably a bit overkill for a workstation, but I liked the utilitarian look of it and the above-mentioned features. It's black, plain, and doesn't have LED's, so that's a win for me.

GPU: I almost pulled the trigger on a Quadro, but Nvidia released their "Studio" drivers when I was spec'ing out this build which is what I needed the GPU for anyways. The MSI 1660 Super that I selected for this build has been great, the fans are quiet, massive heatsinks/heat pipes, and there are two tiny cables that you can unplug to completely turn off the rainbow LEDs on the card. Best of all, this is one of the few cards that actually has a "fan stop" feature that works. Hence, if you're not running the graphics hard, the fans completely stop and the entire card is quiet for 2D desktop tasks. The fans ramp up to about 30-40% when running heavy video editing or Folding@Home and are still very quiet. This is at least 2-3x the card that I could have gotten for the price, compared to a Quadro, so I've been happy with it.

As far as whether to go Rome or Threadripper, if you game or do anything that would really benefit from single thread or needs just a couple of threads, I would give Threadripper or Ryzen a hard look. Rome is good when your applications can really load up all the cores, but I think you'll find the max 3.3 GHz somewhat limiting if your applications are more in the single thread category. If the $1400 Threadripper 3960 is above your budget (it was mine!), the Ryzen 9 3950X 16-Core 3.5 GHz/4.7GHz may be worth a look.
 

Attachments

balnazzar

Member
Mar 6, 2019
105
7
18
Mind that TR cannot use RDIMMs/LRDIMMs, which limits the maximum amount of memory. Don't know if that can be an issue for you.
 

lpallard

Member
Aug 17, 2013
216
6
18
Maximum amount of RAM shouldn't be a big issue. I guess 64GB would be plenty enough (16 for host OS, 32 for main work VM, and another 16 for buffer/smaller VM's).

TR is pretty interesting for the max clock speeds VS core counts. But Supermicro lack support for TR.

Quite honestly, I'd pair a Supermicro board with a 16 core TR and 64GB RAm and call it a day, but they only support Epyc or Xeon AFAIK.

I am a bit like you in the sense that I much prefer the utilitarian VS entertainment nature of computers. I really dont like gaming stuff and RGB crap.... I also tend to keep my machines until they fail, and have casually done minor upgrades in the past, but usually I keep them running until they fail.

In that perspective, knowing how Supermicro is reliable, and well built, I would tend to opt for them for the motherboard, but they have no TR products...

Maybe when I am ready to pull the trigger on this project, their product line will have evolved??? I doubt but one can "dream"...
 
  • Like
Reactions: balnazzar