Epyc NVME 7443P

lihp

Active Member
Jan 2, 2021
179
46
28
Build’s Name: Epyc NVME
Operating System/ Storage Platform: RHEL 8.4
CPU: EPYC 7443P
Motherboard: H12SSL-NT
Chassis: Fractal Design Meshify 2 XL
Drives: 2x 980 Pro 1TB (OS), 4x PM1735 3.2 TB (RAID5 RAIDIX Guest VM FS, Fast Storage, Cache), 8 (16)x WD Ultrastar 14 TB SATA
RAM: 256 GB
Add-in Cards: 1x CX455A 100Gb IB EDR, 4x PM1735 (see storage)
Power Supply: Fractal Design ION+ 860P
Other Bits:

Usage Profile:
This machine is destined to be a KVM-server to fulfill various tasks as well as being an iSER server.
  1. Backup: Customer/Internet server backup: I have various servers over the Internet as well as servers I manage onsite. For that I need a Borg target with decent storage.
  2. Server Virtualization: infrastructure servers (AD, Web,...), a lot of testing with VMs and containers as well as the ability to "bare metal restore" customer servers in VMs.
  3. SAN: I/O - local environment: I am doing alot of work with files of sizes 20MB - 1GB+ and sometimes videos. I hate wait times and target 10GB/s + transfer.
  4. Client Virtualization: local and remote workstations for production (Office), graphics (Adobe) VMs for browsing, VMs for testing,.. also bootable via iSER (IB)
  5. Security: basically I separate private and business workstations and also separate browsing and productive work on one client by being able to either boot them or (regular case) starting them as VMs under Gnome.
  6. High-I/O project: I plan to saturate 100G to my workstation in iSER and make my current and additional future workstations diskless.
  7. Crypto: Obligatory Chia

My little project is nearing it's first goal: most parts are delivered except for the PM1735 drives. I hope to have them by the end of the month latest. So today is build day, which I plan to do with my 4-year-old son, just to make him the youngest child ever to build a Milan server at this performance level (ok, I admit he wants to see Spiderman on youtube - meaning he is HIGHLY motivated).

IMG_8956.jpg


FAQ
  1. Why a tower case?
    With today's solutions I don't see a need for a full blown 19" rack. In fact should I need more storage or server capacity, I can still upgrade storage and/or get another server or switch to a dual CPU platform.
  2. Why this power supply?
    To be frank, I never saw a high end power supply fail during operations. While redundancy may be cool there, I am still quite confident in my backups for the worst case possible. Additionally the whole server also has an online backup.
  3. Why a USV?
    Ups and downs in power supply by power providers are quite common (2+ times a year on average). For the most part the USV is to even out those spikes or lo's. Still I will configure it also for power down of the server. Just because I have it.
  4. Why the Dark Rock and liquid metal?
    I have no safe reports on temperatures for Milan during heavy operations. And low temperatures preserve hardware for prolonged times. So at similar prices, this was a no-brainer.
  5. Samsung 980 PRO SSD for a server?
    Hell sure, they fit exactly to the usage profile of the OS. Performance-wise SN850 would have been a better choice, but they lack in encryption...
  6. 256 GB is over the top....!?!
    Not really. Make that 3 servers running minimum, some test servers, a web LANMP stack and several workstations simultanously. Together thats almost 90 - 128GB RAM already. Add test machines and caching and suddenly nothing is left over. Apart from that in cost:GB that was my sweet spot - most ram per buck.
  7. Why a 7443P?
    Because it is my dream CPU. Clock rates in the high end area aka high single core performance, least cost:core, low TDP. It's imho the coolest processor in Epyc Milan land.
  8. Why a Meshify?
    It will probably be in my office, so I wanted it to look good. Also the storage config is awesome with 18-24x 3.5" (however you count/see it) and 4+ 2.5" slots.
  9. Thats a Wicked Bunny gaming keyboard and mouse? ;) ?
    Yes. I love instant feedback of this mechanical keyboard and the backlights. It's what I enjoy most, when typing.
    (PS - Edit: and it's waterproof - I can't count how many great keyboards entered the trash because of spilled coffee, even after cleaning. Having a keyboard I can use even underwater is imho cool - no worries on spilled coffee ever again.)
 
Last edited:

lihp

Active Member
Jan 2, 2021
179
46
28
Build Time

Once upon a time... there was a lil boy... Damn, that's another story. So parts arrived and chief engineer Jonathan decided it's time to start with the motherboard by opening the CPU tray.
IMG_8959.jpg

A firm, decisive grab and off is the socket cover:
IMG_8963.jpg

My heart was pounding, sweat on my cheeks as our technician puts the CPU into places and closed the socket at once, before anyone could react... (at that point I was sure something broke).
IMG_8969.jpg

Time to insert the RAM. Afterwards placing the cooler socket bracers and tighten the mounting screws:
IMG_8984.jpg

Pro tip: 4-year olds trying to firmly tighten those screws achieve the perfect amount of force: enough to hold all together, but never too tight.IMG_8987.jpg

Part 2 almost done also:
IMG_8991.jpg

Place the additional screws:
IMG_8995.jpg

Mount and secure the CPU cooler:
IMG_8999.jpg

Insert the final screws:
IMG_9001.jpg

Time for the NVME drives:
IMG_9005.jpg

Number 1 done, number 2 to go:
IMG_9008.jpg

One last pin to close:
IMG_9009.jpg

All nicely done and firmly seated:
IMG_9010.jpg

Jonathan made it clear: storage config is the way to go - so some screws need to be removed.
IMG_9019D.jpg

With the crucial tasks done, our chief technician takes a nap and ordered the rest of us to complete the build. After all his job was done. Perfectly rested its time to remove the last bits of protective cover.:
IMG_9044.jpg

Only to order all involved persons that its time to clean up the mess we made (he really did that: "Aufräumzeit es ist soweit" - german for "clean up time, it is my rhyme")
IMG_9046D.jpg

Will it turn on (I was sure it wont):
IMG_9049.jpg

Well I was wrong - that specific attitude prevailed:
IMG_9053.jpg

A RHEL 8.4 installation later its time to properly test the new server:
IMG_9054.jpg

Well, that´s it for now about building a server with a 4-year old as chief technician. Some tips:
  • I thought it's possible to oversee everything or control it. Forget it! Impossible! No way. Several times where I was sure its broken. Nonetheless I had a perfect overnight burnin test - so all is fine and I was wrong.
  • Try to make sure they got enough to drink, ideally far away from the electronics.
  • Have some alcohol tissues closeby for cleaning, because they will find a way to spill fluid over the board or CPU nonetheless.
  • Enjoy the time, it´s a blast.
First few notes on the server:
  • The 4 SSDs hopefully arrive by the end of the month. That´s when the fun really starts. I will keep you posted.
  • I am actually positively surprised by the system performance. I assumed something like 49,000 to 53,000 in multicore passmark and 2,750 single core. Surprise, surprise we hit around 59,000 multi core and 2,930 single core (values in my signature are actually the worst result ; passmark has currently issues in automated uploads, hence only this result until later).
  • Cooling is over the top. During full load burnin testing over prolonged time, the CPU had a max temperature of 63°C in a 25°C room. For the most part I blame the liquid metal being way over the top in quality.
  • The build is super quiet. Nothing can be heard except a faint noise of air moving. Yet no mechanical noise, no HDD noise, nothing.
  • Single core performance according Passmark is ahead of most HEDT-CPUs (TOP 15 single core), last gen consumer and gaming CPUs. Count in the large CPU cache, cores, PCIe 4.0 and the PCIe lanes and you got a great option for almost any build right now. Of course this will change with next gen Threadripper and Ryzen. But overall it gives testament to the Milan design.




 
Last edited:

lihp

Active Member
Jan 2, 2021
179
46
28
Do not forget to share more pictures :)
Sure thing, I like to do a more interesting write-up, yet a first glimpse (I was serious in having my 4-year old do it - dead serious). He actually inserted also the RAM sticks, as well as the CPU, tightened the screws and started to mount the CPU cooler brackets. As well as much more later:
IMG_8982.jpg

PS - Before you ask: yes accidents happen this way - my server board smells after "passion fruit yogurt". It's cleaned with alcohol, yet the smell persists... At least its unique ;)
 

lihp

Active Member
Jan 2, 2021
179
46
28
Did you manage to get the extra drive trays?
Waiting for the HDD tray type-B from Fractal still. Even inquired at Fractal directly. No answer. Thats why I wrote 8 (16) SATA drives.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,368
5,497
113
I have a TR Pro build that will likely make an appearance on the STH main site/ YouTube in the next few weeks. It is in the Define 7 XL, and I wish it was in the Meshify 2 XL instead :-/
 

lihp

Active Member
Jan 2, 2021
179
46
28
I have a TR Pro build that will likely make an appearance on the STH main site/ YouTube in the next few weeks. It is in the Define 7 XL, and I wish it was in the Meshify 2 XL instead :-/
The Meshify 2 XL is a blast. I wasn't aware how much fun it is.

On your TR Pro build: I ran several tests and benchmarks. The 7443 (24 core) trades blows with the Epyc 7R32 (48 core) and the TR Pro 3975WX (32 core). Easily getting ahead of the 7R32 (full load - all cores) and slightly below the 3975WX (full load - all cores). In single core performance the 7443 is ahead of both. With Milan available - especially the 7443P - I fail to see any reason to go for TR Pro builds from a simple performance perspective. If you consider costs and per core licenses, this becomes even worse for the TR Pro.

I am really curious if AMD pursues TR Pro on the long run. Imho consumer (Ryzen), HEDT (Threadripper) and server (Epyc) is all that is needed. TR Pro had it's time for a few month before Milan arrived - imho that's it. I am curious on your view on that.
 
Last edited:

lihp

Active Member
Jan 2, 2021
179
46
28
Another person trusting "hardware encryption" in storage devices?
is only a tip of an iceberg.
Thank you. I am aware of that paper. The listed security holes luckily don't apply in my case and additionally a TPM 2.0 module will be added later on. Yet I see it - most likely similar to you - as a clear hint on potential additional security weaknesses, which are not know - yet. Still this does not stop me from deploying it as long as my system does not enter a lock down state (I am wary of that ;) ). And what's better to test security and review it myself on my own system :D?
 
Last edited:

Rand__

Well-Known Member
Mar 6, 2014
6,193
1,544
113
Very interesting build, will be interesting to see if it can meet expectations (100G) :)
 

tinfoil3d

QSFP28
May 11, 2020
557
198
43
Japan
Thank you. I am aware of that paper. The listed security holes luckily don't apply in my case and additionally a TPM 2.0 module will be added later on.
My inner critic just refuses to believe in lack of debug port/options in tpm. as well as just about anything hardware. Now go ahead and try proving it's 100% trustworthy to yourself :)
The more something mentions "security" the more obvious it is something is very wrong here.
 

lihp

Active Member
Jan 2, 2021
179
46
28
My inner critic just refuses to believe in lack of debug port/options in tpm. as well as just about anything hardware. Now go ahead and try proving it's 100% trustworthy to yourself :)
The more something mentions "security" the more obvious it is something is very wrong here.
Seems you didn't fully see that I am way more on your page. Because... how am I meant to break it, if I didn't have time to test it myself :D ?
 

lihp

Active Member
Jan 2, 2021
179
46
28
Very interesting build, will be interesting to see if it can meet expectations (100G) :)
It's actually a wager, which started on reddit (for single file, single thread, synchronous loads). I am curious myself, because I told: Milan systems can do that. I hoped with ZFS (close to impossible, but not 100% sure) but almost sure I can on EXT4 and XFS (via direct samba) and also almost sure via iSER (iSER is imho quite a safe bet).
 
Last edited:

Rand__

Well-Known Member
Mar 6, 2014
6,193
1,544
113
It's actually a wager, which started on reddit (for single file, single thread, synchronous loads). I am curious myself, because I told: Milan systems can do that. I hoped with ZFS (close to impossible, but not 100% sure) but almost sure I can on EXT4 and XFS (via direct samba) and also almost sure via iSER (iSER is imho quite a safe bet).
Are you looking to local or remote performance? Its quite the challenge for a single thread...
 

lihp

Active Member
Jan 2, 2021
179
46
28
Are you looking to local or remote performance? Its quite the challenge for a single thread...
LAN performance via EDR (IB), hence the CX455A. It's a challenge definitely since the bottleneck here is single core performance. Zen2 was barely able to reach 9 - 13.5 GB/s locally on the file system alone (depending on file system due to core performance). Here RAIDIX comes into play, which may be able to tip it over and do the trick.
 
Last edited:
  • Like
Reactions: Patrick

lihp

Active Member
Jan 2, 2021
179
46
28
Ah, sounds interesting. Too bad there is no community edition ;)
Free for 4 NVME drives. That's the reason I ordered 4x PM1735. But its correct I also got access to perpetual licenses by them.
 

Rand__

Well-Known Member
Mar 6, 2014
6,193
1,544
113
Really? Could you provide a link? Have not been able to find that :)