Cheap 16-core build with IPMI: what's possible currently?

hmw

Active Member
Apr 29, 2019
236
77
28
My Tyan S8030GM2NE arrived today. Provantage has 50 of these in stock at the cheapest price I have seen anywhere. Now waiting patiently for the HP upgrade kit for the EPYC 7302P to arrive :)

  • It's a barebones board, doesn't ship with anything except two SATA cables and two M.2 pins
  • No manuals - just the cables and I/O shield
 

Kev

Active Member
Feb 16, 2015
439
84
28
37
Cool. Avadirect also let me know it was inbound to them and they price matched plus no taxes for me.
 

hmw

Active Member
Apr 29, 2019
236
77
28
Have you considered the ASRock Rack X470D4U?


I've been running it with a 3950X and 2x32GB ECC UDIMMs for a few weeks. So far, so good. I haven't tried GPU passthrough to a VM yet, though.
I did consider. However I got the 7302P for $670 and the S8030GM2NE for $380. So in the end it was cheaper than the X470/3950x combo. Not to mention that I can put 128GB on this motherboard and get 200GB/sec memory bandwidth
 

TXAG26

Active Member
Aug 2, 2016
274
81
28
I did consider. However I got the 7302P for $670 and the S8030GM2NE for $380. So in the end it was cheaper than the X470/3950x combo. Not to mention that I can put 128GB on this motherboard and get 200GB/sec memory bandwidth
This boils down my decision earlier in 2020 to also go with the 7302P & Supermicro H11SSL-NC board.
 

hmw

Active Member
Apr 29, 2019
236
77
28
This boils down my decision earlier in 2020 to also go with the 7302P & Supermicro H11SSL-NC board.
Indeed - with 8 DIMMs and ESXi with 3-4 VMs running, each VM gets a healthy chunk of the 200GB/sec memory bandwidth.

As an aside - anyone have experience of (1) buying from memory.net and (2) if this Samsung 16GB RDIMM memory module works with EPYC?
 
  • Like
Reactions: TXAG26

Tom S

New Member
Jan 31, 2017
13
5
3
34
What model and make are the 32GB ECC UDIMMs?

They're in short supply, but seem to be more available lately.

The X470D4U has a halfway decent selection of DRAM overclocking options. If I get adventurous I may bump up the voltage and try to push for the speed up to 3000MHz with decent timings.
 

Tom S

New Member
Jan 31, 2017
13
5
3
34
I did consider. However I got the 7302P for $670 and the S8030GM2NE for $380. So in the end it was cheaper than the X470/3950x combo. Not to mention that I can put 128GB on this motherboard and get 200GB/sec memory bandwidth
Good price on the 7302P.

I picked up the 3950X for around $700 shipped and the X470D4U came in around $230 from an eBay steal. Mainstream AMD processor prices decline rapidly whenever the next generation is released, so an X470D4U with a cheap Ryzen 3600 ($170-ish new) could be a good starter system for someone with a clear upgrade path to 12 and 16-core parts when they get cheap in a year or two.

As for memory bandwidth: That was my only real concern, coming from a 4-channel DDR4 Xeon system. My workloads on this machine are 90% compiling, and as far as I can tell the dual channel memory isn't too much of a bottleneck. 16 cores with high clockspeeds goes a long way. I'm seeing over 4GHz all-core boost speeds much of the time.

Would be interesting to run some compile benchmarks on both of our systems to test memory throughput (and double L3) vs. raw clockspeed of the 3950X.

If I get bored, I might try forcing single-channel memory to see how much it impacts my build times. I've always wondered.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,269
428
83
If I get bored, I might try forcing single-channel memory to see how much it impacts my build times. I've always wondered.
Single-channel was a death sentence for performance on Zen 1 due to the CPU topology but it's much less of a concern on the Zen 2 chips. It's only a single data point, but TPU did a basic review when the Ryzen 3000 chips came out:


CPU-bound processes like rendering or compilation see negligible decreases in performance. On my own testing with ffmpeg, dropping to one DIMM it gave maybe a 2-3% reduction in performance, be interesting to see if your compilation process is any different.
 

Tom S

New Member
Jan 31, 2017
13
5
3
34
Thanks for that TechPowerUp link. That's a good test. Surprising to see that little of an impact from dropping down to single-channel RAM.

I retired my old workstation (Intel X99 5820K with quad-channel DDR4) and VM host (Xeon E3v6 with quad-channel DDR4) to upgrade to this dual-channel DDR4 system. It felt strange to step down in memory bandwidth while increasing the CPU processing power by 2-3X, but apparently memory bandwidth isn't even close to a bottleneck for my workloads.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,269
428
83
That's basically it, if your workload isn't memory-bandwidth constrained then single channel might not as bad a limitation. Rendering and video encoding are usually relatively predictable in what data needs to be shunted to the CPU and when, games and AI workloads tend to be a lot less predictable so low-latency, high-bandwidth memory access is usually more important there. Sadly I'm not aware of any way to measure real-time memory bandwidth.

Zen 1 was something of an aberration because its topology meant that inter-CCX communication (Infinity Fabric) was directly related to memory performance.

The IO die used in Zen 2 allowed inter-core communication to be decoupled from the memory interface so you once again only got memory problems on tasks that were actually memory-starved.
 

hmw

Active Member
Apr 29, 2019
236
77
28
I see the Tyan S8030 has two SFF-8654 ports. Each being a SFF-8654-8i - is supposed to be able to power two NVMe drives each. But that would require a SFF-8654 -> 2 * U.2 cable

I've been searching in vain for such a cable. Tyan even teases about the existence of this cable ...

1590213247234.png


... but of course FRU-CS-1230 doesn't exist on their site and Tyan's customer service is non-existent. Support tickets for info on cable or Samsung memory RDIMMs have gone unanswered, getting a bit anxious now and wondering if I should have waited for the SuperMicro H12SSL-xx boards :)
 

Spartacus

Well-Known Member
May 27, 2019
688
265
63
Austin, TX
  • Like
Reactions: hmw

hmw

Active Member
Apr 29, 2019
236
77
28
I found one option, not sure how shady of site you wanna try its sata power instead of molex though:

That said its not cheap at $100 and its a 6 week lead time, but if you truly need all 4 U.2 drives it is what it is, (there are plenty of single cable options which im sure you found)
$104:eek: ... and a 3 week lead time :(

the 4i cables aren’t cheap either. Slim SAS SFF-8654 4i Straight to SFF-8639 U.2 Cable - Newegg.com for example is $36


I think I’ll go ahead with a couple of 1TB 970 EVOs...
 

SINN78

Active Member
Apr 3, 2016
152
30
28
41
My Tyan S8030GM2NE arrived today. Provantage has 50 of these in stock at the cheapest price I have seen anywhere. Now waiting patiently for the HP upgrade kit for the EPYC 7302P to arrive :)

  • It's a barebones board, doesn't ship with anything except two SATA cables and two M.2 pins
  • No manuals - just the cables and I/O shield

Do they just have the wrong pic?