Will it work? Epyc 7F72 + 25 NVMe's

[bill3000]

Build’s Name: Epyc NVMe Envy
Operating System/ Storage Platform: Proxmox + JBOD
CPU: Epyc 7F72 (used)
Motherboard: Asrock ROMED8-2T
Chassis: TBD
Drives: 25x WD Black SN850x 1TB
RAM: 4x 64gb 3200 Supermicro
Add-in Cards: 6x Asus Hyper M.2 x16 Gen 4
Power Supply: TBD
Other Bits: targeting ~$4k USD

Usage Profile: Home lab for running VMs for testing distributed databases and Ceph.

I'm neither a hardware nor VM guy, so I'd appreciate hearing what's flawed in my reasoning.

EPYC 7F72

• 7Fx2 combines cheap 7002 used prices with high frequency cores
• Higher MHz seems better than more cores, threadrippers are pricey
• 7F72 has 24 cores, the highest of the 7Fx2 options
• Xeon has too many SKUs to understand what's best for me

Asrock ROMED8-2T

• 24 cores at 4 NVMe per PCIE x16 means 6 PCIE slots are needed
• Asrock ROMED8-2T has 7 PCIE x16 slots

RAM

• 8gb per VM x 24 cores == 192GB
• 4 x 64GB is a decent compromise of room to upgrade vs cost

 

[NPS]

Whats flawed? The thought to need 24 VMs each with one core and one SSD to learn about distributed systems.

 

[seany]

For how much this system is going to cost (which isn't totally crazy in the grand scheme of things); I would just max the ram up front and get all 8 channels going. 64gb sticks aren't _that_ bad these days MEM-DR464MC-ER32 64GB DDR4 3200MHz ECC 2Rx4 RDIMM Memory for SuperMicro servers | eBay

 

[bill3000]

Whats flawed? The thought to need 24 VMs each with one core and one SSD to learn about distributed systems.

Ah, well, distributed systems is my career. This feels like the largest performance testing setup my wife and wallet might support. I could save and buy 24 Craigslist HP 800 G4's, but then I'd have to explain why I brought home twenty four computers.

 

[msvirtualguy]

For the record I don't have 24 M.2 NVMe disks, but I do have 12 spread across three AICs not including the 2 onboard. I am currently testing TrueNAS Scale same board Epyc Milan 7B13 64core CPU 512GB RAM. Just setting it up...plan on testing this weekend.

 

[nexox]

I would say that unless your workload is very storage IO intensive and quite low CPU utilization you'd get a better perf testing rig making VMs share NVMe drives (multiple namespaces probably) and spending the money you save there on more CPU.

 

[MrGuvernment]

Databases = ram. A well designed database sits as much in ram as possible..

 

[ano]

non enterprise nvme + ceph = no good.

 

[NPS]

Ah, well, distributed systems is my career. This feels like the largest performance testing setup my wife and wallet might support. I could save and buy 24 Craigslist HP 800 G4's, but then I'd have to explain why I brought home twenty four computers.

I think for development and performance testing in most cases it does not matter if have 10 or 24 VMs. Things might only get interesting in the hundredths or thousandths of nodes from a software perspective. And in reality I think the problems you can not solve with 3-5 nodes will not be solved with a setup you proposed. The devil of real world performance is in the detail of real systems and you even rule out a factor that is not a detail at all: networking. I think it's a waste of money.

 

[nexox]

In my experience you have to solve problems at 10 and 24 nodes before you can even start to look at what issues show up at 100 or 1000, but yeah one box with any amount of SSDs is going to be of limited use testing scale out. Maybe try to grab one of those Supermicro 3U systems with 8 or 12 nodes and 10G networking, they're not cheap or quiet but then neither is the system you're suggesting.

 

[bill3000]

Thank you all for your input. I appreciate all the arrows pointing to "overkill", in particular that the compute is unbalanced against the IOPS; and building out a 200gpbs network to use that IOPS is beyond my budget. Looking into ZFS sLog + L2Arc and Open CAS, I think I'd be better off paying for more compute, and tiering my IOPS more intelligently. In addition, I worry the PCIE x16 slots can't be treated as 4 PCIE x4, which tends to be a BIOS feature. Finally, I question if the overall PCIE power delivery is adequate to support all those drives. I suspect I'd need an HBA-base solution.

 

[bill3000]

Oooh, neat. All 7 PCIE slots support bifurcation: https://forums.servethehome.com/ind...ock-rack-romed8-2t-support-bifurcation.29794/.

 

[T_Minus]

I'm enjoying the contrast of the current builds... one person going for xLanes per nvme device, and the other cramming all 24 into one HBA both with unique additional problems Thanks for never making it dull around here everyone