Threadripper Proxmox build, need suggestions

[jena]

I work in a research lab that we recently purchased a Threadripper 3970x workstation from a System Integrator.
It is a better better deal than Dell-Intel, which would have cost us twice as much.
The role is to run Proxmox as base Hypervisor and run multiple Windows and Linux for scientific computations.

Build:
Operating System/ Storage Platform: Proxmox
CPU: Threadripper 3970x
Motherboard: MSI TRX40 (two M.2 SSD on MB, came with a Dual M.2 on add-in PCIE card)
Chassis: CORSAIR 110Q
Drives: See details delow
RAM: 256GB Non-ECC 3200MHz CL16 RAM (not ECC option from this S.I.)
Add-in Cards: Single RTX2080Ti (we will add Ampere RTX3080-ish later)
Power Supply: 1000W Corsair
Other Bits:

The case is CORSAIR 110Q, which only has two 3.5in bay and one 5.25 bay.
I can use adapter to convert 5.25 to 3.5.
Not sure how to JerryRig the fourth 3.5 HDD in it.

Plan to:

1. ASUS HYPER M.2 X16 GEN 4 card to have Quad NVME. Total of 6 (we won't use MSI's Dual NVME card).
2. [VM-pool] 6 x 2TB WD SN750 NVME SSD at about $300 each (RAID10, or RAID Z2 for VMs)
3. [BootDrive] 2 x 1TB Samsung 860 Evo as Proxmox boot drive (RAID1-mirror at install), and some ISOs (I know its way too big, but I have heard that Proxmox wearout boot drive fast?)
4. [BulkDataPool] 2 x 12TB HGST UltraStar as bulk data storage (RAID1 mirror as one VDEV, add future VDEVs)

My question:
Based on my use case, maybe about 20 VMs simultaneously (not all at full load). Around 2~3 VMs at heavy CPU and GPU load simultaneously.

1. Can I skip using ZIL(SLOG) on all-SSD-VM-pool?
   because I don't see any performance gain using ZIL and it doubles the write wear on SSD or I have to buy separate Optane drives for ZIL.
2. Should I use RAID10 or RAIDZ2 for all-SSD-VM-pool? I think RAID-Z2 has better redundancy but have heard that RAID-Z2 will have performance gotchas in small block file application (not sure if VM is considered as small block operation). If RAID10 has two drive fails within the same VDEV (especially during rebuilt), my pool falls apart.
3. Should I use the hugh 1TB RAID1 boot drive as L2ARC for BulkDataPool?
4. Does Proxmox wearout boot drive fast?
5. Does ZFS on Proxmox support Trim for SSD?
6. Since I only have 256GB RAM for the entire system, mostly for computations. It is sufficient if I only give VM-Pool a 8GB ARC cache and give BulkDataPool a 16GB ARC?

Any other suggestions?

 

[Peanuthead]

Older TR may be a better value.

 

[Wolvez]

1. Since you are already using nvme drives in your VM pool a ZIL isn't really going to help and may actually hurt performance.
2. RAID10 for performance. RAIDZ2 for redundancy. I don't know how important you data is and I'm not familiar with those SSDs.
3. I wouldn't use a consumer level drive for L2ARC or ZIL.
4. If you aren't using your boot drive for VM storage and/or something like L2ARC Proxmox isn't much different than a normal Debian install.
5. Yes.
6. Not sure if you are able to set ARC per pool. It may be a newer feature I don't know about. The answer on ARC size depends on your workload. If you have 20 VMs all trying to read from your HDD pool performance is going to tank. In that case getting most of your active data into the ARC will give you a huge increase in speed. If your VMs are mostly taking turns reading larger files and not using the same data over and over ARC doesn't really help much.

 

[bob_dvb]

Something I recently purchased to play with is a 6x2.5in, 5.25in bay enclosure. I've populated it with SATA SSDs for the hell of it. Could be an alternative use for your 5.25in bay...

Alternatively, a couple of companies do 4xU.2 5.25in bay enclosures and there are U.2 HBA controllers.

 

[jena]

Something I recently purchased to play with is a 6x2.5in, 5.25in bay enclosure. I've populated it with SATA SSDs for the hell of it. Could be an alternative use for your 5.25in bay...

Alternatively, a couple of companies do 4xU.2 5.25in bay enclosures and there are U.2 HBA controllers.

Thanks for the tip. I saw a few of those. might be good for my 2.5inch SSDs
I was wanting to have more 3.5inch bay for large HDD storage.

 

[jena]

1. Since you are already using nvme drives in your VM pool a ZIL isn't really going to help and may actually hurt performance.
2. RAID10 for performance. RAIDZ2 for redundancy. I don't know how important you data is and I'm not familiar with those SSDs.
3. I wouldn't use a consumer level drive for L2ARC or ZIL.
4. If you aren't using your boot drive for VM storage and/or something like L2ARC Proxmox isn't much different than a normal Debian install.
5. Yes.
6. Not sure if you are able to set ARC per pool. It may be a newer feature I don't know about. The answer on ARC size depends on your workload. If you have 20 VMs all trying to read from your HDD pool performance is going to tank. In that case getting most of your active data into the ARC will give you a huge increase in speed. If your VMs are mostly taking turns reading larger files and not using the same data over and over ARC doesn't really help much.

Thanks for the reply.

6. My VMs will be on 6x2TB nvme RAID-Z2. The [BulkDataPool] with 2x12TB HDD is the one that might need ARC. The computation data sets are usually not repetitive, however during computation tuning, we could run one data set over and over with different parameters.

 

[jena]

Older TR may be a better value.

We already bought the workstation. now we are considering upgrade the storage ourselves.