Could Someone Give me Advice for Building a High-Performance Storage Server for Virtualization Environment?

[Padrickk]

Hello there,

I am in the process of planning a high-performance storage server build that will primarily be used in a virtualization environment. My main goals are to achieve high IOPS; low latency; and sufficient capacity to support multiple virtual machines running resource intensive applications. Although, I am still debating the best hardware configuration and would greatly appreciate some guidance.

I am leaning towards NVMe SSDs for their speed; but I am also considering a tiered storage approach with NVMe for hot data and SATA SSDs or even HDDs for cold storage. What would be the best strategy to balance cost and performance?

I am considering RAID 10 for the NVMe array to ensure redundancy and performance. Is RAID 10 the optimal choice for such a setup; or would another RAID level (like RAID 5 or 6) be more suitable given my needs?

Should I opt for a dedicated hardware RAID controller, or is software RAID sufficient given the power of modern CPUs? Also; any specific brands/models you would recommend?

The server will be running either VMware ESXi or Proxmox. Are there any specific hardware configurations that work better with one over the other? I am also curious about experiences with passthrough for NVMe devices in these environments.

Also, I have gone through this post; https://forums.servethehome.com/index.php?threads/advice-for-blue-prism-a-home-server-workstation.34808/ which definitely helped me out a lot.

Considering the potential heat generated by multiple NVMe drives; what cooling solutions would you suggest?

Thanks in advance for your help and assistance.

 

[Falloutboy]

I know what I would use but I am sure that there are some that would disagree with me. I personally would go for an Supermicro H12SSI, it has 5 PCIe 4 x 16 Slots, 2 PCIe 4 x 8 Slots, two 4 Lane PCIE 4.0 x 4 connectors on board and an 8 lane 8654 connector.

If you were wanting to go Raid on NVMe - get some of the cheap four NMVe bifrucating controllers from ALIExpress - your choice of fan or heatsink version run say 20 2tb NVMes spread across 5 of those controllers using ZFS Z2, make two of the drives Samsung PMA9A3s and use those for the intent log and cache, this would give you 32Tb of Raid space for your VM's..

I am personally building a system like this at the moment but it will have 15 x 10Tb Sata 3 drives in Z2 as well as 8 x 2tb PCIe v4 nvmes for a total of 12TB of VM.

 

[louie1961]

What kind of network connection will you have between your storage server and your virtualization servers?

 

[nabsltd]

The server will be running either VMware ESXi or Proxmox.

I would not run any sort of hypervisor on a storage server...I would have a bare-metal OS install that provides storage to other machines running whatever (hypervisor, user workstation, etc.).

 

[Fallen Kell]

Agreed on not running this through a hypervisor. In the past if you were simply passing though SAS or RAID controllers, I would say, yes, you could use a hypervisor, but once you go to NVMe, you are going to want to avoid a hypervisor and run on the baremetal as you will have significant performance hit otherwise.

 

[Sean Ho]

Agreed on avoiding hardware raid for NVMe; it just tends to be another point of failure. Also I'm not sure what kind of chassis you're thinking of, but signal attenuation from backplane, cables, and cards is a consideration, especially at PCIe 4.0 and up. Redriver vs retimer vs switch. There have been a few threads with folks' experience using specific combinations of cards and cables.

 

[thatdoodle]

Before anybody can answer your question properly, you need to give us a better idea of what you will be virtualising.

> high IOPS; low latency; and sufficient capacity to support multiple virtual machines running resource intensive applications

You need to quantify this a bit better - How many VMs roughly? When you say resource-intensive applications, what does this mean? Huge cluster of SQL servers, or something else? I ask as there are some frankly ridiculous suggestions in this thread considering we don't really know what you will be doing in your environment.

If you're already running some or all of these VMs you have the benefit of being able to run LiveOptics first to get an accurate picture of your I/O requirements.

> I am considering RAID 10 for the NVMe array to ensure redundancy and performance. Is RAID 10 the optimal choice for such a setup; or would another RAID level (like RAID 5 or 6) be more suitable given my needs?

Depends on the capacity of your chosen disks, amount of disks, and the level of redundancy you require. I'd say your choice is between RAID 10 for speed and redundancy or RAID-Z for capacity and cost savings. Parity-based RAID levels can be run in software, but really should be run on a controller to not lose performance. I'd recommend getting a better controller than the cheapest you can find - Battery-backed cache is important if you don't want to potentially corrupt your array

> I am also considering a tiered storage approach with NVMe for hot data and SATA SSDs or even HDDs for cold storage. What would be the best strategy to balance cost and performance?

Depends on the workload/LiveOptics output. Tiered HDD and nvme will likely be fine, but it's impossible to say without numbers.

Please also remember that RAID is not a backup. Things can and do go wrong during rebuilds (especially on larger disks) and disk MTBF means that when one goes, the others usually aren't far behind...

Some users are suggesting you run your VMs on a separate server rather than on the same server as your storage - While I agree in principle, the cost of building a SAN, a separate hypervisor, then hooking them up with iscsi/FC is going to add what I would consider too much complexity and cost for the benefit in a home environment.

Another user said this;

> you are going to want to avoid a hypervisor and run on the baremetal as you will have significant performance hit otherwise.

Both ProxMox and ESXI are type 1 hypervisors so your VMs are running on the bare metal. I'd recommend a separate drive/2 disk RAID 1 for the OS install but you really don't need to run true bare metal installs.

 

[Sean Ho]

Yes, well said. First step of upgrades should be to gather metrics on the current workload with the current hardware to find pain points.

 

[mattventura]

Consider that you probably don't want the storage server to be a single point of failure for your entire virtualization environment. You'd ideally want something to cluster or replicate them, so don't put all your eggs in one basket. Proxmox and ESXi both make it fairly easy to set that up.

 

[i386]

24x optane p5800x 1.6tb ssds should do the job