ESXi all-NVMe storage array requirements

ectoplasmosis

Active Member
Jul 28, 2021
115
45
28
I'm looking to move a Proxmox VE host to ESXi, to experiment with Nvidia vGPU (RTX A5000 shared between three guests).

The Proxmox host currently has an all-NVMe ext4 striped storage array as 'hot' storage for media ingest, before the content gets archived to a magnetic ZFS pool. The NVMe array consists of 4x Samsung PM9A3 Gen4 drives in software RAID0, connected via a 16-lane PCIe Gen4 bifurcation riser.

Does ESXi support something similar, without using a hardware NVMe RAID controller?
 

zer0sum

Well-Known Member
Mar 8, 2013
672
338
63
As far as I'm aware there is no native software raid capability at al in ESXi :(

If it's just a home lab you could do something silly like pass the disks through to a freenas VM, and then present that storage back to ESXi using iSCSI or NFS :p
 
  • Like
Reactions: ectoplasmosis

ectoplasmosis

Active Member
Jul 28, 2021
115
45
28
As far as I'm aware there is no native software raid capability at al in ESXi :(

If it's just a home lab you could do something silly like pass the disks through to a freenas VM, and then present that storage back to ESXi using iSCSI or NFS :p
Ah, that’s a shame!

Are there any other hypervisors that have a free/community tier, support software NVMe RAID and Nvidia vGPU?
 

gregsachs

Active Member
Aug 14, 2018
463
139
43
As a role it's not free (part of windows server 20*) and I'm not sure if hyper-v server (the "free" gui free hypervisor) supports storage spaces...
100% sure that h-v 2016 will do storage spaces on a single host.
 

markarr

Active Member
Oct 31, 2013
421
120
43
nvidia vGPU has drivers out there for xen, kvm, hyper-v, and esxi. Only used esxi so cant speak to usability of the others.
 
  • Like
Reactions: ectoplasmosis

nk215

Active Member
Oct 6, 2015
365
115
43
48
I'm looking to move a Proxmox VE host to ESXi, to experiment with Nvidia vGPU (RTX A5000 shared between three guests).

The Proxmox host currently has an all-NVMe ext4 striped storage array as 'hot' storage for media ingest, before the content gets archived to a magnetic ZFS pool. The NVMe array consists of 4x Samsung PM9A3 Gen4 drives in software RAID0, connected via a 16-lane PCIe Gen4 bifurcation riser.

Does ESXi support something similar, without using a hardware NVMe RAID controller?
Make sure you look at the license cost for vGPU from Nvidia. The last card that has free vGPU is the kepler card (Grid K2/K1).
 
  • Like
Reactions: ectoplasmosis

ectoplasmosis

Active Member
Jul 28, 2021
115
45
28
Make sure you look at the license cost for vGPU from Nvidia. The last card that has free vGPU is the kepler card (Grid K2/K1).
I’m happy to pay for vGPU licenses, just need to find the right hypervisor to standardise on...
 

Dean

Member
Jun 18, 2015
102
10
18
46
Agreed - no nvme raid driver, etc.

BUT, VMware just released details for esxi 7.0 cU3 - supposed to support nvme/tcp protocol. I'd check into it. Release is late October I believe.
 
  • Like
Reactions: ectoplasmosis

jerryxlol

Member
Nov 27, 2016
38
5
8
30
Uff i believe you have travelled to the road that is full of licensing :)

VGPU on esxi? - I dont know if its possible to share one gpu more nodes. I am planing horizon on two nodes and both HAS TO HAVE nv a40.
Sharing one gpu over more vms is possible with predefined GPU power and memory. NVIDIA vGPU for vSPhere/ESXi is a premium VMware-licensed feature | NVIDIA Nvidia license ... (recently it has not been and now ? pay for every feature ... )

Esxi native NVME raid? Esxi as is does not provide complex storage layer. So for single host not possible. Is it possible to do FTT when you have it licensed. (it does copy of your data to the more hosts / datastores) (https://www.vmware.com/products/vsphere/fault-tolerance.html)

Multiple hosts Vmware VSAN. (i strongly recommend to study what are the main cons of vsan solution and its pricey)


It depends on the usage ... If it is for production then anything below is higly risky:

NVME passtrough to the VM - > provision NFS or ISCSI back to the esxi. VM NEEDS TO HAVE RESERVED CPU AND MEMORY (pcie passtrough req)
I have tried freenas and openmediavault, I use commonly centos. .. runs okay, but as i mentioned NO PRODUCTION. Limited snapshot capability, etc.