ESXi or FreeNAS GPU passthrough for deep learning?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

josh

Active Member
Oct 21, 2013
615
190
43
Hi all,

Trying to build a machine for both media storage and deep learning. Thinking of the following setup, not sure if it makes any sense:

2x E5-2670
128GB DDR3
ASUS 2070 Turbo
4xPM953 NVMe MIRROR (Scratch)
4xHUSML1640201 RAIDZ2 (VM OS)
6xWD EasyStores RAIDZ2 (Large storage)

The idea is to run ESXi baremetal and virtualize FreeNAS which will export the SSD RAIDZ2 over iSCSI which will be imported by ESXi as datastore for VM OS.
The spinners will be used for large files, exported as NFS and Plex.
GPU passthrough to a Ubuntu VM which will be used to run tensorflow. Currently unsure if I should import the NVMe as iSCSI or pass it directly to the VM.
Will the 8 vCPU limitation on free ESXi be a bottleneck?

Thoughts and criticisms appreciated.

Thanks!
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
-Gpu passthrough of non quadro cards to ESXi VMs might prove a challenge - make sure you read up on that before committing
-Raid Z2 of a 4 disk pool is not going to work - switch to 2x2 mirrors. Or is that a 4 x mirror/stripe of a z2 vdev with 4+2 drives?
 

josh

Active Member
Oct 21, 2013
615
190
43
-Gpu passthrough of non quadro cards to ESXi VMs might prove a challenge - make sure you read up on that before committing
-Raid Z2 of a 4 disk pool is not going to work - switch to 2x2 mirrors. Or is that a 4 x mirror/stripe of a z2 vdev with 4+2 drives?
Not sure what you mean by 4 disk pool not going to work? The recommended sizes for Z2 are 4,6,10.
I spent 2 days trying to get passthrough to work on ESXi and I gave up. Currently testing out XCP-ng.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
I have no clue if a Z2 pool with 4 disks works, but this is not going to be a good layout for VMs as you basically get 1 disks worth of write IO.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
I have no clue if a Z2 pool with 4 disks works, but this is not going to be a good layout for VMs as you basically get 1 disks worth of write IO.
Maybe that's enough for his workload? His disks are SSD and NVME it looked like, and spinners for storage only it said.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Everything is possible :D
But a single HUSML might be underpowered for multiple VMs.
Ok, I have to concede that my performance expectations are probably not representative, so maybe you are right;)
 

josh

Active Member
Oct 21, 2013
615
190
43
Maybe that's enough for his workload? His disks are SSD and NVME it looked like, and spinners for storage only it said.
Yup. The SSDs are 12G SAS and the NVMes are PCIe. I'm highly doubtful of these being the bottleneck even if I'm only getting 2 disks of speed in Z2.
 

josh

Active Member
Oct 21, 2013
615
190
43
Everything is possible :D
But a single HUSML might be underpowered for multiple VMs.
Ok, I have to concede that my performance expectations are probably not representative, so maybe you are right;)
I'm supposed to get 2 disks for write and 4 disks for read in a 4 disk Z2 right?
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Quoting an iX blog (Six Metrics for Measuring ZFS Pool Performance Part 2 - iXsystems, Inc. - Enterprise Storage & Servers):

N-wide RAIDZ, parity level p:

  • Read IOPS: Read IOPS of single drive
  • Write IOPS: Write IOPS of single drive
  • Streaming read speed: (N – p) * Streaming read speed of single drive
  • Streaming write speed: (N – p) * Streaming write speed of single drive
  • Storage space efficiency: (N – p)/N
  • Fault tolerance: 1 disk per vdev for Z1, 2 for Z2, 3 for Z3 [p]
Now these are theoretical values, real life performance might be below

Now for VMs depending on what you do you will have lots of random IO and only a bit of streaming; therefore the general recommendation is to build VM datastores IOPS centric