ESXi or FreeNAS GPU passthrough for deep learning?

Discussion in 'Machine Learning, Deep Learning, and AI' started by josh, Dec 5, 2019.

  1. josh

    josh Active Member

    Joined:
    Oct 21, 2013
    Messages:
    289
    Likes Received:
    55
    Hi all,

    Trying to build a machine for both media storage and deep learning. Thinking of the following setup, not sure if it makes any sense:

    2x E5-2670
    128GB DDR3
    ASUS 2070 Turbo
    4xPM953 NVMe MIRROR (Scratch)
    4xHUSML1640201 RAIDZ2 (VM OS)
    6xWD EasyStores RAIDZ2 (Large storage)

    The idea is to run ESXi baremetal and virtualize FreeNAS which will export the SSD RAIDZ2 over iSCSI which will be imported by ESXi as datastore for VM OS.
    The spinners will be used for large files, exported as NFS and Plex.
    GPU passthrough to a Ubuntu VM which will be used to run tensorflow. Currently unsure if I should import the NVMe as iSCSI or pass it directly to the VM.
    Will the 8 vCPU limitation on free ESXi be a bottleneck?

    Thoughts and criticisms appreciated.

    Thanks!
     
    #1
  2. hoping

    hoping New Member

    Joined:
    Dec 5, 2019
    Messages:
    15
    Likes Received:
    4
    Based on your needs, Sounds good to me.
     
    #2
  3. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,704
    Likes Received:
    591
    -Gpu passthrough of non quadro cards to ESXi VMs might prove a challenge - make sure you read up on that before committing
    -Raid Z2 of a 4 disk pool is not going to work - switch to 2x2 mirrors. Or is that a 4 x mirror/stripe of a z2 vdev with 4+2 drives?
     
    #3
  4. josh

    josh Active Member

    Joined:
    Oct 21, 2013
    Messages:
    289
    Likes Received:
    55
    Not sure what you mean by 4 disk pool not going to work? The recommended sizes for Z2 are 4,6,10.
    I spent 2 days trying to get passthrough to work on ESXi and I gave up. Currently testing out XCP-ng.
     
    #4
  5. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,704
    Likes Received:
    591
    That looks like a 4 disk pool ?;)
     
    #5
  6. josh

    josh Active Member

    Joined:
    Oct 21, 2013
    Messages:
    289
    Likes Received:
    55
    I still don't get you. A 4 disk Z2 works the last time I checked. It's not 4+2, just 4.
     
    #6
  7. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,704
    Likes Received:
    591
    I have no clue if a Z2 pool with 4 disks works, but this is not going to be a good layout for VMs as you basically get 1 disks worth of write IO.
     
    #7
  8. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,883
    Likes Received:
    1,509
    Maybe that's enough for his workload? His disks are SSD and NVME it looked like, and spinners for storage only it said.
     
    #8
  9. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,704
    Likes Received:
    591
    Everything is possible :D
    But a single HUSML might be underpowered for multiple VMs.
    Ok, I have to concede that my performance expectations are probably not representative, so maybe you are right;)
     
    #9
  10. josh

    josh Active Member

    Joined:
    Oct 21, 2013
    Messages:
    289
    Likes Received:
    55
    Yup. The SSDs are 12G SAS and the NVMes are PCIe. I'm highly doubtful of these being the bottleneck even if I'm only getting 2 disks of speed in Z2.
     
    #10
  11. josh

    josh Active Member

    Joined:
    Oct 21, 2013
    Messages:
    289
    Likes Received:
    55
    I'm supposed to get 2 disks for write and 4 disks for read in a 4 disk Z2 right?
     
    #11
  12. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,704
    Likes Received:
    591
    Quoting an iX blog (Six Metrics for Measuring ZFS Pool Performance Part 2 - iXsystems, Inc. - Enterprise Storage & Servers):

    N-wide RAIDZ, parity level p:

    • Read IOPS: Read IOPS of single drive
    • Write IOPS: Write IOPS of single drive
    • Streaming read speed: (N – p) * Streaming read speed of single drive
    • Streaming write speed: (N – p) * Streaming write speed of single drive
    • Storage space efficiency: (N – p)/N
    • Fault tolerance: 1 disk per vdev for Z1, 2 for Z2, 3 for Z3 [p]
    Now these are theoretical values, real life performance might be below

    Now for VMs depending on what you do you will have lots of random IO and only a bit of streaming; therefore the general recommendation is to build VM datastores IOPS centric
     
    #12

Share This Page