Disk or HBA passthrough

Discussion in 'FreeBSD and FreeNAS' started by weust, Apr 11, 2018.

  1. weust

    weust Member

    Joined:
    Aug 15, 2014
    Messages:
    211
    Likes Received:
    18
    I've asked this question in Windows Server/Hyper-V part too, but perhaps it's better to place it here.
    https://forums.servethehome.com/ind...assthrough-the-same-as-hba-passthrough.19386/

    I am wondering if ZFS wants direct disk access or HBA access.
    If I understand it correctly, it's all about direct disk access, and that (for example) ESXi can do HBA passthrough is handy for all disks at once, but if you passthrough each disk directly (like you can with Hyper-V) it would be the same?

    Just looking for clarification here.
     
    #1
  2. Linda Kateley

    Linda Kateley New Member

    Joined:
    Apr 25, 2017
    Messages:
    21
    Likes Received:
    4
    The answer to this question is the same as the answer to most technical questions.. is.. it depends.

    So zfs is designed to manage the disk by default. It wants to do things like tell the disk occasionally to flush it's cache. Sometimes when you pass through you may run into problems. When I had customers dealing with large san frames we just tried to make sure you accommodated for not owning the disk.

    Comparing it to what happens in hyper-v isn't really apples to apples though. It's like apples to kale :)

    The best answer is give zfs what it wants.. it wants the whole disk.
     
    #2
  3. weust

    weust Member

    Joined:
    Aug 15, 2014
    Messages:
    211
    Likes Received:
    18
    But that's what Hyper-V can do. Pass the disk entirely, just not the controller it's connected to.
    Or did I miss something somewhere?

    I'm not talking about creating a VHD(X) the maximum size of the disk it can hold, and attach that to the VM.
     
    #3
  4. Linda Kateley

    Linda Kateley New Member

    Joined:
    Apr 25, 2017
    Messages:
    21
    Likes Received:
    4
    So the reason I don’t like this is because it has none of the benefits of zfs. Pooling, checksums, growth, hybrid. With zfs I want to use the bandwidth of all the disks and use ram to cache stuff. 1 disk 1 workload.. might as well not use zfs or virtualization. Just buy a server and run an os. With virtualization you want to pool resources and make them available to multiple workloads.. including disk. Take 10 disks and carve out datasets or zvols to put workload on..
     
    #4
  5. weust

    weust Member

    Joined:
    Aug 15, 2014
    Messages:
    211
    Likes Received:
    18
    I just noticed I missed out an important part here.
    The pass through would be to a VM running NAS4Free, for example.
    That would then handle the ZFS part.
     
    #5
  6. Linda Kateley

    Linda Kateley New Member

    Joined:
    Apr 25, 2017
    Messages:
    21
    Likes Received:
    4
    Ok gotcha.. hopefully more than one disk :) There are a whole bunch of horror stories around this. I personally have had really good luck. I just make disk cache flushing gets turned off.
     
    #6
  7. weust

    weust Member

    Joined:
    Aug 15, 2014
    Messages:
    211
    Likes Received:
    18
    At least six disks.

    Bought a Dell T320 recently, and trying out different scenarios.
    My current hypervisor is Hyper-V 2016 and a separate NAS, but trying to go with a AIO model.
    And I'm waiting for prices of the 8TB WD Red to drop below 200 Euro, so I've got a while to test things.
     
    #7
    Last edited: Apr 12, 2018
  8. ttabbal

    ttabbal Active Member

    Joined:
    Mar 10, 2016
    Messages:
    626
    Likes Received:
    175
    I haven't tried hyper-v, so I could be wrong. However, my thought would be that passing disks vs controllers leaves the host OS involved. So that's one more layer things can go wrong at. All it would take is the driver in Windows messing something up and you're going to have issues. If you already plan to use 6 drives, an 8 port HBA doesn't lose much when you pass the controller into the VM. And it leaves some options for l2arc/slog or just adding more storage later.

    This is the kind of thing where everything may well work great, until it doesn't. And it's basically uncharted territory, so you will be largely on your own should something happen, as there just isn't experience to back you up available. If you decide to go there, have a 100% backup available.
     
    #8
  9. weust

    weust Member

    Joined:
    Aug 15, 2014
    Messages:
    211
    Likes Received:
    18
    In case of Hyper-V and disk passthrough, the VM gets exclusive access to the disk.
    The physical disk has to be offline in the host OS before a VM can even use it.
    So afaik understand it, the host OS has nothing to do with the physical disk anymore.

    I did came across a nice article talking about disk passthrough in Hyper-V, and why it's no longer feasible anymore.
    Except in situations where you need to use a USB disk for whatever reason.

    The Hyper-V Pass-Through Disk: Why its time has Passed

    Of course, this does not talk about using a VM with ZFS.

    I understand it's uncharted territory, and for now I'm just testing and trying things and getting to know ZFS better.
    Before I build a permanent setup it will be months from now I think.
     
    #9
  10. ttabbal

    ttabbal Active Member

    Joined:
    Mar 10, 2016
    Messages:
    626
    Likes Received:
    175
    But the host is still running the controller/HBA. This may or may not be an issue, I have no way to know. Just pointing out possible things to consider. If you're willing, build it and beat the crap out of it and let us know. It would be interesting information for a lot of people. It's not something I would want to do with my data, and I don't have a Windows 10 license to do it anyway. It might be great, it might fall over and die. Only one way to know for sure. :)
     
    #10
  11. weust

    weust Member

    Joined:
    Aug 15, 2014
    Messages:
    211
    Likes Received:
    18
    I will setup Hyper-V 2016 on the T320 and see how things go.
    Not moving data from my Synology NAS untill I have found what I want to run in the end anyway.
     
    #11
  12. whitewidow

    whitewidow New Member

    Joined:
    Apr 23, 2018
    Messages:
    1
    Likes Received:
    1
    #12
    Evan likes this.
  13. weust

    weust Member

    Joined:
    Aug 15, 2014
    Messages:
    211
    Likes Received:
    18
    Read the article in the train to work. Totally missed that somehow.
    Very interesting.

    I installed Windows Server 2016 last night on the T320. Needs a bunch of drivers installed, but after that I will work on that blog and a VM using some ZFS appliance.

    Thanks for the link!
     
    #13
  14. weust

    weust Member

    Joined:
    Aug 15, 2014
    Messages:
    211
    Likes Received:
    18
    I didn't go through with Server 2016. Couldn't be bothered with the missing drivers for hardware.

    I installed ESXi 6.7 and eventually set up my own FreeBSD installation with ZFS and Samba 4.
    Going to swap FreeBSD for HardenedBSD to get security too.
     
    #14
    whitey likes this.
  15. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,684
    Likes Received:
    821
    Now you are on the right track!
     
    #15
  16. weust

    weust Member

    Joined:
    Aug 15, 2014
    Messages:
    211
    Likes Received:
    18
    It's "a" track.

    Also, not going to bother with Samba anymore. Trying NFS now.
     
    #16
Similar Threads: Disk passthrough
Forum Title Date
FreeBSD and FreeNAS Can the S3 disk be read/write? Mar 27, 2018
FreeBSD and FreeNAS FreeNas script for a convenient way to report installed disks Aug 28, 2017
FreeBSD and FreeNAS FreeNas - 2 disks of pool with 512B size Apr 9, 2017
FreeBSD and FreeNAS Do you guys spindown disks in zfs pools? Mar 17, 2017
FreeBSD and FreeNAS performance of KVM/QEMU disks on ZFS volumes Dec 27, 2016

Share This Page