Disk or HBA passthrough

[weust]

I've asked this question in Windows Server/Hyper-V part too, but perhaps it's better to place it here.
https://forums.servethehome.com/ind...assthrough-the-same-as-hba-passthrough.19386/

I am wondering if ZFS wants direct disk access or HBA access.
If I understand it correctly, it's all about direct disk access, and that (for example) ESXi can do HBA passthrough is handy for all disks at once, but if you passthrough each disk directly (like you can with Hyper-V) it would be the same?

Just looking for clarification here.

 

[Linda Kateley]

The answer to this question is the same as the answer to most technical questions.. is.. it depends.

So zfs is designed to manage the disk by default. It wants to do things like tell the disk occasionally to flush it's cache. Sometimes when you pass through you may run into problems. When I had customers dealing with large san frames we just tried to make sure you accommodated for not owning the disk.

Comparing it to what happens in hyper-v isn't really apples to apples though. It's like apples to kale

The best answer is give zfs what it wants.. it wants the whole disk.

 

[weust]

But that's what Hyper-V can do. Pass the disk entirely, just not the controller it's connected to.
Or did I miss something somewhere?

I'm not talking about creating a VHD(X) the maximum size of the disk it can hold, and attach that to the VM.

 

[Linda Kateley]

So the reason I don’t like this is because it has none of the benefits of zfs. Pooling, checksums, growth, hybrid. With zfs I want to use the bandwidth of all the disks and use ram to cache stuff. 1 disk 1 workload.. might as well not use zfs or virtualization. Just buy a server and run an os. With virtualization you want to pool resources and make them available to multiple workloads.. including disk. Take 10 disks and carve out datasets or zvols to put workload on..

 

[weust]

I just noticed I missed out an important part here.
The pass through would be to a VM running NAS4Free, for example.
That would then handle the ZFS part.

 

[Linda Kateley]

Ok gotcha.. hopefully more than one disk There are a whole bunch of horror stories around this. I personally have had really good luck. I just make disk cache flushing gets turned off.

 

[weust]

At least six disks.

Bought a Dell T320 recently, and trying out different scenarios.
My current hypervisor is Hyper-V 2016 and a separate NAS, but trying to go with a AIO model.
And I'm waiting for prices of the 8TB WD Red to drop below 200 Euro, so I've got a while to test things.

 

[ttabbal]

I haven't tried hyper-v, so I could be wrong. However, my thought would be that passing disks vs controllers leaves the host OS involved. So that's one more layer things can go wrong at. All it would take is the driver in Windows messing something up and you're going to have issues. If you already plan to use 6 drives, an 8 port HBA doesn't lose much when you pass the controller into the VM. And it leaves some options for l2arc/slog or just adding more storage later.

This is the kind of thing where everything may well work great, until it doesn't. And it's basically uncharted territory, so you will be largely on your own should something happen, as there just isn't experience to back you up available. If you decide to go there, have a 100% backup available.

 

[weust]

In case of Hyper-V and disk passthrough, the VM gets exclusive access to the disk.
The physical disk has to be offline in the host OS before a VM can even use it.
So afaik understand it, the host OS has nothing to do with the physical disk anymore.

I did came across a nice article talking about disk passthrough in Hyper-V, and why it's no longer feasible anymore.
Except in situations where you need to use a USB disk for whatever reason.

The Hyper-V Pass-Through Disk: Why its time has Passed

Of course, this does not talk about using a VM with ZFS.

I understand it's uncharted territory, and for now I'm just testing and trying things and getting to know ZFS better.
Before I build a permanent setup it will be months from now I think.

 

[ttabbal]

But the host is still running the controller/HBA. This may or may not be an issue, I have no way to know. Just pointing out possible things to consider. If you're willing, build it and beat the crap out of it and let us know. It would be interesting information for a lot of people. It's not something I would want to do with my data, and I don't have a Windows 10 license to do it anyway. It might be great, it might fall over and die. Only one way to know for sure.

 

[weust]

I will setup Hyper-V 2016 on the T320 and see how things go.
Not moving data from my Synology NAS untill I have found what I want to run in the end anyway.

 

[whitewidow]

Maybe try discreet device assignment to pass the hba to the VM?
Passing through devices to Hyper-V VMs by using discrete device assignment

Sent from my LG-LS997 using Tapatalk

 

[weust]

Read the article in the train to work. Totally missed that somehow.
Very interesting.

I installed Windows Server 2016 last night on the T320. Needs a bunch of drivers installed, but after that I will work on that blog and a VM using some ZFS appliance.

Thanks for the link!

 

[weust]

I didn't go through with Server 2016. Couldn't be bothered with the missing drivers for hardware.

I installed ESXi 6.7 and eventually set up my own FreeBSD installation with ZFS and Samba 4.
Going to swap FreeBSD for HardenedBSD to get security too.

 

[whitey]

I didn't go through with Server 2016. Couldn't be bothered with the missing drivers for hardware.

I installed ESXi 6.7 and eventually set up my own FreeBSD installation with ZFS and Samba 4.
Going to swap FreeBSD for HardenedBSD to get security too.

Now you are on the right track!

 

[weust]

It's "a" track.

Also, not going to bother with Samba anymore. Trying NFS now.