Thought I’d share this after only silently reading here so far: I tinkered around with my totally „unsupported“ rig (2080Ti, X399/Threadripper 1920x) the last couple of days and finally figured out how to passthrough both an „onboard chipset“ USB3 controller as well as the 2080Ti, in particular on 6.7U3. Getting this to work „once“ is not such a big problem, but it gets tricky if both devices are to „survive“ and still work after a reboot of the VM.
As some of you are aware, until now - and without the settings below - the GPU would throw the famous „error code 43“ after a reboot of the VM and only work again once the ESXi host is rebooted as a whole.
So far the most common (only?) workaround seemed to be to disable the GPU in the device manager before VM-reboot and enable it again once the VM is up again. Some automated this procedure with respective scripts.
The following worked with Windows 10 (1903) VMs started both in BIOS as well as EFI Mode and did not require any manual / scripted interventions.
The secret is:
1. Edit passthru.map and delete / comment-out the default NVIDIA setting („#10de ffff bridge false“). This general/wildcard setting for all NVIDIA devices does not work - at least not for my 2080Ti FE. Instead I needed a more granular
approach: I had to set d3do for ALL Nvidia „sub-devices“ of my graphic card EXCEPT the GPU itself. The GPU has now no override anymore and ESXi will use its defaults (which works for the GPU, but unfortunately in particular not for the USB controller...). In my case the passthru.map now looks like this:
...
#NVIDIA
#Audio
10de 10f7 d3d0 false
#Serial Bus
10de 1ad7 d3d0 false
#USB
10de 1ad6 d3d0 false
...
(Reboot the host after you made changes to the passthru.map)
2. Add ALL devices of your NVIDIA graphic card as PCI Passthrough devices to your VM, i.e. for the 2080Ti: GPU, Audio, USB and Serial Bus.
NOTE: I did not test in detail whether this is really necessary or whether it is enough to only add one or certain devices in addition to the GPU. Adding all seemed reasonable to get all devices properly resetted at reboot (was also in line with some snippets I read somewhere) and it worked.
NOTE2: would be interesting to see what happens if you just delete the NVIDIA settings without adding anything else... ah... I need more time...
3. In addition to the usual hypervisor.cpuid.v0 =FALSE“, set for ALL NVIDIA passthrough devices EXCEPT the NVIDIA USB controller of the 2080Ti:
pciPassthru0.msiEnabled = FALSE
NOTE: replace the 0 (zero) after pciPassthru with the correct number of your devices respectively.
NOTE for AMD:
Last but not least: my board (X399D8A-2T) / X399 chipset / Threadripper is a bitch when it comes to ESXi, in particular USB passthrough. Didn’t work properly even with 6.5U2. For the onboard controller to survive a VM-Reboot I also had to modify the passthru.map like this:
# AMD
1022 ffff d3d0 false
Probably instead of the ffff-wildcard the specific device works as well, but I was lazy and haven’t tested further (yet)...
Works now also in 6.5U2 and 6.7U3, with either EFI/BIOS VM startup setting.