First, everyone, thank you for this post.
Now I was able to follow the directions posted here to get ESXi working with my GTX1080.
hypervisor.cpuid.v0 with value of FALSE
I then diabled the vmware SVGA driver and set the diplay to 2, which was my monitor.
The above while worked, left my computer is a continuous cylce of BSOD whenever I connected to it with either vnc or rdp.
Eventually I added the option pciPassthru0.msiEnabled = "FALSE" and this stopped my BSODs. So thanks @
soild79.
Now, I followed through to get a USB controller pcie passthrough and I got the card mentioned by
@marcoi:
StarTech.com 2 Port PCI Express PCIe SuperSpeed USB 3.0 Controller Card with SATA Power - Dual Port PCI Express USB 3 Adapter PEXUSB3S23
I set the device to passthrough just fine, and then added the slot onto my vm, so my vm's passthrough are now:
View attachment 8197
And windows recognizes it:
View attachment 8198
But no device are seen when I connect them... Any help would be appreciated.