Troubleshooting GPU passthrough ESXi 6.5

[Ch33rios]

First, my build:

Gigabyte MX31-BS0
Xeon E3-1230v5
2x16GB DDR4
MSI 970 GTX
Esxi 6.5

VTd is enabled in the BIOS and best I can tell all other settings are good to go. Boot up the system and go to the web client and on the tab for hardware mgmt I can see the GTX card detected but it is grayed out. Oddly enough the audio controller on the card IS available for passthrough (note I had an HDMI cable plugged into my monitor at the time...Not sure if that makes any difference)

I know Nvidia is troublesome getting passthrough but all my reading indicated that was more for the driver piece in the guest VM.

My question here is what can I do, if anything, to further diagnose what COULD be the problem? Any CLI specific things to check (still new to esxi)? I'm also thinking of trying a KVM based distribution like unRAID or Proxmox since it seems people have the most luck in those vs esxi. But obviously before I fully go down those paths I figured I'd try to get some additional diagnostics.

Thanks in advance!

 

[Rand__]

I assume you have set Primary Display to sth different than PCI-e?
And you are not using the GTX to display anything?

 

[marcoi]

Nvidia a PITA but i got my 750 TI working after a lot of research and playing.


1. In the gui enable the HDMI but dont reboot. Copy down the pci info, you will need it for next steps.

2. you need to enable esxi SSH access then putty into box and edit some settings.
3. Run the following from putty after logging in as root
vi /etc/vmware/esx.conf
4. add under device section the nvidia card. Since you enabled the HDMI you will already see a section for that. you can copy the three lines and modify them for the main card then paste them back in.
In my case, I had to add the first three lines and update the device id to match what was shown in the gui.

5. Save the changes and reboot the host. You will eventually lose the esxi boot screen as the card will be passthrough mode.
6. Log back into the GUI and you will now see both card and hdmi enabled even though the card is still greyed out.

First part done!!!

Next when building the VM. I did windows 10 pro. Build out the OS without adding any pci devices. you need to install esxi guest drivers and VNC, I used tightvnc. You should also download Nvidia drivers but dont run them yet. Power off the VM then edit the settings as follows:
1. Edit CPU and enable Expose hardware assisted virtualization to the guest OS.
2. Change the memory to reserve all memory.

3. Tab over to VM Options, then expand Advance and click on Edit Configuration

4. add parameter: hypervisor.cpuid.v0 with value of FALSE click on ok and save changes.

This part you might need to mess with and try it a few times
5. Start the VM back up and start the nvidia driver install. it will fail since you havent passed the card yet. but it will expand the files.
6. shutdown and add the PCI card and hdmi and usb passthought for keyboard to the settings then start it back up.
7. install the drivers. you might need to reboot, etc..
8. At this point, you should see the display change over to your vm display. If you do you can then disable the esxi vga card from device management. if you dont it will crash on you. if you need to work on the vm use VNC as the console will not work right.

Good luck hope this helps.

 

[Ch33rios]

Nvidia a PITA but i got my 750 TI working after a lot of research and playing.


1. In the gui enable the HDMI but dont reboot. Copy down the pci info, you will need it for next steps.
View attachment 4096
2. you need to enable esxi SSH access then putty into box and edit some settings.
3. Run the following from putty after logging in as root
vi /etc/vmware/esx.conf
4. add under device section the nvidia card. Since you enabled the HDMI you will already see a section for that. you can copy the three lines and modify them for the main card then paste them back in.
In my case, I had to add the first three lines and update the device id to match what was shown in the gui.
View attachment 4097
5. Save the changes and reboot the host. You will eventually lose the esxi boot screen as the card will be passthrough mode.
6. Log back into the GUI and you will now see both card and hdmi enabled even though the card is still greyed out.

First part done!!!

Next when building the VM. I did windows 10 pro. Build out the OS without adding any pci devices. you need to install esxi guest drivers and VNC, I used tightvnc. You should also download Nvidia drivers but dont run them yet. Power off the VM then edit the settings as follows:
1. Edit CPU and enable Expose hardware assisted virtualization to the guest OS.
2. Change the memory to reserve all memory.
View attachment 4098
3. Tab over to VM Options, then expand Advance and click on Edit Configuration
View attachment 4099
4. add parameter: hypervisor.cpuid.v0 with value of FALSE click on ok and save changes.
View attachment 4100
This part you might need to mess with and try it a few times
5. Start the VM back up and start the nvidia driver install. it will fail since you havent passed the card yet. but it will expand the files.
6. shutdown and add the PCI card and hdmi and usb passthought for keyboard to the settings then start it back up.
7. install the drivers. you might need to reboot, etc..
8. At this point, you should see the display change over to your vm display. If you do you can then disable the esxi vga card from device management. if you dont it will crash on you. if you need to work on the vm use VNC as the console will not work right.

Good luck hope this helps.

Awesome I will try this ASAP!! Thanks for the detailed steps!

Just out of curiosity, is this process easier on an AMD GPU? Is Nvidia that finicky?

 

[marcoi]

similar process but the AMD seems to play nicer with esxi and I was able to get my r9 290X running with less hassle. The Nvidia is like an exact science to get it working....

 

[RyC]

With AMD cards, there shouldn't be any need to edit the /etc/vmware/esx.conf or add the vm parameter. Just passthrough and go!

 

[Ch33rios]

similar process but the AMD seems to play nicer with esxi and I was able to get my r9 290X running with less hassle. The Nvidia is like an exact science to get it working....

Typical

I only really have a GTX 970 at my disposal so until I figure out all the quirks and finalise my setup, I can't justify buying something different.

Now for your setups, does the GPU 'power down' when the VM it's assigned to powers off? The very first post I read on reddit which talked about virtualizing a gaming setup mentioned how the card didn't even spin up the fans which I sort of read as it being powered down almost completely... Now that might be perhaps a bit unrealistic but I'm wondering if there is any potential for NOT blowing out the power consumption due to a beefy gaming GPU running all the time.

 

[marcoi]

@Ch33rios -I just tested it. with two VMs running, one being the HTPC with pass through it idled at 72 watts. When i shutdown the htpc vm it dropped down to 60. So my assumptions is it would drop the power down. but only one way to really tell is testing it out.

 

[nk215]

I am not sure if anything was changed from ESXi 6 to ESXi 6.5 but when I tried the pass-thru of GTX card (on ESXi 6 way back), it sort of worked but not completely.

+ Access the VM via RDP worked

+ Monitor plugged into the video out port of the card properly display the VM screen.

+ Access the VM via view client didn't work

Basically, it works as a multi-head game system but not as VDI remote login via horizon view.

On my GTX-580, then fan spin with the VM powered down (if I remember correctly). Haven't try it again for a while since I need horizon view for my zero clients.

 

[acmcool]

Did your audio work well?

 

[marcoi]

I havent tested the hdmi yet since the pc still sitting near my desk as i finish the build out but I did pass through the MB audio and that works fine. I plan to use the spdif out to receiver when the pc makes it way back to the tv area. I also got PowerDVD working and tested a blueray movie without issues using the blue-ray drive.

 

[marcoi]

on a side note vmtools no longer runs since it doesnt detect the OS running on a virtual host now lol.
It did run with AMD card.

 

[Rand__]

Basically, it works as a multi-head game system but not as VDI remote login via horizon view.

Same here with my recent tries on 6.5/Horizon 7

@nk215 - have you tried one of these ? (Or similar ones maybe)?
Amazon.com: CompuLab 4K Display Emulator (fit-Headless 4K): Computers & Accessories

 

[Ch33rios]

I am not sure if anything was changed from ESXi 6 to ESXi 6.5 but when I tried the pass-thru of GTX card (on ESXi 6 way back), it sort of worked but not completely.

+ Access the VM via RDP worked

+ Monitor plugged into the video out port of the card properly display the VM screen.

+ Access the VM via view client didn't work

Basically, it works as a multi-head game system but not as VDI remote login via horizon view.

On my GTX-580, then fan spin with the VM powered down (if I remember correctly). Haven't try it again for a while since I need horizon view for my zero clients.

I dont believe the 3rd point would be an issue directly for me since my primary use cases are it being a Steam Link streaming host (no real interaction unless Im doing maintenance which would be over RDP anyway) and then possibly down the road a full gaming desktop replacement.

Power consumption will be something to check...looks like my first Belkin WeMo insight is in my future since I cant seem to directly monitor power consumption on the board (my PSU doesn't have a PMBus plug)

 

[nk215]

Same here with my recent tries on 6.5/Horizon 7

@nk215 - have you tried one of these ? (Or similar ones maybe)?
Amazon.com: CompuLab 4K Display Emulator (fit-Headless 4K): Computers & Accessories

No I have not tried display emulator. I connected the monitor during my test however. if I remember correctly w/o a monitor attached, AMD driver wont even load properly.

I currently use a bunch of quadro 4000s as they fit my needs as a cheap grid workaround.

By the time grid cards get to $6-800 or so I'll just get one.

 

[Ch33rios]

@marcoi - so quick update. I got everything setup I believe and installed the nvidia drivers but Im getting the dreaded Code 43 at the moment. No output to the secondary video output.

Is there a specific order of operations for ensuring the hypervisor.cpuid.v0 setting is input? Like before the systems very first boot even? Perhaps I need to UNINSTALL vmware tools and just manually install the drivers one by one? Also do I need to do anything with the "Video Card" device in the VM settings? Your's shows custom settings but is it anything special?

Additionally, I've been unsuccessful in getting ESXi to allow me to pass a USB device connected to the host into a VM. I thought in ESXi 6.x you could just plug a device into a USB slot on the ESXi machine and then in the web client you could physically pass that device to the VM, there-by avoiding any need for an additional PCIe USB controller card...I'll keep trying I guess.

 

[marcoi]

if the usb device is a HID (keyboard or mouse) it wont pass through. You need to get a usb card either on the MB or PCIE and pass that into the vm with keyboard. if you need usb card this one works. Amazon.com: StarTech.com 2 Port PCI Express PCIe SuperSpeed USB 3.0 Controller Card with SATA Power - Dual Port PCI Express USB 3 Adapter PEXUSB3S23: Computers & Accessories make sure to get the eltron chipset.

As for 43, that means it still seeing it as a VM. so you might missed a step in the settings.
1. Edit CPU and enable Expose hardware assisted virtualization to the guest OS.
2. Change the memory to reserve all memory.
3. add the hypervisor.cpuid.v0 = FASLE to configuration.

Also does your GPU init with the pc starts then shows esxi loading then stops mid loading screen? If not you dont have the GPU set to primary in the bios. if you have CPU GPU you need to disable it and select PEG (pcie graphics) which will use your main video card. Not sure why this is needed but it wouldnt work for me with IGP enabled.

 

[Ch33rios]

if the usb device is a HID (keyboard or mouse) it wont pass through. You need to get a usb card either on the MB or PCIE and pass that into the vm with keyboard. if you need usb card this one works. Amazon.com: StarTech.com 2 Port PCI Express PCIe SuperSpeed USB 3.0 Controller Card with SATA Power - Dual Port PCI Express USB 3 Adapter PEXUSB3S23: Computers & Accessories make sure to get the eltron chipset.

As for 43, that means it still seeing it as a VM. so you might missed a step in the settings.
1. Edit CPU and enable Expose hardware assisted virtualization to the guest OS.
2. Change the memory to reserve all memory.
3. add the hypervisor.cpuid.v0 = FASLE to configuration.

Also does your GPU init with the pc starts then shows esxi loading then stops mid loading screen? If not you dont have the GPU set to primary in the bios. if you have CPU GPU you need to disable it and select PEG (pcie graphics) which will use your main video card. Not sure why this is needed but it wouldnt work for me with IGP enabled.

Ok that last part I definitely have screwed up. When I first went down this path I changed the primary display to the on-board graphics (changed away from PCIe)...perhaps this would explain some of the other strange things I've been experiencing. I'll change that for sure and try again. Having the USB keyboard attached shouldn't be a requirement to at least get the GPU passed through (I wouldn't think it would anyway...cant still do RDP or tightVNC as was previously mentioned).

And thanks for the add-on recommendation, one-day shipping FTW!

 

[Ch33rios]

Well after another hour or so of messing around and double/triple checking my setup and reinstalling Windows (win10 enterprise) from scratch, Im unhappy to report no further progress. Still Code 43

Tried EFI/BIOS
Tried NOT installing VM tools

I know the card is being passed to the VM because on boot of said VM it spins the GPU fan up like its under full load and then turning the VM off puts it back to a normal state of quietness (like its getting power but not doing anything). Most people seem to be successfully doing this on ESXi 6 where as I'm on 6.5....possibility?

Im going to take a break for tonight and recheck things again maybe tomorrow or Friday. I'll also bite the bullet and pull my current desktops 970 GTX out and stick that in there to see how IT fairs. Who knows but its worth a shot

 

[Rand__]

Not the same use case but when I tried passing my GTX1080 the only thing I did was to enable the passthrough and add the hypervisor line
hypervisor.cpuid.v0 = "FALSE" (correctly written and with quotes).
That passed the GPU. I then deactivated the SVGA card from vmware tools in hardware to make sure it wouldnt be in use.
No fiddling around in bios at all (primary display set to AST2400).

3dMark didnt run either (config check failed, it never got to run a 3d load) and when I couldnt connect via view i stopped messing around so am not sure whether it would work with a monitor.