Troubleshooting GPU passthrough ESXi 6.5

[Ch33rios]

I think my perseverance has finally paid off!

The past few tests I had done were with actually with a different motherboard than I had originally tried. I had actually bought the MSI board previously linked just to try and see if it was in fact the motherboard.

Long story short, that was a mistake I switched back to my original Gigabyte MX31-BS0 board and decided to give it one more shot. Confirmed the configuration per the recommendations, added the GPU, GPU audio controller, and the PCIe USB controller. Then installed the drivers, and rebooted. After only 1 or 2 BSoDs, I switched the outputs and disabled the SVGA adapter....and sat there.

Sure enough, no flickering or any BSoDs anymore! Everything seemed stable! I rebooted a few times, installed Chrome and some other very basic apps to see how it responds and all was well. Then the real test....a game. Downloaded Overwatch and loaded it up and I proceeded to play the game for the next hour without a hitch and I was definitely getting within 5 FPS of my original dedicated desktop!

I've got some more sort of stress testing to do but damn am I pleased and I owe you a big thanks, @marcoi !

I'll report back probably tomorrow as I try a few more stress tests tonight.

 

[Eric Faden]

What was the end result of this? I am trying to pass through a GeForce 1050 Ti to Debian.

 

[vinay]

Nice you are close. I had the same issues.
First make sure vnc is working, then turn off the VM monitor using display properties. Select only show on monitor 2(whichever is attached to your Nvidia card.) Then go into device manager and disable the VM svga. It might take a few tried since the BSOD can happen randomly. If done you should not see any video in VM console.

Hi.. I am trying to do the this action but i am not sure how do i turn of VM monitor in the display settings. I am assuming your talking about windows display settings. Please help in exactly to do this setup..

 

[marcoi]

In the windows display select disable for the VM monitor. I'm not near a PC to include a screenshot.

 

[vinay]

In the windows display select disable for the VM monitor. I'm not near a PC to include a screenshot.

This is what is seen in windows display or am i looking at the wrong section ? Also the second screen is not detected as you can see in the screenshot.

 

[marcoi]

That's the area. I'll post more tomorrow when I am in front of the PC. You should also have both display listed. Also make sure to connect a monitor to the video card or it may not detect it correctly.

 

[vinay]

That's the area. I'll post more tomorrow when I am in front of the PC. You should also have both display listed. Also make sure to connect a monitor to the video card or it may not detect it correctly.

Thanks.. I will wait for your response.. Also i do have my monitor connected. Which is why i am not sure if this will work but i will try for few more days i think..

 

[marcoi]

@vinay - First thing is make sure setup of esxi and vm are good. The ati video card should be put into pass-through mode in esxi, you shouldnt need to edit any esxi files since amd doesnt lock their cards. You need to make sure to get all the devices used by your video card, usually two of them. Here is a sample of my nvidia card


You will need to reboot the server at this point to make sure the card is in pass-through.
Next follow these steps: You can refer to original post with pics on page one.

Nvidia a PITA but i got my 750 TI working after a lot of research and playing.
Next when building the VM. I did windows 10 pro. Build out the OS without adding any pci devices. you need to install esxi guest drivers and VNC, I used tightvnc. You should also download Nvidia drivers but dont run them yet. Power off the VM then edit the settings as follows:
1. Edit CPU and enable Expose hardware assisted virtualization to the guest OS.
2. Change the memory to reserve all memory.
View attachment 4098
3. Tab over to VM Options, then expand Advance and click on Edit Configuration
View attachment 4099
4. add parameter: hypervisor.cpuid.v0 with value of FALSE click on ok and save changes.
View attachment 4100
This part you might need to mess with and try it a few times
5. Start the VM back up and start the nvidia driver install. it will fail since you havent passed the card yet. but it will expand the files.
6. shutdown and add the PCI card and hdmi and usb passthought for keyboard to the settings then start it back up.
7. install the drivers. you might need to reboot, etc..
8. At this point, you should see the display change over to your vm display. If you do you can then disable the esxi vga card from device management. if you dont it will crash on you. if you need to work on the vm use VNC as the console will not work right.

you may also need to add to the vm vmx file the following if your vm has more then 2gb ram.
pciHole.start = "1200"
pciHole.end = "2200"

Also you need to make the video card primary display on the bios, else it doesnt seem to init right when you pass it to the VM. This means that as you boot esxi the screen will stop updating and look like it hung, but it should still be booting up in background. Once you power on your vm you monitor should show the vm display.

if that isnt happening then something is configured incorrectly.

hope this helps.

 

[vinay]

@vinay - First thing is make sure setup of esxi and vm are good. The ati video card should be put into pass-through mode in esxi, you shouldnt need to edit any esxi files since amd doesnt lock their cards. You need to make sure to get all the devices used by your video card, usually two of them. Here is a sample of my nvidia card
View attachment 5100

You will need to reboot the server at this point to make sure the card is in pass-through.
Next follow these steps: You can refer to original post with pics on page one.


you may also need to add to the vm vmx file the following if your vm has more then 2gb ram.
pciHole.start = "1200"
pciHole.end = "2200"

Also you need to make the video card primary display on the bios, else it doesnt seem to init right when you pass it to the VM. This means that as you boot esxi the screen will stop updating and look like it hung, but it should still be booting up in background. Once you power on your vm you monitor should show the vm display.

if that isnt happening then something is configured incorrectly.

hope this helps.

I had kept the iGPU as the primary and now changed that to GPU and also added the pciHole start and end to vmx file... but other than the fact that it reaches the point of booting up nothing happens after.. i.e., for some reason i cannot get past the installation of amd driver.. whenever amd driver installs reaches a point, say 70% , then the VM crashes with VIDEO TDR FAILURE..
I have not yet tried to passthrough any usb device yet.. thought if gpu works then i will work on the USB..

 

[marcoi]

Do you see the device in windows device manager prior to installing the drivers?

You may also want to run a driver clean program to complete remove the ATI drivers that maybe installed.

Also are you using vnc to remote access the windows 10 and not using esxi display?

 

[vinay]

Do you see the device in windows device manager prior to installing the drivers?

I see it as Microsoft basic display adapter ..

You may also want to run a driver clean program to complete remove the ATI drivers that maybe installed.

I usually revert to snapshot before adding the GPU ,when it fails...

Also are you using vnc to remote access the windows 10 and not using esxi display?

yes using vnC but on the esxi display also it is showing up..

 

[marcoi]

That is what I see, do you see two display adapters? As you can see i disabled the vmware display (which i dont think you need for amd cards)

 

[vinay]

View attachment 5105

That is what I see, do you see two display adapters? As you can see i disabled the vmware display (which i dont think you need for amd cards)

Before installing the driver it detects as Microsoft basic display adapter..

Before adding the GPU

After adding the GPU..


Now when I directly try to install the driver the VM crashes. There was once when i tried to update the driver through microsoft first and then install the driver which kinda worked but eventually VM slowed down and crashed..

After crash if the PC come up..

 

[marcoi]

Looks like it's not initialized due to yellow exclamation. Did you disable the onboard video card and set the amd card to be primary in bios?

You should notice on boot, esxi start on the monitor attached to amd card. Then it should look like esxi hangs. If that isn't happening then I don't think your card is initialized correctly and passthrough won't work.

 

[vinay]

Looks like it's not initialized due to yellow exclamation. Did you disable the onboard video card and set the amd card to be primary in bios?

You should notice on boot, esxi start on the monitor attached to amd card. Then it should look like esxi hangs. If that isn't happening then I don't think your card is initialized correctly and passthrough won't work.

I did.. and the appearance of the server hangs is seen..

But it think that exclamation mark is coming because the driver is not properly updated.. I remember when i had this on different hypervisor(not exactly, I used UNRAID) , even microsoft update to GPU works fine.. when i try it here the same exclamation mark appears and it crashes when i install using amd drivers..

 

[marcoi]

ok that is good. I wanted to confirm that behavior.
Did you add both amd card and amd audio devices to the vm?
Can you confirm the following on vm settings

 

[vinay]

Yes.. i added both the amd card and audio .. But i did not enable performance counters.. But I did enable expose hardware and reserve all guest memory..

 

[marcoi]

you can try to drop the memory down to 2 gb see if that resolves the issue with driver install. if it does, then you need to expand the pci hole size.
The hole need to be big enough for all the pci devices. so you might need to experiment and try numbers like 1200 - 4800 etc. I dont have an exact means of figuring it out but i read it somewhere that values might need to change base on vid memory.

pciHole.start = "1200"
pciHole.end = "2200"

 

[vinay]

I did try 2gb ram yesterday , but that also crashed.. will try experimenting with different values of the pcihole and see what happens..

 

[marcoi]

also did you try running the box with a different OS? IE try to install windows 10 on a separate hdd and see if you can install the drivers. maybe the port you have the video card is causing some issues?