Troubleshooting GPU passthrough ESXi 6.5

Serj_82

New Member
Aug 11, 2020
5
0
1
For 1 - see if you can move the cards to different pcie slots, you might be drawing too much power or hitting a pcie lane limit. You will need to experiment since it isnt an exact science.
For 2 - nvidia drivers are super picky and it might take a lot of trial and error to get them working. Also you may need to monitor or dummy plug setup on the video card. The other thing i recall is i had to remove the virtual video driver as nvidia drivers looked for that as a clue to running virtual machine. there is alot of info in the threads here so you may need to spend some time reading the threads and looking for what worked for various people and trying it out.

good luck.
Unfortunately, the motherboard has only one riser-board slot.
And on the riser there are only three PСI-E ports (two short and one long). The long one takes up the video card and blocks access to the first short slot. And in the second short slot - a wi-fi card is installed.
 

marcoi

Well-Known Member
Apr 6, 2013
1,404
225
63
Gotha Florida
Unfortunately, the motherboard has only one riser-board slot.
And on the riser there are only three PСI-E ports (two short and one long). The long one takes up the video card and blocks access to the first short slot. And in the second short slot - a wi-fi card is installed.
Have you tried using another card in the third slot to see if the same thing happens?
 

chrissiemoore1

New Member
Aug 19, 2020
1
1
3
spent two days trying to get this to work..went thru variations of blue screen with video error and booted but disabled in device manager.

in the end.. i made a brand new VM
i have esxi 6.7.. however when i made the VM i chose 6.0 version for that win10 vm this time

added BOTH the Nvidia geforce card AND its SOUND portion .. so TWO pci additional pieces.
installed windows.
updated windows
shutdown vm

added JUST this one parameter.
hypervisor.cpuid.v0
to false

restarted.... installed the video card fine. drivers works. everyone happy.

my specs:

amd ryzen 5
geforce 1050TI
64GB ram


i made this account just to post this and help other poor souls

EDIT: ill add that after booting up after adding the param i gotblack screen in esxi console.
so before I rebooted I installed VNC server. I do that on all my machines so.
 
Last edited:
  • Like
Reactions: epicurean

brentk

New Member
Aug 9, 2020
1
0
1
I have spent the last two weeks trying to get GPU passthrough working with an RTX 2060 Super GPU / Gigabyte Aourus Z390 MB / Intel i9 system. Running ESXi 7.0 and Windows 10 Pro guest. Everything works fine until I reboot the guest, then I get the Code 43 Nvidia driver error until I reboot the ESXi host (unless I disable the Nvidia device before reboot).
I have tried all the settings I found in this thread with no luck (Edited passthru.map, set hypervisor.cpuid.v0 =FALSE, set pciPassthru{0/1/3}.msiEnabled = FALSE, tried EFI & BIOS, even tried patching the Nvidia drivers). I've also tried multiple different BIOS settings.
I have also tried unsuccessfully to use earlier versions of ESXi, but each version I tried had different problems which stopped me from getting to the point of running the VM with passthrough (missing network drivers on ESXi 6.0 / ESXi hanging on startup if any PCI devices are passed through With ESXi 6.7.0U3B etc.)

I have managed to successfully get Passthrough to work on this machine using a KVM setup with UNRAID insteqad of ESXi - but I'd way prefer to use ESXi as it has much better snapshot features.

Is anybody able to please provide any advice on how to get rid of the Code 43 error?
 

hmw

Active Member
Apr 29, 2019
261
83
28
I have spent the last two weeks trying to get GPU passthrough working with an RTX 2060 Super GPU / Gigabyte Aourus Z390 MB / Intel i9 system. Running ESXi 7.0 and Windows 10 Pro guest. Everything works fine until I reboot the guest, then I get the Code 43 Nvidia driver error until I reboot the ESXi host (unless I disable the Nvidia device before reboot).
I have tried all the settings I found in this thread with no luck (Edited passthru.map, set hypervisor.cpuid.v0 =FALSE, set pciPassthru{0/1/3}.msiEnabled = FALSE, tried EFI & BIOS, even tried patching the Nvidia drivers). I've also tried multiple different BIOS settings.
I have also tried unsuccessfully to use earlier versions of ESXi, but each version I tried had different problems which stopped me from getting to the point of running the VM with passthrough (missing network drivers on ESXi 6.0 / ESXi hanging on startup if any PCI devices are passed through With ESXi 6.7.0U3B etc.)

I have managed to successfully get Passthrough to work on this machine using a KVM setup with UNRAID insteqad of ESXi - but I'd way prefer to use ESXi as it has much better snapshot features.

Is anybody able to please provide any advice on how to get rid of the Code 43 error?
The GPU is actually multiple devices and these are all passed thru - e.g. one might be the GPU itself and the other might be the HDMI audio controller (dev_id 10f0 is the HDMI Audio device) that is part of the GPU. In the virtual machine's VMX file Set pciPassthruX.msiEnabled=FALSE for every device associated with the GPU except the GPU itself.

In the passthru.map - comment out the default NVIDIA entry (It will '10de ffff bridge false'') and put in

Code:
# <ven_id> <dev_id> <reset_method> <setting>
10de  10f0  d3d0  false
Also - add the following to the VMX file

Code:
pciPassthru.use64bitMMIO="TRUE"
 

das1996

Member
Sep 4, 2018
37
2
8
Any new progress on nvidia passthrough with esxi 7 (update 1)?

I can get passthrough working with a 1660 ti just fine on esxi 6.5 (latest update) with just gpu vid/pid d3d0 false (no audio/usb lines, just the single one for the gpu) the in the passthru.map and hypervisor.cpuid.v0 false in the vmx. This survives reboots without any error 43's.

This doesn't carry over into 6.7 or 7. After host reboot it works fine, but can't reboot vm without getting error 43. I tried many variations in the passthru.map and vmx file. I think this still comes back to the reset pcie resets.

This thread - Deep investigation on GPU Passthrough not worki... |VMware Communities - points out that with 6.7 and newer the reset for each passed through device is initiated at the same time where as in 6.5, each is 4 seconds apart. Maybe this is a none issue and is irrelevant, but for whatever reason passthrough fails in 7.0 update 1 with the 456.71 driver.

Suggestions?
 

hmw

Active Member
Apr 29, 2019
261
83
28
Any new progress on nvidia passthrough with esxi 7 (update 1)?

I can get passthrough working with a 1660 ti just fine on esxi 6.5 (latest update) with just gpu vid/pid d3d0 false (no audio/usb lines, just the single one for the gpu) the in the passthru.map and hypervisor.cpuid.v0 false in the vmx. This survives reboots without any error 43's.

This doesn't carry over into 6.7 or 7. After host reboot it works fine, but can't reboot vm without getting error 43. I tried many variations in the passthru.map and vmx file. I think this still comes back to the reset pcie resets.

This thread - Deep investigation on GPU Passthrough not worki... |VMware Communities - points out that with 6.7 and newer the reset for each passed through device is initiated at the same time where as in 6.5, each is 4 seconds apart. Maybe this is a none issue and is irrelevant, but for whatever reason passthrough fails in 7.0 update 1 with the 456.71 driver.

Suggestions?
I have the passthrough working fine with a GTX1080 and ESXi 6.7, 7.0 and 7U1
 

hmw

Active Member
Apr 29, 2019
261
83
28
Can you post the contents of your passthru.map and *.vmx files? Which version of the driver are you using?

win10.vmx

Code:
    pciPassthru0.id = "00000:033:00.0"
    pciPassthru0.deviceId = "0x1b80"
    pciPassthru0.vendorId = "0x10de"
    pciPassthru0.present = "TRUE"
    ....
    pciPassthru1.id = "00000:033:00.1"
    pciPassthru1.deviceId = "0x10f0"
    pciPassthru1.vendorId = "0x10de"
    pciPassthru1.present = "TRUE"
    pciPassthru1.msiEnabled = "FALSE"
    ....
    pciPassthru0.pciSlotNumber = "192"
    pciPassthru1.pciSlotNumber = "224"
    hypervisor.cpuid.v0 = "FALSE"
    pciPassthru.use64bitMMIO = "TRUE"
passthru.map

Code:
# Intel 82579LM Gig NIC can be reset with d3d0
...

# LSILogic 1068 based SAS controllers
1000  0056  d3d0     default
1000  0058  d3d0     default
# NVIDIA
# 10de  ffff  bridge   false
# added
10de  10f0  d3d0  false
BIOS has ACS and ARI enabled along with AMD IOMMU
 

das1996

Member
Sep 4, 2018
37
2
8
^^10de:10f0 that's an audio controller right?

What happened if you set to the id of the gpu itself?
 

hmw

Active Member
Apr 29, 2019
261
83
28
^^10de:10f0 that's an audio controller right?

What happened if you set to the id of the gpu itself?
Why would you do that? You're commenting out the bridge reset method for Nvidia and also telling ESXi not to use anything BUT PCI power functions for resetting (d3=off d0=on) the audio controller. The reason is that the GPU itself responds well to FLR and bus resets but the other PCIe components (HDMI Audio, USB etc) don't like that reset method. Hence you use D3D0 on everything except the GPU
 

das1996

Member
Sep 4, 2018
37
2
8
Ok. I've duplicated your settings to a T, including just passing the gpu and audio to the vm, and setting the msiEnabled to false for the audio adapter only.

hypervisor.cpuid.v0= false has been set all along. Still getting the error 43 after a guest reboot.

I've confirmed several times the correct pid for the nvidia audio (lspci -v) and set accordingly in passthru.map.

The card in question is a 1660 ti, but really shouldn't it all be the same?

What version driver and windows are you using? 456.71 and 1803 here. Same behavior with 2004.
 
Last edited:

hmw

Active Member
Apr 29, 2019
261
83
28
Ok. I've duplicated your settings to a T, including just passing the gpu and audio to the vm, and setting the msiEnabled to false for the audio adapter only.

hypervisor.cpuid.v0= false has been set all along. Still getting the error 43 after a guest reboot.

I've confirmed several times the correct pid for the nvidia audio (lspci -v) and set accordingly in passthru.map.

The card in question is a 1660 ti, but really shouldn't it all be the same?

What version driver and windows are you using? 456.71 and 1803 here.
Windows 10 2004 and 446.14 driver


1602992529667.png


I am using Moonlight/VNC to access the screen and there's a HDMI Dummy plugged into the GTX1080 HDMI input.
 

das1996

Member
Sep 4, 2018
37
2
8
I appreciate all your help. Still coming up short. Trying to install your version of the driver.

motherboard - asus h170 pro gaming
cpu - i5 6600 (6500?)

Gpu's hdmi is connected to a real monitor. Remote access is through vnc/rdp.

It crazy how inconsistent this is. In 6.5, to get passthrough working with this card, I just had to pass the vid/pid of the gpu itself and the hypervisor parameter in the vmx. Nothing else.
 

hmw

Active Member
Apr 29, 2019
261
83
28
I appreciate all your help. Still coming up short. Trying to install your version of the driver.

motherboard - asus h170 pro gaming
cpu - i5 6600 (6500?)

Gpu's hdmi is connected to a real monitor. Remote access is through vnc/rdp.

It crazy how inconsistent this is. In 6.5, to get passthrough working with this card, I just had to pass the vid/pid of the gpu itself and the hypervisor parameter in the vmx. Nothing else.
I got errors when I used RDP, had to switch to either VNC or Moonlight
 

das1996

Member
Sep 4, 2018
37
2
8
Which build of esxi 7 are you on? Update 1 or earlier? I've been trying with update 1. Maybe need to revert to something earlier.
 

hmw

Active Member
Apr 29, 2019
261
83
28
I have this working on 6.7U3 and 7.0b. I've also upgraded last week to 7U1

What's your virtualHW.version ? Mine is set to 14

Code:
guestOS = "windows9-64"
virtualHW.version = "14"
vmci0.present = "TRUE"
Also - what does lspci return for the Nvidia card? How many devices are on the graphics card? Just the HDMI Audio?
 

das1996

Member
Sep 4, 2018
37
2
8
virtualHW = 18

lspci -v
0000:01:00.0 Display controller VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1660 Ti]
Class 0300: 10de:2182

0000:01:00.1 Multimedia controller Audio device: NVIDIA Corporation TU116 High Definition Audio Controller
Class 0403: 10de:1aeb

0000:01:00.2 Serial bus controller USB controller: NVIDIA Corporation Device 1aec
Class 0c03: 10de:1aec

0000:01:00.3 Serial bus controller : NVIDIA Corporation Device 1aed
Class 0c80: 10de:1aed

Total of 4. I was initially passing all 4. Using your methodology I tried passing just the gpu and audio.
 

hmw

Active Member
Apr 29, 2019
261
83
28
virtualHW = 18

lspci -v
0000:01:00.0 Display controller VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1660 Ti]
Class 0300: 10de:2182

0000:01:00.1 Multimedia controller Audio device: NVIDIA Corporation TU116 High Definition Audio Controller
Class 0403: 10de:1aeb

0000:01:00.2 Serial bus controller USB controller: NVIDIA Corporation Device 1aec
Class 0c03: 10de:1aec

0000:01:00.3 Serial bus controller : NVIDIA Corporation Device 1aed
Class 0c80: 10de:1aed

Total of 4. I was initially passing all 4. Using your methodology I tried passing just the gpu and audio.


Shouldn't it be something like:

Code:
passthru.map

# <ven_id> <dev_id> <reset_method> <setting>
10de  1aeb  d3d0  false
10de  1aec  d3d0  false
10de  1aed  d3d0  false
Code:
vmx file

    pciPassthru0.id = "00000:01:00.0"
    pciPassthru0.deviceId = "0x2182"
    pciPassthru0.vendorId = "0x10de"
    pciPassthru0.present = "TRUE"
  ...
    pciPassthru1.id = "00000:01:00.1"
    pciPassthru1.deviceId = "0x1aeb"
    pciPassthru1.vendorId = "0x10de"
    pciPassthru1.present = "TRUE"
    pciPassthru1.msiEnabled = "FALSE"
...
    pciPassthru2.id = "00000:01:00.1"
    pciPassthru2.deviceId = "0x1aec"
    pciPassthru2.vendorId = "0x10de"
    pciPassthru2.present = "TRUE"
    pciPassthru2.msiEnabled = "FALSE"
...
    pciPassthru3.id = "00000:01:00.2"
    pciPassthru3.deviceId = "0x1aed"
    pciPassthru3.vendorId = "0x10de"
    pciPassthru3.present = "TRUE"
    pciPassthru3.msiEnabled = "FALSE"
i.e everything except GPU has D3D0 reset - and everything except GPU has pciPassthruN.msiEnabled set to false ...

You might have to enable message signaled interrupts (MSI) for the Nvidia USB controller if it is flaky. And you *MUST* set the Nvidia GPU as the primary GPU - and not the VMware SVGA gpu