Troubleshooting GPU passthrough ESXi 6.5

helsyeah

Active Member
Aug 22, 2015
111
30
28
39

richtj99

New Member
Jul 8, 2017
9
0
1
47
Hi,

I am trying to passthrough a Nvidia GTX 1060 card to a VM using Esxi 6.5. I am getting the Code 43 error.

Hardware:
Asus P8Z68V Pro & 32GB ram, I7 3770.

VM OS - Windows 8.1

I used the hypervisor.cpuid.v0 = "FALSE"
The 16 gb for my workstation is reserved,
CPU set to expose VM hardware to OS.

I turned off the onboard intel GPU.

I tried the pciPassthru0.msiEnabled= false Helsyeah suggested - still stuck on code 43.

Any other suggestions on how to get it going?

Thanks,
Rich
 

helsyeah

Active Member
Aug 22, 2015
111
30
28
39
Quick couple questions: does the motherboard/cpu support VT-d (I don't know off hand)? Is it enabled?
 

Rand__

Well-Known Member
Mar 6, 2014
4,610
918
113
Just did the same procedure with a 1080 yesterday, worked fine.
Had 43 first as well, turned out my config setting didn't take (v0=false), had to add that a second time (maybe after vmware tools install).
 

helsyeah

Active Member
Aug 22, 2015
111
30
28
39
Just did the same procedure with a 1080 yesterday, worked fine.
Had 43 first as well, turned out my config setting didn't take (v0=false), had to add that a second time (maybe after vmware tools install).
I have had the issue where v0=false did not take either. Last time I did it I added it directly via the vsphere web client and THAT seemed to work the first time.
 

richtj99

New Member
Jul 8, 2017
9
0
1
47
I keep getting the error 43. I tried adding it via the web client. It shows up in the VMX file as well as the config edit within the web client. I also downloaded the VMX, edited it with notepad++, uploaded it & still nothing.

I am going to try with windows 10 (not my favorite) but just to see if i can get it working. I am really confused.

EDIT: Do you have the on board GPU disabled? Do I need a second card for ESXI to display out?
 
Last edited:

helsyeah

Active Member
Aug 22, 2015
111
30
28
39
I keep getting the error 43. I tried adding it via the web client. It shows up in the VMX file as well as the config edit within the web client. I also downloaded the VMX, edited it with notepad++, uploaded it & still nothing.

I am going to try with windows 10 (not my favorite) but just to see if i can get it working. I am really confused.
Have you tried manually setting pciHole? I know some have needed to in order to get it to work.

In the VM vmx file set:

pciHole.start = "1200"
pciHole.end = "2200"

Values may need tweaking beyond the above.
 

richtj99

New Member
Jul 8, 2017
9
0
1
47
So to go through the whole process:

1. Start new Win8 VM (Ram reserved), Install OS, install vmware tools, Shutdown.
2. Added hypervisor.cpuid.v0 = false on W8 VM via VSphere VM Settings->VM Options->Advanced->Configuration.
3. Passthrough two devices, Nvidia 1060 & 'audio' device
4. Install latest Nvidia drivers
5. Device shows but has code 43 error

Other items:
Expose CPU Option
pciHole.start = "1200"
pciHole.end = "2200"
pciPassthru0.msiEnabled= false

Helsyeah - you mentioned confirming " GTX 970 is pciPassthru0" - Should the VM PCI 'slot' match the VMX & the VM - so if the VMX says port 172 & the VM (within device manager) shows port 190 - does that matter?
 

helsyeah

Active Member
Aug 22, 2015
111
30
28
39
Helsyeah - you mentioned confirming " GTX 970 is pciPassthru0" - Should the VM PCI 'slot' match the VMX & the VM - so if the VMX says port 172 & the VM (within device manager) shows port 190 - does that matter?
I check the vmx file to ensure that the device ID on pciPassthru0 matched the device ID in ESXi.
 

Rand__

Well-Known Member
Mar 6, 2014
4,610
918
113
Have you done memory reservation and CPU config as indicated?
I set GPU to PCIe explicitly
 

richtj99

New Member
Jul 8, 2017
9
0
1
47
So I have been trying with a HD4350 (or something close to that number) & it keeps crashing when trying to install the drivers.

I am now installing esxi 6.0 & will continue to try. Not sure where my issue is coming from.
 

richtj99

New Member
Jul 8, 2017
9
0
1
47
So I am not not 100% sure that my Asus P8Z68-v Pro mobo is compatible. I am seeing varying reports on the Z68 chipset & VT-D.

I tried a Dell T110 ii server I have sitting around & it passed the AMD card through with no issues. That mobo doesnt have the proper PCIe port on it so I cant try the Nvidia card.

I may do a mobo shuffle with another box I have for a further test.
 

calvinz360

New Member
Jul 12, 2017
8
0
1
34
I am currently trying to figure out why my ubuntu guest (16.04 server x64) is not able to INIT the passthrough GPU (AMD R7 240)

Before we begin, here's my host specs
Intel i7-3770
AsRock Fatal1ty Z68 Professional (VT and VT-d enabled)
Sapphire AMD R7 240 (Oland), as primary display on host
ESX 6.5 , 5310538

Configuration of DirectPath I/O
Capture.JPG

Configuration on the guest hardware
Capture2.JPG

lspci output on the guest
Code:
00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (rev 01)
00:01.0 PCI bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev 01)
00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 08)
00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
00:07.7 System peripheral: VMware Virtual Machine Communication Interface (rev 10)
00:0f.0 VGA compatible controller: VMware SVGA II Adapter
00:10.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 01)
00:11.0 PCI bridge: VMware PCI bridge (rev 02)
00:15.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.7 PCI bridge: VMware PCI Express Root Port (rev 01)
02:01.0 SATA controller: VMware SATA AHCI controller
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240/340]
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
13:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
lspci -k output
Code:
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240/340]
        DeviceName: pciPassthru0
        Subsystem: PC Partner Limited / Sapphire Technology Oland PRO [Radeon R7 240/340]
        Kernel modules: radeon
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
        Subsystem: PC Partner Limited / Sapphire Technology Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
Some errors found on dmesg output, where the kernel was not able to initialize the GPU ?
Code:
[    2.906307] [drm] radeon kernel modesetting enabled.
[    2.918981] radeon 0000:03:00.0: enabling device (0000 -> 0003)
[    2.954560] radeon 0000:03:00.0: BAR 6: can't assign [??? 0x00000000 flags 0x20000000] (bogus alignment)
[    2.954574] radeon 0000:03:00.0: BAR 6: can't assign [??? 0x00000000 flags 0x20000000] (bogus alignment)
[    2.954610] [drm:radeon_get_bios [radeon]] *ERROR* Unable to locate a BIOS ROM
[    2.954612] radeon 0000:03:00.0: Fatal error during GPU init
[    2.954615] [drm] radeon: finishing device.
[    2.962587] radeon: probe of 0000:03:00.0 failed with error -22
The guest is already configured with 4vCores and 4GB of fully allocated vRAM. hypervisor.cpuid.v0 = "FALSE" was already set on the advanced configuration too. I have been going heads and tails with this for few days and still have no clue what could be the issue. On a side note, I was able to pass through the same hardware on a windows guest.
 

richtj99

New Member
Jul 8, 2017
9
0
1
47
So I did a mobo shuffle, using a H77 chipset & the 3770 with both the HD6540 & GTX1060 - neither worked. I then did another shuffle to a supermicro box i have (xeon v5) & the HD6540 worked, the GTX1060 did not.

I picked up a RX-560 & that seems to be working great - so I have video passthrough working.

My only issue is in the passthrough VM, I am not able to change Windows8 screen orientation to my monitors. The option just doesnt exist.

A google finds this: On Windows 7 and Windows 8, the Orientation option is missing under screen resolution (2091394) | VMware KB

However disabling the VMware display doesnt fix the issue after several reboots of the VM (not host).

I could try again with a fresh windows install & no vmware tools at all?
 

speedyrazor

New Member
Jul 16, 2017
1
0
1
51
Hi, I have been running the below setup for a couple of months now with no real issues, but all of a sudden when I add the USB infrared remote box to the VM, it crashes continuously with "video tdr failure" and reboots. If I un-attach the USB infrared remote box then All is good once again. Like I said, I have been using this setup for a while, then last night shutdown the host, rebooted and now got this issue, any ideas please?

My Setup:
Gigabyte Intel GA-Z270XP-SLI Motherboard
Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
16GB DDR4 Ram
ESXi 6.5.0 (Build 5310538)
Windows 10 VM
Nvidia GT710 passthrough (hypervisor.cpuid.v0 = FALSE)
USB eHome infrared Transciever

I have tried other USB infrared Transceiver but had exactly the same issue.
I am using it as a media centre with Kodi, so need to remote control.
I don't know whats going on, any ideas please?
 

roswellian

Member
Oct 18, 2013
74
8
8
[ 2.954610] [drm:radeon_get_bios [radeon]] *ERROR* Unable to locate a BIOS ROM
This error message basically tells everything. You need to load the BIOS ROM of the graphics card. Unfortunatelly current linux kernel is unable to do that, but you can hack it and recompile the kernel as this links suggested: VMDirectPath and ATI Radeon |VMware Communities

I have been doing this with my 7570 and it works pretty well.
 

calvinz360

New Member
Jul 12, 2017
8
0
1
34
This error message basically tells everything. You need to load the BIOS ROM of the graphics card. Unfortunatelly current linux kernel is unable to do that, but you can hack it and recompile the kernel as this links suggested: VMDirectPath and ATI Radeon |VMware Communities

I have been doing this with my 7570 and it works pretty well.
Is your setup running on 16.04 ? or it needs something older since the thread is backdated till 2010-13