1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Troubleshooting GPU passthrough ESXi 6.5

Discussion in 'VMware, VirtualBox, Citrix' started by Ch33rios, Jan 2, 2017.

  1. helsyeah

    helsyeah Active Member

    Joined:
    Aug 22, 2015
    Messages:
    107
    Likes Received:
    30
    I'm just glad I have it working now. I had tried several non-quadro cards in the past and never got past the BSOD.

    RE msiEnabled, solid79 linked an nvidia post on it:

    VMware vDGA / GPU Passthrough Requires That MSI is Disabled on VMs | NVIDIA
     
    #101
  2. richtj99

    richtj99 New Member

    Joined:
    Jul 8, 2017
    Messages:
    9
    Likes Received:
    0
    Hi,

    I am trying to passthrough a Nvidia GTX 1060 card to a VM using Esxi 6.5. I am getting the Code 43 error.

    Hardware:
    Asus P8Z68V Pro & 32GB ram, I7 3770.

    VM OS - Windows 8.1

    I used the hypervisor.cpuid.v0 = "FALSE"
    The 16 gb for my workstation is reserved,
    CPU set to expose VM hardware to OS.

    I turned off the onboard intel GPU.

    I tried the pciPassthru0.msiEnabled= false Helsyeah suggested - still stuck on code 43.

    Any other suggestions on how to get it going?

    Thanks,
    Rich
     
    #102
  3. helsyeah

    helsyeah Active Member

    Joined:
    Aug 22, 2015
    Messages:
    107
    Likes Received:
    30
    Quick couple questions: does the motherboard/cpu support VT-d (I don't know off hand)? Is it enabled?
     
    #103
  4. richtj99

    richtj99 New Member

    Joined:
    Jul 8, 2017
    Messages:
    9
    Likes Received:
    0
    #104
  5. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    1,065
    Likes Received:
    111
    Just did the same procedure with a 1080 yesterday, worked fine.
    Had 43 first as well, turned out my config setting didn't take (v0=false), had to add that a second time (maybe after vmware tools install).
     
    #105
  6. helsyeah

    helsyeah Active Member

    Joined:
    Aug 22, 2015
    Messages:
    107
    Likes Received:
    30
    I have had the issue where v0=false did not take either. Last time I did it I added it directly via the vsphere web client and THAT seemed to work the first time.
     
    #106
  7. richtj99

    richtj99 New Member

    Joined:
    Jul 8, 2017
    Messages:
    9
    Likes Received:
    0
    I keep getting the error 43. I tried adding it via the web client. It shows up in the VMX file as well as the config edit within the web client. I also downloaded the VMX, edited it with notepad++, uploaded it & still nothing.

    I am going to try with windows 10 (not my favorite) but just to see if i can get it working. I am really confused.

    EDIT: Do you have the on board GPU disabled? Do I need a second card for ESXI to display out?
     
    #107
    Last edited: Jul 10, 2017
  8. helsyeah

    helsyeah Active Member

    Joined:
    Aug 22, 2015
    Messages:
    107
    Likes Received:
    30
    Have you tried manually setting pciHole? I know some have needed to in order to get it to work.

    In the VM vmx file set:

    pciHole.start = "1200"
    pciHole.end = "2200"

    Values may need tweaking beyond the above.
     
    #108
  9. richtj99

    richtj99 New Member

    Joined:
    Jul 8, 2017
    Messages:
    9
    Likes Received:
    0
    So to go through the whole process:

    1. Start new Win8 VM (Ram reserved), Install OS, install vmware tools, Shutdown.
    2. Added hypervisor.cpuid.v0 = false on W8 VM via VSphere VM Settings->VM Options->Advanced->Configuration.
    3. Passthrough two devices, Nvidia 1060 & 'audio' device
    4. Install latest Nvidia drivers
    5. Device shows but has code 43 error

    Other items:
    Expose CPU Option
    pciHole.start = "1200"
    pciHole.end = "2200"
    pciPassthru0.msiEnabled= false

    Helsyeah - you mentioned confirming " GTX 970 is pciPassthru0" - Should the VM PCI 'slot' match the VMX & the VM - so if the VMX says port 172 & the VM (within device manager) shows port 190 - does that matter?
     
    #109
  10. helsyeah

    helsyeah Active Member

    Joined:
    Aug 22, 2015
    Messages:
    107
    Likes Received:
    30
    I check the vmx file to ensure that the device ID on pciPassthru0 matched the device ID in ESXi.
     
    #110
  11. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    1,065
    Likes Received:
    111
    Have you done memory reservation and CPU config as indicated?
    I set GPU to PCIe explicitly
     
    #111
  12. richtj99

    richtj99 New Member

    Joined:
    Jul 8, 2017
    Messages:
    9
    Likes Received:
    0
    I did both the memory & CPU. GPU is turned off & set for PCIe.

    Are you using 6.5 or 6.0 ESXI?
     
    #112
  13. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    1,065
    Likes Received:
    111
    6.5, win 10 for this round
     
    #113
  14. richtj99

    richtj99 New Member

    Joined:
    Jul 8, 2017
    Messages:
    9
    Likes Received:
    0
    So I have been trying with a HD4350 (or something close to that number) & it keeps crashing when trying to install the drivers.

    I am now installing esxi 6.0 & will continue to try. Not sure where my issue is coming from.
     
    #114
  15. richtj99

    richtj99 New Member

    Joined:
    Jul 8, 2017
    Messages:
    9
    Likes Received:
    0
    So I am not not 100% sure that my Asus P8Z68-v Pro mobo is compatible. I am seeing varying reports on the Z68 chipset & VT-D.

    I tried a Dell T110 ii server I have sitting around & it passed the AMD card through with no issues. That mobo doesnt have the proper PCIe port on it so I cant try the Nvidia card.

    I may do a mobo shuffle with another box I have for a further test.
     
    #115
  16. calvinz360

    calvinz360 New Member

    Joined:
    Jul 12, 2017
    Messages:
    1
    Likes Received:
    0
    I am currently trying to figure out why my ubuntu guest (16.04 server x64) is not able to INIT the passthrough GPU (AMD R7 240)

    Before we begin, here's my host specs
    Intel i7-3770
    AsRock Fatal1ty Z68 Professional (VT and VT-d enabled)
    Sapphire AMD R7 240 (Oland), as primary display on host
    ESX 6.5 , 5310538

    Configuration of DirectPath I/O
    Capture.JPG

    Configuration on the guest hardware
    Capture2.JPG

    lspci output on the guest
    Code:
    00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (rev 01)
    00:01.0 PCI bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev 01)
    00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 08)
    00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
    00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
    00:07.7 System peripheral: VMware Virtual Machine Communication Interface (rev 10)
    00:0f.0 VGA compatible controller: VMware SVGA II Adapter
    00:10.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 01)
    00:11.0 PCI bridge: VMware PCI bridge (rev 02)
    00:15.0 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:15.1 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:15.2 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:15.3 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:15.4 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:15.5 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:15.6 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:15.7 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:16.0 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:16.1 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:16.2 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:16.3 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:16.4 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:16.5 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:16.6 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:16.7 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:17.0 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:17.1 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:17.2 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:17.3 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:17.4 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:17.5 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:17.6 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:17.7 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:18.0 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:18.1 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:18.2 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:18.3 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:18.4 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:18.5 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:18.6 PCI bridge: VMware PCI Express Root Port (rev 01)
    00:18.7 PCI bridge: VMware PCI Express Root Port (rev 01)
    02:01.0 SATA controller: VMware SATA AHCI controller
    03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240/340]
    03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
    13:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
    lspci -k output
    Code:
    03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240/340]
            DeviceName: pciPassthru0
            Subsystem: PC Partner Limited / Sapphire Technology Oland PRO [Radeon R7 240/340]
            Kernel modules: radeon
    03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
            Subsystem: PC Partner Limited / Sapphire Technology Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
            Kernel driver in use: snd_hda_intel
            Kernel modules: snd_hda_intel
    Some errors found on dmesg output, where the kernel was not able to initialize the GPU ?
    Code:
    [    2.906307] [drm] radeon kernel modesetting enabled.
    [    2.918981] radeon 0000:03:00.0: enabling device (0000 -> 0003)
    [    2.954560] radeon 0000:03:00.0: BAR 6: can't assign [??? 0x00000000 flags 0x20000000] (bogus alignment)
    [    2.954574] radeon 0000:03:00.0: BAR 6: can't assign [??? 0x00000000 flags 0x20000000] (bogus alignment)
    [    2.954610] [drm:radeon_get_bios [radeon]] *ERROR* Unable to locate a BIOS ROM
    [    2.954612] radeon 0000:03:00.0: Fatal error during GPU init
    [    2.954615] [drm] radeon: finishing device.
    [    2.962587] radeon: probe of 0000:03:00.0 failed with error -22
    
    The guest is already configured with 4vCores and 4GB of fully allocated vRAM. hypervisor.cpuid.v0 = "FALSE" was already set on the advanced configuration too. I have been going heads and tails with this for few days and still have no clue what could be the issue. On a side note, I was able to pass through the same hardware on a windows guest.
     
    #116
  17. richtj99

    richtj99 New Member

    Joined:
    Jul 8, 2017
    Messages:
    9
    Likes Received:
    0
    So I did a mobo shuffle, using a H77 chipset & the 3770 with both the HD6540 & GTX1060 - neither worked. I then did another shuffle to a supermicro box i have (xeon v5) & the HD6540 worked, the GTX1060 did not.

    I picked up a RX-560 & that seems to be working great - so I have video passthrough working.

    My only issue is in the passthrough VM, I am not able to change Windows8 screen orientation to my monitors. The option just doesnt exist.

    A google finds this: On Windows 7 and Windows 8, the Orientation option is missing under screen resolution (2091394) | VMware KB

    However disabling the VMware display doesnt fix the issue after several reboots of the VM (not host).

    I could try again with a fresh windows install & no vmware tools at all?
     
    #117
  18. speedyrazor

    speedyrazor New Member

    Joined:
    Jul 16, 2017
    Messages:
    1
    Likes Received:
    0
    Hi, I have been running the below setup for a couple of months now with no real issues, but all of a sudden when I add the USB infrared remote box to the VM, it crashes continuously with "video tdr failure" and reboots. If I un-attach the USB infrared remote box then All is good once again. Like I said, I have been using this setup for a while, then last night shutdown the host, rebooted and now got this issue, any ideas please?

    My Setup:
    Gigabyte Intel GA-Z270XP-SLI Motherboard
    Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
    16GB DDR4 Ram
    ESXi 6.5.0 (Build 5310538)
    Windows 10 VM
    Nvidia GT710 passthrough (hypervisor.cpuid.v0 = FALSE)
    USB eHome infrared Transciever

    I have tried other USB infrared Transceiver but had exactly the same issue.
    I am using it as a media centre with Kodi, so need to remote control.
    I don't know whats going on, any ideas please?
     
    #118
Similar Threads: Troubleshooting passthrough
Forum Title Date
VMware, VirtualBox, Citrix HP Z820, ESXi 6.5d, AMD GPU passthrough issue Yesterday at 9:27 AM
VMware, VirtualBox, Citrix RX480 passthrough on ESXi6.5 Mar 31, 2017
VMware, VirtualBox, Citrix How can I tell which ports belong to my motherboards 'passthrough-able' USB controller Mar 9, 2017
VMware, VirtualBox, Citrix ESXI 6.5 - Cant passthrough my GPU Jan 22, 2017
VMware, VirtualBox, Citrix ESXI 6(.5) - OSX with GPU Passthrough Jan 5, 2017

Share This Page