Help: ESXI 6.5U1 passthrough GTX560ti successfully, but fail on...

Discussion in 'VMware, VirtualBox, Citrix' started by davidng, Mar 18, 2018.

  1. davidng

    davidng New Member

    Joined:
    Mar 4, 2018
    Messages:
    1
    Likes Received:
    2
    I am new to ESXI, recently I came across the following hardware
    Supermicro X10DRi + 32G*4 Samsung ECC RAM + E5-2650L V4*2

    I want this machine to be my server, gaming pc and workstation at the same time.
    So I add MSI GTX560 Ti+ USB3.0 PCIe + SATA3.0 PCIe + 4 port Lan PCIe + Sound Card
    I know GTX 560 is an old GPU, but I only got this graphic card.
    I have done my research, I install ESXI 6.5 U1 on a USB, and follow the below steps, it works.

    esxi host bios
    above 4G decoding = disabled (enable also ok)
    SR-IOV support = disabled

    ssh to vm host, and modify /etc/vmware/passthru.map to
    #NVIDIA ( it was 10de ffff bridge false )
    10de ffff link false

    Create new windows 10 64bit guest VM.
    Expose CPU (14 core * 1 socket)
    reserved all ram (16G ram)

    More guest VM Settings->VM Options->Advanced->Configuration.
    hypervisor.cpuid.v0 = FALSE
    pciHole.start = 1200
    pciHole.end = 2200
    pciPassthru0.msiEnabled = FALSE

    install windows 10 enterprise
    install vm tools
    shutdown guest VM

    passthrough GTX560 ti
    boot up the guest VM, install nVidia 391.01 driver.
    reboot again, both vm graphic card and GTX 560 ti are working fine, but sometimes the GTX560 will fail after some mouse move.
    I later found out, if I disable the vm graphic card in windows 10 device management, then everything works, the 3d gaming also working.

    Then I add more passthrough to this guest VM, like usb host, sound card.
    So this is my guest VM final Hardware Configuration
    CPU 14 vCPUs
    Memory 16 GB
    Hard disk 1 64 GB
    Hard disk 2 1,024 GB
    Network adapter 1 VM Network (Connected)
    Video card 4 MB
    CD/DVD drive 1 ATAPI
    PCI device 0 GF114 [GeForce GTX 560 Ti]
    PCI device 1 GF114 HDMI Audio Controller
    PCI device 2 Sound Core3D [Sound Blaster Recon3D / Z-Series]
    PCI device 3 VL805 USB 3.0 Host Controller
    PCI device 4 VL805 USB 3.0 Host Controller



    -= Problem =-
    When I shutdown my win 10 guest VM (shutdown inside win 10 or using esxi web interface to shut it down)
    then start the win 10 guest VM again in esxi web interface, it will try to power up, then it shutdown automatically.
    here is some of the important log (I think), I also attached the detail log.

    Code:
    2018-03-18T18:30:41.090Z| vcpu-0| I125: PCIPassthru: Attempted to program PCI cacheline size 32 not a power of 2 factor of original physical 64 for device 0000:03:00.0
    2018-03-18T18:30:41.099Z| vcpu-0| I125: PCIPassthru: Attempted to program PCI cacheline size 32 not a power of 2 factor of original physical 64 for device 0000:81:00.0
    2018-03-18T18:30:41.108Z| vcpu-0| I125: PCIPassthru: Attempted to program PCI cacheline size 32 not a power of 2 factor of original physical 64 for device 0000:02:00.0
    2018-03-18T18:30:41.108Z| vcpu-0| I125: PCIPassthru: Attempted to program PCI cacheline size 32 not a power of 2 factor of original physical 64 for device 0000:02:00.1
    2018-03-18T18:30:41.109Z| vcpu-0| I125: PCIPassthru: Attempted to program PCI cacheline size 32 not a power of 2 factor of original physical 64 for device 0000:82:00.0
    
    2018-03-18T18:30:41.287Z| vcpu-0| E105: PANIC: PCIPassthruChangeIntrSettings: 0000:81:00.0 failed to register interrupt (error code 195887105)
    2018-03-18T18:30:42.112Z| vcpu-0| W115: A core file is available in "/vmfs/volumes/5a9af3a8-eff1cb22-beac-ac1f6b450ede/Win10Enterprise/vmx-zdump.001"
    2018-03-18T18:30:42.119Z| vcpu-0| I125: Writing monitor file `vmmcores.gz`
    
    2018-03-18T18:30:53.062Z| vcpu-0| I125: SymBacktrace[12] 0000002c0966fea8 rip=0000000000000000
    2018-03-18T18:30:53.062Z| vcpu-0| I125: Msg_Post: Error
    2018-03-18T18:30:53.062Z| vcpu-0| I125: [msg.log.error.unrecoverable] VMware ESX unrecoverable error: (vcpu-0)
    2018-03-18T18:30:53.062Z| vcpu-0| I125+ PCIPassthruChangeIntrSettings: 0000:81:00.0 failed to register interrupt (error code 195887105)

    the strange thing is
    1. if I reboot the esxi host, the guest win 10 vm will boot fine.
    and if I shutdown the guest win 10 vm, same problem, can not start guest vm again unless I reboot the esxi host again.

    2. if I remove all PCI passthrough, and only passthrough the gtx 560 ti and its HDMI audio
    PCI device 0 GF114 [GeForce GTX 560 Ti]
    PCI device 1 GF114 HDMI Audio Controller
    the shutdown guest VM, can not start guest VM again issue resolved.
    so this is not the graphic card problem.

    3. so the issue seems to be in PCIe sound card or the PCIe usb (I try them individually, all fail, same issue)
    MB: Supermicro X10DRi has 2 onboard usb controller, one for usb 3.0 port*5, another one for usb 2.0*4
    I passthrough the usb 3.0 controller to win 10 guest VM,
    the shutdown guest VM, can not start again issue resolved.
    however, if I passthrough the usb 2.0 controller, it happen again, can not boot after shutdown.

    4. I later try add this to vm guest vm, no luck
    pciPassthru1.msiEnabled = FALSE
    pciPassthru2.msiEnabled = FALSE
    pciPassthru3.msiEnabled = FALSE
    pciPassthru4.msiEnabled = FALSE

    5. and also try ssh to vm host
    esxcli system module set -m=vmkusb -e=FALSE
    or
    esxcli system module set -m=vmkusb -e=TRUE
    problem persists.

    6. and also try vm host bios setting
    SR-IOV support= enable
    after the first guest VM shutdown, start again, this time, it stuck at the start up proces (no video output).
     

    Attached Files:

    #1
    Last edited: Mar 18, 2018
    mathiastro and Ub3r like this.
  2. Ub3r

    Ub3r New Member

    Joined:
    Jun 16, 2016
    Messages:
    4
    Likes Received:
    0
    Salutations, davidng!

    Did you ever figure this out? I just ran across your thread having already tried everything you've stated. I'm also having the USB 3.0 passthrough issue on a gaming VM :-(. I'm passing through a 1070 TI just fine and have a GRID K1 doing vGPU as well without issue on other VMs. When I add the passthrough PCIe USB 3.0 controller for keyboard/mouse I get screwed. I remove it and it boots fine. I've been reduced to using a super shitty program named Synergy which constantly freezes for 5-30seconds at a time to do keyboard/mouse sharing on the VM.

    For what it's worth, here's a link to the USB controller I'm using which does not work: https://www.amazon.com/gp/product/B07Z34BJYH/ref=ppx_yo_dt_b_asin_title_o03_s00?ie=UTF8&psc=1
     
    #2
Similar Threads: Help ESXI
Forum Title Date
VMware, VirtualBox, Citrix ESXi 6.7u3 home build went boom - need help troubleshooting Feb 9, 2020
VMware, VirtualBox, Citrix Help choosing m.2 NVMe enterprise drives for ESXi servers Nov 2, 2019
VMware, VirtualBox, Citrix Need networking help - ESXi 6.7 AIO / pfSense & napp-it Mar 29, 2019
VMware, VirtualBox, Citrix Can you help me with the vmware esxi 6.5? Oct 17, 2018
VMware, VirtualBox, Citrix ESXI Networking Help May 2, 2018

Share This Page