1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

GPU passthrough issues

Discussion in 'Processors and Motherboards' started by outsider, Jan 21, 2017.

  1. outsider

    outsider New Member

    Joined:
    Jan 21, 2017
    Messages:
    4
    Likes Received:
    0
    I'm no stranger when it comes to building computers (been at it for over 20 years) but this one problem stumped me.

    [back story]
    Last summer I bought a pair of E5-2670 (since they were so cheap) from ebay, and also picked up a ASRock EP2C602 Mobo to consolidate my file server/ESXi server/router/and a another test machine.
    I decided to run everything under ESXi 6.0 and everything installed and transitioned just fine into one large case with plenty of drive bays, and expansion.

    I then decided to virtualize a GPU so I can run an instance of Kodi to run my projector.
    I had no end of troubles with GPU passthrough on ESXi 6. Lots of PSOD (purple screen of death) where the ESXi os would hang, and take down all the current running VMs.
    I then downgraded to ESXi 5.5 thinking that maybe the latest ESXi version wasn't supported, but the troubles followed. As soon as I passed a GPU, upon first boot of the VM with the GPU, the server would freeze. However sometimes the system ran well with a GPU passed through for either a few hours, or a few days, but the end result was always a PSOD, and all the VMs would crash.
    With no VM using the GPU, ESXi was happy as punch. No problems whatsoever.

    I lived without GPU passthrough until last fall (didn't run the VM with the GPU), but really wanted to play with that feature (to virtualize a few machines that needed GPUs).
    Last December I installed UnRaid since it had a good following and passing GPUs to VMs seemed trivial.
    Unfortunately, I tried everything I could think of to get the VM to run reliably with a GPU under UnRaid, but had no success.
    [\back story]

    So here we are today.
    I have moved all the VMs to separate machines and the E5-2670 machine is not being fully utilized at all at the moment due to frustration.
    In the ESXi days, I was trying to pass through a Quadro 4000 which is on the approved list for ESXi 5.5. With UnRaid I also tried a GTX 560 Ti

    When I purchased the E5-2670 CPUs I made sure to grab the SR0KX versions (and not the SR0H8 since they have a IOMMU bug)
    I have gone through all the BIOS settings, and turned on all the virtualization and IOMMU settings availabe

    At this point I'm trying to think of ways to diagnose my problem for the least amount of wasted money:
    1. Do I buy a couple more SR0KX CPUs and try them out?
    2. Do I buy some 2011-v2 CPUs instead?
    3. Buy another motherboard? (this was the most expensive part of the initial upgrade)

    Any thoughts or advice?
     
    #1
    Last edited: Jan 21, 2017
  2. Netwerkz101

    Netwerkz101 Active Member

    Joined:
    Dec 27, 2015
    Messages:
    101
    Likes Received:
    28
    No help.. just suggest you include make/model of GPU you are attempting to pass through.
     
    #2
  3. outsider

    outsider New Member

    Joined:
    Jan 21, 2017
    Messages:
    4
    Likes Received:
    0
    Goodpoint.
    In the ESXi days, I was trying to pass through a Quadro 4000 which is on the approved list for ESXi 5.5. With UnRaid I also tried a GTX 560 Ti
     
    #3
  4. realtomatoes

    realtomatoes Member

    Joined:
    Oct 3, 2016
    Messages:
    30
    Likes Received:
    0
    #4
  5. herby

    herby Member

    Joined:
    Aug 18, 2013
    Messages:
    66
    Likes Received:
    8
    I don't know much of anything about ESXi, but I do know consumer Nvidia cards are gimped to make passing them through difficult.
    I passthough a couple of AMD GPUs and USB card in Xenserver 6.5 without too much trouble; although 7.0 seems to have issues with some PCI devices. It may not be stable enough for production but good enough for me screwing around at home running my desktops, and a few servers.
     
    #5
  6. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    1,608
    Likes Received:
    225
    For what it worth I hear gpu pass through got better in esxi 6
     
    #6
  7. epicurean

    epicurean Member

    Joined:
    Sep 29, 2014
    Messages:
    277
    Likes Received:
    6
    In my limited experience, I found that esxi 5.5 was more forgiving for GPU passthrough, even with AMD GPUs. Since esxi 6.0u1 onwards, the HDMI audio component was always tricky at best.
     
    #7
    gigatexal likes this.
  8. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    1,608
    Likes Received:
    225
    That's unfortunate even with a hardware double that fakes connectivity to a monitor?
     
    #8
  9. herby

    herby Member

    Joined:
    Aug 18, 2013
    Messages:
    66
    Likes Received:
    8
    That's interesting. It can be a bit of an issue in Xenserver, I believe it has something to with the video cards being presented as two devices:
    Code:
    01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950]
    01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] 
    I've noticed on the first boot of the VM after starting the Xenserver the audio device isn't presented with the GPU but it does appear after rebooting the VM. If it was explicitly passed though I suspect it would behave better.
     
    #9
  10. outsider

    outsider New Member

    Joined:
    Jan 21, 2017
    Messages:
    4
    Likes Received:
    0
    UPDATE:
    I borrowed an old AMD HD6870 GPU from a friend and tossed it in the system.
    Booted up and did a quick install of UnRaid, and imagine my surprise when the GPU passed without any hassle to a VM.
    Next test is to verify stability, and to maybe try ESXi.

    My love for nvidia is starting to waiver....
     
    #10
  11. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    1,608
    Likes Received:
    225
    so you got this working in Xen?
     
    #11
  12. outsider

    outsider New Member

    Joined:
    Jan 21, 2017
    Messages:
    4
    Likes Received:
    0
    Yes.
    UnRaid's implementation of Xen.
     
    #12
  13. epicurean

    epicurean Member

    Joined:
    Sep 29, 2014
    Messages:
    277
    Likes Received:
    6
    Sorry I missed your post. Answer is yes.

    Sent from my SM-G925F using Tapatalk
     
    #13
  14. TLN

    TLN Member

    Joined:
    Feb 26, 2016
    Messages:
    126
    Likes Received:
    10
    I got a Radeon cards and pased it through with no issues. Make sure you edit vm.pchole setings in vmx file, and choose to allocate all the memory for VM.
     
    #14
  15. helsyeah

    helsyeah Member

    Joined:
    Aug 22, 2015
    Messages:
    97
    Likes Received:
    27
    As a data point (not the same CPUs or Mobo), I have an AMD RX480 on pass-through on ESXI 6.0U1 and its been working well since Oct/Nov 2016 running on a Supermicro X10SRL (UP) + E5-2630 V4 ES2 + 64 GB RAM.

    I also had a Quadro 4000 (actually a hard modded GTX 660) that worked fine as well on ESXI 6.0 (no update).
     
    #15
  16. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    1,608
    Likes Received:
    225
    kudos to AMD for being open to letting us hack with them, or being so small / poor not to have the people to lock down their hw to these kinds of use-cases
     
    #16
    marcoi likes this.
  17. nk215

    nk215 Active Member

    Joined:
    Oct 6, 2015
    Messages:
    174
    Likes Received:
    38
    This doesnt make any sense. Switching a few resistors cant change a kepler to a fermi.
     
    #17
  18. helsyeah

    helsyeah Member

    Joined:
    Aug 22, 2015
    Messages:
    97
    Likes Received:
    27
    #18
  19. zer0sum

    zer0sum Member

    Joined:
    Mar 8, 2013
    Messages:
    60
    Likes Received:
    10
    Did you have to do anything special to get the RX480 working?
    I'm trying with a Gigabyte GA7-PESH2, RX480, ESXi 6.5.0-4564106 and when I boot the VM it hangs the whole system :(
     
    #19
  20. helsyeah

    helsyeah Member

    Joined:
    Aug 22, 2015
    Messages:
    97
    Likes Received:
    27
    Not that I am aware of. As I recall all I did was simply setup passthrough and add the 480 to the VM. Its a windows 10 VM if that matters.
     
    #20
Similar Threads: passthrough issues
Forum Title Date
Processors and Motherboards Intel iGPU passthrough possible? Jan 24, 2017
Processors and Motherboards GPU for passthrough Jan 12, 2017
Processors and Motherboards Intel S2600CP dual GPU passthrough issue Apr 2, 2016
Processors and Motherboards Supermicro C202 mainboard and SATA passthrough. Jan 21, 2013
Processors and Motherboards X10SDV-4C+-TLN4F-O issues with 5V Dual Dec 11, 2016

Share This Page