GPU passthrough issues

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

outsider

New Member
Jan 21, 2017
17
3
3
48
I'm no stranger when it comes to building computers (been at it for over 20 years) but this one problem stumped me.

[back story]
Last summer I bought a pair of E5-2670 (since they were so cheap) from ebay, and also picked up a ASRock EP2C602 Mobo to consolidate my file server/ESXi server/router/and a another test machine.
I decided to run everything under ESXi 6.0 and everything installed and transitioned just fine into one large case with plenty of drive bays, and expansion.

I then decided to virtualize a GPU so I can run an instance of Kodi to run my projector.
I had no end of troubles with GPU passthrough on ESXi 6. Lots of PSOD (purple screen of death) where the ESXi os would hang, and take down all the current running VMs.
I then downgraded to ESXi 5.5 thinking that maybe the latest ESXi version wasn't supported, but the troubles followed. As soon as I passed a GPU, upon first boot of the VM with the GPU, the server would freeze. However sometimes the system ran well with a GPU passed through for either a few hours, or a few days, but the end result was always a PSOD, and all the VMs would crash.
With no VM using the GPU, ESXi was happy as punch. No problems whatsoever.

I lived without GPU passthrough until last fall (didn't run the VM with the GPU), but really wanted to play with that feature (to virtualize a few machines that needed GPUs).
Last December I installed UnRaid since it had a good following and passing GPUs to VMs seemed trivial.
Unfortunately, I tried everything I could think of to get the VM to run reliably with a GPU under UnRaid, but had no success.
[\back story]

So here we are today.
I have moved all the VMs to separate machines and the E5-2670 machine is not being fully utilized at all at the moment due to frustration.
In the ESXi days, I was trying to pass through a Quadro 4000 which is on the approved list for ESXi 5.5. With UnRaid I also tried a GTX 560 Ti

When I purchased the E5-2670 CPUs I made sure to grab the SR0KX versions (and not the SR0H8 since they have a IOMMU bug)
I have gone through all the BIOS settings, and turned on all the virtualization and IOMMU settings availabe

At this point I'm trying to think of ways to diagnose my problem for the least amount of wasted money:
1. Do I buy a couple more SR0KX CPUs and try them out?
2. Do I buy some 2011-v2 CPUs instead?
3. Buy another motherboard? (this was the most expensive part of the initial upgrade)

Any thoughts or advice?
 
Last edited:

outsider

New Member
Jan 21, 2017
17
3
3
48
Goodpoint.
In the ESXi days, I was trying to pass through a Quadro 4000 which is on the approved list for ESXi 5.5. With UnRaid I also tried a GTX 560 Ti
 

herby

Active Member
Aug 18, 2013
187
53
28
I don't know much of anything about ESXi, but I do know consumer Nvidia cards are gimped to make passing them through difficult.
I passthough a couple of AMD GPUs and USB card in Xenserver 6.5 without too much trouble; although 7.0 seems to have issues with some PCI devices. It may not be stable enough for production but good enough for me screwing around at home running my desktops, and a few servers.
 

epicurean

Active Member
Sep 29, 2014
785
80
28
For what it worth I hear gpu pass through got better in esxi 6
In my limited experience, I found that esxi 5.5 was more forgiving for GPU passthrough, even with AMD GPUs. Since esxi 6.0u1 onwards, the HDMI audio component was always tricky at best.
 
  • Like
Reactions: gigatexal

herby

Active Member
Aug 18, 2013
187
53
28
the HDMI audio component was always tricky at best.
That's interesting. It can be a bit of an issue in Xenserver, I believe it has something to with the video cards being presented as two devices:
Code:
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series]
I've noticed on the first boot of the VM after starting the Xenserver the audio device isn't presented with the GPU but it does appear after rebooting the VM. If it was explicitly passed though I suspect it would behave better.
 

outsider

New Member
Jan 21, 2017
17
3
3
48
UPDATE:
I borrowed an old AMD HD6870 GPU from a friend and tossed it in the system.
Booted up and did a quick install of UnRaid, and imagine my surprise when the GPU passed without any hassle to a VM.
Next test is to verify stability, and to maybe try ESXi.

My love for nvidia is starting to waiver....
 

TLN

Active Member
Feb 26, 2016
523
84
28
34
I got a Radeon cards and pased it through with no issues. Make sure you edit vm.pchole setings in vmx file, and choose to allocate all the memory for VM.
 

helsyeah

Active Member
Aug 22, 2015
111
30
28
43
As a data point (not the same CPUs or Mobo), I have an AMD RX480 on pass-through on ESXI 6.0U1 and its been working well since Oct/Nov 2016 running on a Supermicro X10SRL (UP) + E5-2630 V4 ES2 + 64 GB RAM.

I also had a Quadro 4000 (actually a hard modded GTX 660) that worked fine as well on ESXI 6.0 (no update).
 

nk215

Active Member
Oct 6, 2015
412
143
43
49
I also had a Quadro 4000 (actually a hard modded GTX 660) that worked fine as well on ESXI 6.0 (no update).
This doesnt make any sense. Switching a few resistors cant change a kepler to a fermi.
 

zer0sum

Well-Known Member
Mar 8, 2013
849
473
63
As a data point (not the same CPUs or Mobo), I have an AMD RX480 on pass-through on ESXI 6.0U1 and its been working well since Oct/Nov 2016 running on a Supermicro X10SRL (UP) + E5-2630 V4 ES2 + 64 GB RAM.

I also had a Quadro 4000 (actually a hard modded GTX 660) that worked fine as well on ESXI 6.0 (no update).
Did you have to do anything special to get the RX480 working?
I'm trying with a Gigabyte GA7-PESH2, RX480, ESXi 6.5.0-4564106 and when I boot the VM it hangs the whole system :(
 

helsyeah

Active Member
Aug 22, 2015
111
30
28
43
Did you have to do anything special to get the RX480 working?
I'm trying with a Gigabyte GA7-PESH2, RX480, ESXi 6.5.0-4564106 and when I boot the VM it hangs the whole system :(
Not that I am aware of. As I recall all I did was simply setup passthrough and add the 480 to the VM. Its a windows 10 VM if that matters.