GPU for passthrough

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Kapel

New Member
Dec 26, 2016
14
1
3
32
First of all - sorry if I'm in the wrong forum. It's not a Hypervisor-related topic, also I have found no GPU related forum so this might be the best for this one. If I'm mistaken - please move the topic if it's possible.

I would like to build a Virtualization server and learn GPU passthrough on different hypervisors. My idea is to have multiple running VM's with graphic acceleration. The question is:

Is there a single card (with multiple cores?) that can be passed to multiple VMs. For example - one card with four cores could accelerate up to 4 different VMs. Is it possible?

I'm looking for a good price:quality:easy-using (so fairly cheap, quite good and not very hard to work with) ratio. In other words: What is the best card I could get for the lowest budget? What options are on the 2nd hand market? Quadros? Grid? Firepros? Consumer GPUs?

Thanks,
K.
 

Rand__

Well-Known Member
Mar 6, 2014
6,634
1,767
113
Cheap: Get x cards for x VMs, either AMD or Nvidia Quadros for best compatibility.
Proper: Get a Pro card (depending on your Hypervisor, check compatibility) might be expensive - eg Nvidia Grid.

What do you want to do in the VMs? Remote users? Mining? Encoding?
 

kemic

New Member
Aug 15, 2015
26
4
3
For the most part when talking gpu pass through it's 1 to 1. If you're interested in the 1 to 1 stuff I recommend looking at VFIO Discussion and Support • /r/VFIO or VFIO tips and tricks

The technologies you're describing sounds closely related to vDGA or nvidia vGPU for use with virtual desktops and VMWare Horizon suite (as one example).

This carving up of the gpu generally requires the more expensive Quadro, GRID, or FirePro cards.

If you're handy with a soldering iron you can change some resistors on the consumer cards to trick the OS into thinking it's one of the other expensive ones. This is likely one of the cheapest ways to get your feet wet:

http://www.eevblog.com/forum/chat/hacking-nvidia-cards-into-their-professional-counterparts/
 
  • Like
Reactions: Kapel

Kapel

New Member
Dec 26, 2016
14
1
3
32
Cheap: Get x cards for x VMs, either AMD or Nvidia Quadros for best compatibility.
Proper: Get a Pro card (depending on your Hypervisor, check compatibility) might be expensive - eg Nvidia Grid.

What do you want to do in the VMs? Remote users? Mining? Encoding?
I would like to do remote users - start simple apps that require GPU acceleration (won't start without it :(). I possibly could use 1 card per 1 vm, but I'm trying to get as much as I can. So if I can get 1 card that would provide acceleration for 4 vms (example) than I could possibly buy N of those cards and get N*4vms supported.

For the most part when talking gpu pass through it's 1 to 1. If you're interested in the 1 to 1 stuff I recommend looking at VFIO Discussion and Support • /r/VFIO or VFIO tips and tricks

The technologies you're describing sounds closely related to vDGA or nvidia vGPU for use with virtual desktops and VMWare Horizon suite (as one example).

This carving up of the gpu generally requires the more expensive Quadro, GRID, or FirePro cards.

If you're handy with a soldering iron you can change some resistors on the consumer cards to trick the OS into thinking it's one of the other expensive ones. This is likely one of the cheapest ways to get your feet wet:

[MOVED] Hacking NVidia Cards into their Professional Counterparts - Page 1

Thank you the links you provided. I'll take a look on them!

I've read about resoldering the cards and I do consider that, but I need to find all my options first.

Would any (even older) Quadro / FirePro be ok? I'm not familiar with those pro cards. I see that some of them are quite cheap in fact.

Is there any general rule on how problematic a GPU is? Or the problems are hypervisor related?

Thanks,
K.
 

Rand__

Well-Known Member
Mar 6, 2014
6,634
1,767
113
1. I have seen no comment on that thread about resoldering where it really worked for your (mine too) use-case. Happy to be corrected.
2. Compatibility with cards depends on Hypervisor's and versions (i.e. latest ESX only supports Grid officially)
3. GPU pass through support is similar - KVM accepts more GPUs but is more complicated to set up for example, ESX is simple but picky.

So in the ends it depends on your budget (money & time) which way to go :)
 

Kapel

New Member
Dec 26, 2016
14
1
3
32
1. I have seen no comment on that thread about resoldering where it really worked for your (mine too) use-case. Happy to be corrected.
2. Compatibility with cards depends on Hypervisor's and versions (i.e. latest ESX only supports Grid officially)
3. GPU pass through support is similar - KVM accepts more GPUs but is more complicated to set up for example, ESX is simple but picky.

So in the ends it depends on your budget (money & time) which way to go :)
My project is to learn as much as I can, so I would be trying different Hypervisors anyway. That's why I'm looking for a card that might work in most of the systems. Probably I would start with Proxmox (or pure Debian with KVM) > ESXi > Hyper-V > Xen.

Cheers,
K.
 

zir_blazer

Active Member
Dec 5, 2016
357
128
43
The easiest way would be to use Radeons, one per VM that you want to have a Video Card. You will need more Monitors, or a Monitor with multiple inputs that allows you to switch betweem them via buttons, or a KVM Switch (Not the KVM Hypervisor!), since when you do Passthrough the guest OS gets entire control of the Video Card and thus your Hypervisor can't read its framebuffer to display in the host as it would do with an emulated or paravirtualized GPU. You will need extra Software, like TeamViewer or something, to display the guest OS Video Card output on the host.
You can also use GeForces. nVidia Drivers have Hypervisor checks and they refuse to initialize the Video Card if it detects that it is running inside a VM. QEMU with VFIO can workaround that (Disable KVM CPUID Leaf and change the default Hyper-V Vendor ID).
You don't need to mod cards or anything else. GeForce @ Quadro solder mods were popular before VFIO managed to workaround nVidia Drivers checks.

The amount of Video Cards that you have will also influence platform choice. LGA 2011/2011-3 is the best option since the Processor PCIe Controller supports PCIe ACS, which you will need (Workaroundeable by using a Linux kernel patch that is considered an insecure hack, but in practice it works).


Dual GPU cards exists, but these are usually set up as a Crossfire or SLI on the same Video Card, with a master and slave GPU. The master GPU has all the video outputs for itself, the slave GPU has no direct video output. They don't work as you would expect them. The same should apply to nVidia GRIDs. I think there was a model with 4 GPUs, but again, you need an alternative way to get the video output out of them. Besides, nVidia required a propietary Hypervisor extension for their GRIDs, IIRC.
 

Kapel

New Member
Dec 26, 2016
14
1
3
32
The easiest way would be to use Radeons, one per VM that you want to have a Video Card. You will need more Monitors, or a Monitor with multiple inputs that allows you to switch betweem them via buttons, or a KVM Switch (Not the KVM Hypervisor!), since when you do Passthrough the guest OS gets entire control of the Video Card and thus your Hypervisor can't read its framebuffer to display in the host as it would do with an emulated or paravirtualized GPU. You will need extra Software, like TeamViewer or something, to display the guest OS Video Card output on the host.
You can also use GeForces. nVidia Drivers have Hypervisor checks and they refuse to initialize the Video Card if it detects that it is running inside a VM. QEMU with VFIO can workaround that (Disable KVM CPUID Leaf and change the default Hyper-V Vendor ID).
You don't need to mod cards or anything else. GeForce @ Quadro solder mods were popular before VFIO managed to workaround nVidia Drivers checks.

The amount of Video Cards that you have will also influence platform choice. LGA 2011/2011-3 is the best option since the Processor PCIe Controller supports PCIe ACS, which you will need (Workaroundeable by using a Linux kernel patch that is considered an insecure hack, but in practice it works).


Dual GPU cards exists, but these are usually set up as a Crossfire or SLI on the same Video Card, with a master and slave GPU. The master GPU has all the video outputs for itself, the slave GPU has no direct video output. They don't work as you would expect them. The same should apply to nVidia GRIDs. I think there was a model with 4 GPUs, but again, you need an alternative way to get the video output out of them. Besides, nVidia required a propietary Hypervisor extension for their GRIDs, IIRC.
Thank you for your reply! It's not quite my idea to have the cards plugged into a monitor. On some stage I would like to put this machine into a rack and connect to it fully remotely. Also I would like to have multiple passthroughs at a time. To each virtual machine I'll connect via VNC / remote desktop / anything other. I was planning to use a Supermicro 2011 v1 / v2 motherboard - either Supermicro X9DRI-F (3x PCI-E x16 and 3x pci-e x8) or a XDRD-iF (1x pci-e x16 and 4x pci-e x8). So in the 'worst' scenario I could possibly connect up to 5/6 GPU's. It's an fair amount, but I'm still looking.

I've also seen the GRID card with 4 GPU's, but I'm looking if there is any other GPU that could do the job but for less cash. If I won't have a choice, than I'll just buy some low end, passive graphics cards and hope that they'll do the work.

I was just wondering if anyone here has done such thing and if it's possible at all.

Cheers,
K.
 

nk215

Active Member
Oct 6, 2015
412
143
43
50
If you value your time, you only have 2 clear choices, Quadro and Grid.

The minimum I recommended quadro is the old Quadro 4000. It can be used in 2 modes:

+ Virtual Hardware Shared GPU accelerated (which is limited to DirectX 9). This is vSGA and you can share the same GPU among multiple VM, limited by memory. Quadro 4000 has 2Gig and can easily support 8 VMs as long as you don't need anything beyond DirectX 9 (no OpenGL, CUDA in this mode).

+ Virtual Dedicated Graphic. This is vDGA via pci-pass-thru. You have access to every feature of the card, DirectX/OpenGL/CUDA etc. One Card per VM. Quadro 4000 is a single slot card so you can stack 4-5 of those in a box easily.

The above 2 modes are not multually exclusive, you can have 3 cards in vDGA modes for 3 VMs and 1 card in vSGA mode for the other 8 VMs or so.

A step above the Quadro card is the GRID card. With the GRID card, the only realistic mode is vGPU which combined the best of the above 2 methods. The only limitation of vGPU is no CUDA support in the K1/K2 cards.

K1 can support up to 32 VMs in vGPU. K2 can support 16 VMs but at a higher performance. Each user only have half a gig of video memory at max VM support.

You can also use vSGA and vDGA with the GRID (4 for K1, 2 for K2) but there's no point. Please note vGPU is a nVidia technology with ESXi so if don't use ESXi you are limited to the other 2 methods.

AMD cards also work in sVGA and vDGA (firepro series)but support is rather limited. ESXi 6.5 does not support any firepro cards in virtual shared mode. There's no vGPU equivalent in AMD cards.

I have 4 quadro cards in my setup; 3 in vDGA and 1 in vSGA. My "back up" has Quadro 2000 in vDGA. I like Quadro 2000 since it doesn't need an extra power connector but at a much reduced performance vs Quadro 4000. Quadro 4000 is around $75-$80. Quadro 2000 is around $50.

Other hypervisors are not as picky as ESXi in vDGA (vDGA works with just about any cards in KVM for example). The real advantage of other hypervisors is to use high end consumer GPUs (GTX1080 for example) for games. If you don't game in your VM, previous versions of quadro cards work very well.
 
Last edited:

nk215

Active Member
Oct 6, 2015
412
143
43
50
There is no license for vDGA (there is no extra license for vGPU with the grid k1/k2 either. Next gen grid will need special license)

I play games on my VM via zero client often. Nothing too demanding. Games like CIV5, New XCOM etc.
 
  • Like
Reactions: Kapel

Rand__

Well-Known Member
Mar 6, 2014
6,634
1,767
113
And I was only looking at Grid's for Horizon - thanks - Maybe ThinClient Gaming can be done cheaply after all :)

... Reading ...
Ah thats the same tech allowing CPU Graphics to be shared in 6.5 - nice:)

... More Reading ...
Whats the 'best' gaming Quadro to get? Not looking for Doom, but Civ6 would be nice - I assume the 4000 will be too weak for that? 6000's are €400... not sure whether thats worth it (if it would work after all, that seems to bed difficult to determine (no HCL))

And https://forums.servethehome.com/index.php?threads/cheap-gpu-for-esxi-passthrough.9201/page-3 here as a cross reference...
 
Last edited:

zir_blazer

Active Member
Dec 5, 2016
357
128
43
The minimum I recommended quadro is the old Quadro 4000. It can be used in 2 modes:

+ Virtual Hardware Shared GPU accelerated (which is limited to DirectX 9). This is vSGA and you can share the same GPU among multiple VM, limited by memory. Quadro 4000 has 2Gig and can easily support 8 VMs as long as you don't need anything beyond DirectX 9 (no OpenGL, CUDA in this mode).

+ Virtual Dedicated Graphic. This is vDGA via pci-pass-thru. You have access to every feature of the card, DirectX/OpenGL/CUDA etc. One Card per VM. Quadro 4000 is a single slot card so you can stack 4-5 of those in a box easily.
I think you are basing your suggestion entirely on what VMWare ESXi uses. I never used ESXi, only Xen and QEMU-KVM, but these features aren't really exclusive to it.
If sharing the GPU is "limited to DX9", it probabily means that it provides a virtual GPU that does API Forwarding and thus is limited to the APIs that it has implemented. This is also supported in other Hypervisors that can provide paravirtualized Video Cards with a front end/back end pair, so it can get the job done on the host GPU. VirtualBox does that, and QEMU-KVM has the VirtIO GPU/Virgil3D, but it only works on Linux for OpenGL so far. If they get Windows and DirectX support, it will have feature parity with ESXi vSGA, but without the specific Video Card requeriments.
Passthrough works theorically on anything. nVidia actively wants to play against Passthrough users of their consumer GeForces, but QEMU-VFIO can workaround that (This is why for Xen you could only use GeForce @ Quadros, they weren't aware of how to workaround it).

The thing which I never researched about is how to effectively get video output of a VM with a Passthoughed Video Card with no direct video output. Be it on the same system that host via a virtual bridge or a remote system in the LAN, routing the video somewhere else would take an enormous amount of bandwidth, or performance and quality loss due to have to compress it in real time. Not sure how to deal with that.


You can also use vSGA and vDGA with the GRID (4 for K1, 2 for K2) but there's no point. Please note vGPU is a nVidia technology with ESXi so if don't use ESXi you are limited to the other 2 methods.
nVidia was colaborating in a vGPU interface for VFIO to use their GRIDs. There were patchs for it. Not sure how much it will take to fully implement.


I have 4 quadro cards in my setup; 3 in vDGA and 1 in vSGA. My "back up" has Quadro 2000 in vDGA. I like Quadro 2000 since it doesn't need an extra power connector but at a much reduced performance vs Quadro 4000. Quadro 4000 is around $75-$80. Quadro 2000 is around $50.
I wouldn't feel very happy with a 6 years old piece of Hardware. The Quadro 4000 is GF100 based (GF465/470/480), but with near half of its units disabled, it is actually slower than a GF465.
Besides, it may not have UEFI GOP support in its Option ROM, theorically forcing you to only be able to do Primary VGA Passthrough with it with a BIOS Firmware for the guest VM. There was a tool (updGOP) that could patch old ROMs to include UEFI GOP, but I don't know if it also works in Quadros since I don't recall anyone using it for them.


Other hypervisors are not as picky as ESXi in vDGA (vDGA works with just about any cards in KVM for example). The real advantage of other hypervisors is to use high end consumer GPUs (GTX1080 for example) for games. If you don't game in your VM, previous versions of quadro cards work very well.
The fact that you can throw anything to QEMU-VFIO and expect it to work is what makes it soo ridiculous great.
 

Kapel

New Member
Dec 26, 2016
14
1
3
32
If you value your time, you only have 2 clear choices, Quadro and Grid.

The minimum I recommended quadro is the old Quadro 4000. It can be used in 2 modes:

+ Virtual Hardware Shared GPU accelerated (which is limited to DirectX 9). This is vSGA and you can share the same GPU among multiple VM, limited by memory. Quadro 4000 has 2Gig and can easily support 8 VMs as long as you don't need anything beyond DirectX 9 (no OpenGL, CUDA in this mode).

+ Virtual Dedicated Graphic. This is vDGA via pci-pass-thru. You have access to every feature of the card, DirectX/OpenGL/CUDA etc. One Card per VM. Quadro 4000 is a single slot card so you can stack 4-5 of those in a box easily.

The above 2 modes are not multually exclusive, you can have 3 cards in vDGA modes for 3 VMs and 1 card in vSGA mode for the other 8 VMs or so.

A step above the Quadro card is the GRID card. With the GRID card, the only realistic mode is vGPU which combined the best of the above 2 methods. The only limitation of vGPU is no CUDA support in the K1/K2 cards.

K1 can support up to 32 VMs in vGPU. K2 can support 16 VMs but at a higher performance. Each user only have half a gig of video memory at max VM support.

You can also use vSGA and vDGA with the GRID (4 for K1, 2 for K2) but there's no point. Please note vGPU is a nVidia technology with ESXi so if don't use ESXi you are limited to the other 2 methods.

AMD cards also work in sVGA and vDGA (firepro series)but support is rather limited. ESXi 6.5 does not support any firepro cards in virtual shared mode. There's no vGPU equivalent in AMD cards.

I have 4 quadro cards in my setup; 3 in vDGA and 1 in vSGA. My "back up" has Quadro 2000 in vDGA. I like Quadro 2000 since it doesn't need an extra power connector but at a much reduced performance vs Quadro 4000. Quadro 4000 is around $75-$80. Quadro 2000 is around $50.

Other hypervisors are not as picky as ESXi in vDGA (vDGA works with just about any cards in KVM for example). The real advantage of other hypervisors is to use high end consumer GPUs (GTX1080 for example) for games. If you don't game in your VM, previous versions of quadro cards work very well.
I was waiting for a post like this! Thank you!



Cheers,
K.
 

nk215

Active Member
Oct 6, 2015
412
143
43
50
The thing which I never researched about is how to effectively get video output of a VM with a Passthoughed Video Card with no direct video output. Be it on the same system that host via a virtual bridge or a remote system in the LAN, routing the video somewhere else would take an enormous amount of bandwidth, or performance and quality loss due to have to compress it in real time. Not sure how to deal with that.
This is where NICE DCV and PCoIP come in. you can test drive NICE DCV here: TestDrive - NICE

For PCoIP, use the hardware solution such as Tera 2321 zero client device (less than $100 box). I just tested Civ 6 last night with PCoIP and it works (ESXi).
 

Kapel

New Member
Dec 26, 2016
14
1
3
32
I have 4 quadro cards in my setup; 3 in vDGA and 1 in vSGA. My "back up" has Quadro 2000 in vDGA. I like Quadro 2000 since it doesn't need an extra power connector but at a much reduced performance vs Quadro 4000.
I've had a short chat with a friend of mine, which stated that the Quadro 4000 has overheating problems. How true is that? How do you manage to have your quadros in good working temp? How does the cooling of your machine look?

My friend also suggested to buy K4200, but it's way more expensive than the Quadro4000 - it's better to get a GRID card in that budget (as you mentioned).
 

ecosse

Active Member
Jul 2, 2013
463
111
43
I've had a short chat with a friend of mine, which stated that the Quadro 4000 has overheating problems. How true is that? How do you manage to have your quadros in good working temp? How does the cooling of your machine look?

My friend also suggested to buy K4200, but it's way more expensive than the Quadro4000 - it's better to get a GRID card in that budget (as you mentioned).
I've never had a problem with my Quadro 4000 overcooling either in a DL380pgen9 or a custom chassis. They both had / have reasonably decent cooling though it was no typhoon or anything.