ESXI 6.5u3 - GPU passthrough to single VM

[poto]

Hi all, I'm planning to virtualize personal workstation to minimize electrical cost and space; would appreciate experiences/opinions regarding GPU choice. Primary goal is to have stable GPU passthrough to single VM with minimum of tweaking. Rack is close enough to attach monitors, kb/mouse, audio directly, no vSGA or vGPU needed.

Current W/S:
SM X10SRA, E5-2667 v3, 32GB ram, Intel P3605 1.6tb NVME AIC, Nvidia GTX660 (3x HD monitors), Windows Server 2012R2
- this machine never "breaks a sweat", but the ancient GPU is pretty power-hungry even at idle

Target host:
SM X10SRL, E5-2697 v3, 128GB ram, Intel 750 1.2tb NVME AIC
Intel 10Gbase-T nic, 2x LSI 9207-8e (FreeNAS backup chassis)
- 5-8 VM's (media-management - Sonarr, Radarr, etc)
- plenty of ram/cpu to spare, planning to use P3605 from W/S to store VM's, iops should be sufficient

Use case:
basic business apps - MS Office, security cam monitoring, web browsing, occasional light gaming (vintage low-res stuff), must be able to light up 3x HD monitors so probably 4x mini-DP, single-slot and power-efficient, Windows Server 2012 R2

Plan:
1 - convert/clone WS to VM and migrate to secondary host as proof-of-concept
2 - validate operation via RDP (spare laptop will become host controller for maint/upgrade/migration)
3 - passthrough usb hub and test kb/mouse/audio
4 - install/passthrough GPU, reboot to validate passthrough VM hardware persistence and host stability
4 - attach monitors and profit

Incomplete information:
From reviewing available threads last 24 months, I'm concluding a safe choice would be Quadro P2000, currently between $250-$350. It seems to be supported for vDGA, and would also be useful if future needs included streaming transcodes.

Would appreciate hearing from anyone with similar OS/hardware experience. Also, if I'm missing any value choices for stable passthrough on ESXI 6.5u3 to single VM, saving a few $$$ is always welcome!

Cheers,
John

 

[besterino]

You forgot to mention what OS the VM with GPU will be running. As you mentioned ms office, I suppose Windows?

Not running your specific use case, or even close, but I currently have a X11SPi-TF/4108 with P400 and Renesas USB 3 controller pass-through’ed for a 24/7 browsing / Video watching / light weight application VM. Works like a charm. All with ESXi 6.7 („no U“) and Windows 10 VM. Uptime is months at a time (shutdown only for maintenance/ tinkering etc.). Setup was also easy.

As another system I run a Threadripper 1920X (X399D8A-T2) with a 2080Ti on ESXi 6.7U3.... totally different story. TR4-plattform causes a few more headaches than Intel for me, plus getting the consumer GPU to work in Passthrough also took a while. Currently use it as a „high end“ Gaming rig for testing purposes only, but 6 weeks now and no stability issues. CPU and RAM overclocked by the way. Yeah well, not recommended for production.

Long story short, Intel E5 plattform (or SP) is IMHO the best way to get it to work. No guarantee on the 2000P though, since NVIDIA is very reluctant to give away official information of what works other than the big Tesla’s and more recent Quadros (IIRC some game streaming services also use Quadros).

Out of curiosity I googled a bit for the 2000p, but only got mixed results / user stories.

 

[poto]

Good point, updated post with target OS.

When you mention "P400", is that a Quadro P400 GPU? If so, was it "plug & play" after standard esxi passthrough configuration, or were additional measures required? They are available < $100, so considerable cost savings vs Quadro P2000.

Common use case for Quadro P2000 seems to be plex servers providing transcoding streams to friends/family. That is really driving the prices up. Probably overkill for my needs, but I'm willing to pay a bit more for more stable/supported solution.

Regards,
John

 

[marv]

I run X9SRL-F with Xeon E5-2660v2 using vt-d for Radeon 7850 and VIA USB3 card into Windows 10 VM.

It works stable enough, but I had issues with GPU drivers. With Radeon Adrenaline 2019 series, sound was crapping out after a few hours-days and reboot was needed. I am now on 18.2.1 which seem stable.

 

[poto]

Thanks to those who contributed data points - much appreciated.

Migration checkpoints resolved - even managed to passthrough ancient GTX460, although drivers reset themselves semi-regularly when stressed.

All that remains is buying Quadro P2000, but current price trends point toward a seller's market. I'll put this project on "pause" while I wait for Quadro P2000 prices to trend downward a bit. One constant that can be counted on is the pipeline of new products, and the relentless upgrade cycle!

 

[poto]

Finally finished up transition to virtualized workstation. Main holdup was finding reasonably-priced used Quadro P2000; once purchased everything came together easily.

ESXI 6.5u3 host:
Datastore - Intel 1.2TB 750 NVME >> P3605 1.6TB NVME

VM workstation:
OS - Windows Server 2012R2 >> Server 2019
PCI passthrough - Quadro P2000 + HDMI audio, Renesas USB 3.0 card

Added Plugable USB 3.0 hub for adequate ports, and Tripplite HDMI audio extractor to drive studio monitors.

GPU passthrough required minor modifications to VM setup:
1 - all guest memory reserved
2 - added svga.vgaonly=TRUE to vm options>advanced
(the default vgpu "SVGA 3D" was randomly detected along with 3 external monitors, causing frequent screen blackouts until parameter changed)

No major issues over last 2 weeks, other than PCI passthrough unassigned itself after host reboot for patch application.

Next up is vcenter and hosts update to vsphere v6.7. Hopefully vmware has not deprecated any drivers or passthrough functionality, but we shall see!

Regards.