First SR-IOV Setup

[sirbamalot]

Hi all I`m a newbie, but I`m aways passionate about new tech and will try to do my first up to date Proxmox SR-IOV server install.
My last successful attempt was whit VMware 7.0.3 and Instinct MI25 where MxGPU worked but the drivers are old and outdated Q2.2018, they still function and you can play and work whit hardware acceleration whit moonlight, but you are stuck to H.254 encoding.
So my next project is more ambitions, I have a mixed GPUs and the server is a Gigabyte MZ22-G20-00, 1 x 7B13 Epyc 64c/128t, 512GB 3200Mhz DDR4, 6 x 2TB Nvme, 2 x 120GB RAID-1 for OS, 1 x Radeon Pro V620 32GB, 2 x Radeon Pro V710 28GB, 1 x RTX 4000 8GB, 1 x Nvidia NVIDIA PH402 SKU 200 which is dual P100 32GB HBM2 on a single PCIe x16 board.
- So far I successfully installed AMD Open Source MxGPU driver and have the 2 V710 partitioned for 4 VF each.
- I installed the Nvidia Grid driver whit only the RTX 4000 able to use the vGPU functionality.
- Apparently the P100 GPU doesn't have mdev option when chosen as PCI device to add in the VM. The "nvidia-smi vgpu" list it so will twinker a bit and see. For now it`s fine for Passthrough.
- For the V620 I`ll wait for AMD to finally support it and use it for Passthrough for now.
- I have one Intel ARC pro B60 on order and when it arrives hopefully will be able to configure it for SR-IOV also on the same machine
To have all 3 major GPU vendors on one server and being able to choose which one you want is cool in my opinion
So I`ll do a lot of testing and if someone is interested in just try out and test PM me or leave a comment. I have a lot to do for the Holidays but I`m still online most of the time.

 

[pr1malr8ge]

Good luck,
I've considered switching over to proxmox from esxi how ever the pci pass through hurdles is what has stopped me from attempting it. Maybe if I can build a new machine and fiddle around to learn proxmox so as not to take down my production for an extended period maybe I might switch.
I will say I just know esxi and it works. Shame that broadcom had to go and ruin it.

 

[sirbamalot]

Good luck,
I've considered switching over to proxmox from esxi how ever the pci pass through hurdles is what has stopped me from attempting it. Maybe if I can build a new machine and fiddle around to learn proxmox so as not to take down my production for an extended period maybe I might switch.
I will say I just know esxi and it works. Shame that broadcom had to go and ruin it.

From my experience I was using Proxmox and then try out VMware just to make MxGPU on Instinct MI25 work it's easier to set GPUs and LAN PCI devices but I was having 20 pcs. of Nvme M.2 drives in that server and I was having a rough time to bind PCIe addresses to actual manufacturer and model. Otherwise Proxmox doesn't have that modern GUI look as newer VMware but if you map your devices it's very easy afterwards and you can also try out the experimental High-availability whit pci pass through.