GPU Passthrough DL580-G7

[CyberianSoldier]

UPDATE:
ISSUE Resolved, see later post named "FINAL UPDATE" for guide and more information.



So, I am super new to VMware vsphere, and gpu passthrough, but after the past two days of trying things and googling, I have hit a huge wall with VMware, and I just don't know what to do, so I was hoping someone here had experienced something similar, or could help.
I installed ESXi on an hp DL580, and installed a gpu (AMD R5-240). The gpu seemed to work fine when I installed Windows sever on the hardware directly, so I switched over to ESXi.

I added the PCI expess cards to VMs, but now when I try to start them I get this fun error message.

Failed to power on virtual machine VM3. The systemId does not match the current system or the deviceId, and the vendorId does not match the device currently at 11:0.0.

I reserved the memory like it asked me to.

I gave the integrated GPU priority in the bios, so the vmware connect to this computer screen shows up on the integrated VGA connector (as I thought that the output of the graphics card may have been causing this issue).

But the thing is, I can't remove the pci device from any of the VMs, like, I'll remove it, hit save, and then I'll go back into the VM settings, and somehow the gpu/pci express device is still there!
So I am basically unable to remove the gpu. The actual device itself is doing strange things, it's grayed out and "active" but why is it active? it's not doing anything! or is it?
I can't find anything on google that remotely helps, the best suggestion is: "remove the pci express device and re add it" and as far as I can tell, that is literally impossible.

 

[marcoi]

have you tried turning off the server, physically removing the card and restarting? Also if the gui isnt letting you remove it you can try to ssh into the exsi host and modify the vmx file directly.

 

[CyberianSoldier]

I had a really hard time getting the card in the server. (the slot is a little too tight) I can try removing the card. Could I understand your rationale for the suggestion?
Does that reset something?
If I need to put it in another slot, I have to wait for a pci express expansion board to come in next week. (there's technically another pci express slot on the current board, but it's almost a 1/4 of an inch too small, so I'd probably need to "modify" the case, which is a process I'd rather not go through.

I am going to give changing slots one more time, it might be something I can bend without needing a dremal or cutting. Will post results.

 

[marcoi]

you can remove it from pci passthrough first without moving it then power off the box and restart it. that might remove it.

i meant from the esxi host configuration and not the VM you added it too.

 

[CyberianSoldier]

I'm not sure. I can't seem to do anything with the card. I tried showing that with screenshots. Maybe I could play with ssh, but now the server won't boot. It can't get past the raid controller screen. I did manage to move the gpu to the other slot by removing the bracket.

 

[marcoi]

i would boot with the card, make sure esxi registers that the card is no longer configured as pci pass through, then shut down and try again.

 

[CyberianSoldier]

So, I ended up doing the following steps:

Took the video card out.
Turned the power off to the server, now it boots again.
Re-installed ESXi.
(ensured it was working)
Added the videocard back to the system.

So now, it is grayed out again, and it's disabled.
(EDIT) and I can't toggle passthrough for the card.

It should be noted, the graphics card has a monitor hooked up to the dvi port. there is no output.

 

[CyberianSoldier]

No luck. Still the same error message. Tried installing vmware tools in the guest OS, still nothing.
Is there a way to over-ride error messages like that?

(EDIT)
Attempts to disable the videocard have not been successful. The video card is still marked as "active" and grayed out (See the first set of screenshots, nothing has changed.) Is there any way to see what is using the resource or why it is grayed out?

I guess it's time to ssh into this server.

(Update 2)
The device is removable with the vsphere client. This isn't particularly helpful for my cause, but if someone has the same error I had, this can be a way to salvage a virtual machine.

 

[CyberianSoldier]

FINAL UPDATE:

So I got it working. If anyone else is running into this issue, here's what I did and what worked.

----------------GPU Passthrough The Sketchy Guide---------------------
Step 1, create a normal virtual machine through V-SPHERE (I made mine the latest version) DO NOT add a pci device.
Step 2, log into the VM and enable remote connections (this guide assumes windows server 2012).
Step 3, add your pci hardware through the WEB interface, restart the host
Step 4, add the PCI hardware through V-SPHERE (VM must be off).
Step 5, log into the VM with remote connection, install drivers, restart VM.
Step 6, the VM should be outputting to the monitor.

Notes:
>If you use the DL580-G7 or similar hp hardware, there is an option to prioritize video output for internal video, this is under "advanced settings" and "video" in the bios. This will free up your video card.
>DO NOT edit the VM settings in the web client, it will die. I don't know why, but it just doesn't work, and as far as I can tell, there is no recovery, except to remake the VM (you can keep the same virtual hard drive, I tested that).

 

[marcoi]

glad you got it working