ESXi 7.0U1c Nvidia passthrough selects multiple devices.

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
R

rhyzar

New Member
Sep 2, 2020
10
0
1
Hello,

I am having an issues with trying to passthough a Nvidia Quadro P1000.
I understand that there is an issue with 7.0 and needing to toggle multiple times, this is not that issue.

When selecting both NVIDIA Corporation GP107GL Quadro P1000 and High Definition Audio Controller, and selecting Toggle passthrough, the High Definition Audio Controller shows 'Active' and Quadro P1000 shows 'Enabled / Needs reboot' as expected.
After rebooting the server, The P1000 still shows 'Enabled / Needs reboot', but my Broadcom MegaRAID SAS Invader Controller (9361-8i) now shows passthrough as 'Active' as well - this is my RAID controller for my main datastores, so of course I can't have it be toggled as passthrough as I am unable to access my vm's and datastore.

Now I have tried to deselect it, and reboot, but it will say enabled as long as the Nvidia card is toggled for passthrough.
I have tried to do this through VCSA and local ESXi, and it produces the same result.
I have tried to manually edit /etc/vmware/ah-trees.conf to remove the RAID controller, but it will simply re-appear after reboot.

I have tried to move the physical cards to different sides of the PCI-E risers, separate them, and still it does the same thing, and they seem to be tied together.

Not sure if this is any help, but the hardware address for MegaRAID 9361-8i is 0000:81:00.0, the P1000 is 0000:83:00.1 (Audio) &0000:93:00.0 (P1000).
The only other things that are on the PCI-E bus are a StarTech PEXUSB312EIC USB controller, and a Dell N20KJ Broadcom 57810 Dual Port 10Gb DA/SFP+ Converged Network Adapter
I would try to go back to 6.5, but I have already updated the VM hardware for all my VM's.

Does anyone have any clue what might be going on, or anything that I can try?

I am running a SuperMicro X10DRW-iT motherboard with RSC-R2UW-2E8E16 & RSC-R2UW-2E8R riser cards
Dual Xeon E5-2630v3 CPU
176gb 1866 MHz DDR4
Broadcom LSI MegaRAID 9361-8i RAID controller
Nvidia P1000 GPU
Dell N20KJ Broadcom 57810 NIC
StarTech PEXUSB312EIC USB 3.1 controller
 
G

gb00s

Well-Known Member
Jul 25, 2018
1,189
600
113
Poland
Just a side-note: Do not assign Nvidia High Definition Audio Controller to passthrough. Never ever :)
 
R

rhyzar

New Member
Sep 2, 2020
10
0
1
It gets selected with the Nvidia P1000.
So if you deselect audio, it deselects the p1000.
 
R

rhyzar

New Member
Sep 2, 2020
10
0
1
even tried through CLI command - same result

Before Reboot:
1612838092970.png
[root@esxi:~] cat /etc/vmware/ah-trees.conf
{
"description" : "Host Devices",
"nodeType" : "ANY_CHILD",
"weightType" : "none",
"children" : [
{
"description" : "pciPassthru",
"nodeType" : "ANY_CHILD",
"weightType" : "none",
"children" : [
{
"description" : "NVIDIA Corporation GP107GL High Definition Audio Controller",
"nodeType" : "INSTANCE",
"InstanceId" : "pcipassthru-0x10de-0x0fb9-0000:83:00.1",
"attributes" : {
"id" : {
"overridable" : false,
"intValue" : 33537
},
"deviceId" : {
"overridable" : false,
"intValue" : 4025
},
"vendorId" : {
"overridable" : false,
"intValue" : 4318
},
"systemId" : {
"overridable" : false,
"strValue" : "5c3de0d7-8d04-c45d-c9e7-0cc47a54e55c"
},
"customLabel" : {
"overridable" : true,
"strValue" : ""
},
"moduleName" : {
"overridable" : false,
"strValue" : "pciPassthru"
},
"subDeviceId" : {
"overridable" : false,
"intValue" : 4540
},
"subVendorId" : {
"overridable" : false,
"intValue" : 4156
},
"revisionId" : {
"overridable" : false,
"intValue" : 161
},
"deviceName" : {
"overridable" : false,
"strValue" : "GP107GL High Definition Audio Controller"
},
"vendorName" : {
"overridable" : false,
"strValue" : "NVIDIA Corporation"
}
}
}
]
}
]
}[root@esxi:~]

After reboot:
1612838564835.png

[root@esxi:~] cat /etc/vmware/ah-trees.conf
{
"description" : "Host Devices",
"nodeType" : "ANY_CHILD",
"weightType" : "none",
"children" : [
{
"description" : "pciPassthru",
"nodeType" : "ANY_CHILD",
"weightType" : "none",
"children" : [
{
"description" : "NVIDIA Corporation GP107GL High Definition Audio Controller",
"nodeType" : "INSTANCE",
"InstanceId" : "pcipassthru-0x10de-0x0fb9-0000:83:00.1",
"attributes" : {
"id" : {
"overridable" : false,
"intValue" : 33537
},
"deviceId" : {
"overridable" : false,
"intValue" : 4025
},
"vendorId" : {
"overridable" : false,
"intValue" : 4318
},
"systemId" : {
"overridable" : false,
"strValue" : "5c3de0d7-8d04-c45d-c9e7-0cc47a54e55c"
},
"customLabel" : {
"overridable" : true,
"strValue" : ""
},
"moduleName" : {
"overridable" : false,
"strValue" : "pciPassthru"
},
"subDeviceId" : {
"overridable" : false,
"intValue" : 4540
},
"subVendorId" : {
"overridable" : false,
"intValue" : 4156
},
"revisionId" : {
"overridable" : false,
"intValue" : 161
},
"deviceName" : {
"overridable" : false,
"strValue" : "GP107GL High Definition Audio Controller"
},
"vendorName" : {
"overridable" : false,
"strValue" : "NVIDIA Corporation"
}
}
},
{
"description" : "Broadcom MegaRAID SAS Invader Controller",
"nodeType" : "INSTANCE",
"InstanceId" : "pcipassthru-0x1000-0x005d-0000:81:00.0",
"attributes" : {
"id" : {
"overridable" : false,
"intValue" : 33024
},
"deviceId" : {
"overridable" : false,
"intValue" : 93
},
"vendorId" : {
"overridable" : false,
"intValue" : 4096
},
"systemId" : {
"overridable" : false,
"strValue" : "5c3de0d7-8d04-c45d-c9e7-0cc47a54e55c"
},
"customLabel" : {
"overridable" : true,
"strValue" : ""
},
"moduleName" : {
"overridable" : false,
"strValue" : "pciPassthru"
},
"subDeviceId" : {
"overridable" : false,
"intValue" : 37729
},
"subVendorId" : {
"overridable" : false,
"intValue" : 4096
},
"revisionId" : {
"overridable" : false,
"intValue" : 2
},
"deviceName" : {
"overridable" : false,
"strValue" : "MegaRAID SAS Invader Controller"
},
"vendorName" : {
"overridable" : false,
"strValue" : "Broadcom"
}
}
}
]
}
]
}
 
T

TLN

Active Member
Feb 26, 2016
523
84
28
34
gb00s said:
Just a side-note: Do not assign Nvidia High Definition Audio Controller to passthrough. Never ever :)
Click to expand...
What's the problem? When I'm passing through card to VM I'm passing all the devices. Worked for both RTX2000 and RTX3000 series cards
 
R

rhyzar

New Member
Sep 2, 2020
10
0
1
My problem specifically is that when passing the Nvidia card, my raid controller follows for the ride. It's not a welcome passenger. I want it to stay home. :)
 
R

rhyzar

New Member
Sep 2, 2020
10
0
1
zack$ said:
Have you tried edited passthru.map to manually pass thru?
Click to expand...
I have not, as I am really not sure what to modify.
Here is the contents of the etc/vmware/passthru.map:

# Intel 82579LM Gig NIC can be reset with d3d0
8086 1502 d3d0 default
# Intel 82598 10Gig cards can be reset with d3d0
8086 10b6 d3d0 default
8086 10c6 d3d0 default
8086 10c7 d3d0 default
8086 10c8 d3d0 default
8086 10dd d3d0 default
# Broadcom 57710/57711/57712 10Gig cards are not shareable
14e4 164e default false
14e4 164f default false
14e4 1650 default false
14e4 1662 link false
# Qlogic 8Gb FC card can not be shared
1077 2532 default false
# LSILogic 1068 based SAS controllers
1000 0056 d3d0 default
1000 0058 d3d0 default
# NVIDIA
10de ffff bridge false
# AMD FCH SATA Controller [AHCI mode]
1022 7901 d3d0 default
 
T

TLN

Active Member
Feb 26, 2016
523
84
28
34
rhyzar said:
My problem specifically is that when passing the Nvidia card, my raid controller follows for the ride. It's not a welcome passenger. I want it to stay home. :)
Click to expand...
I had Nvidia and USB passed to one VM, raid to "NAS VM" and another USB controller to third VM. Works fine. ESXI 6.7 I believe
 
Z

zack$

Well-Known Member
Aug 16, 2018
708
338
63
rhyzar said:
I have not, as I am really not sure what to modify.
Here is the contents of the etc/vmware/passthru.map:

# Intel 82579LM Gig NIC can be reset with d3d0
8086 1502 d3d0 default
# Intel 82598 10Gig cards can be reset with d3d0
8086 10b6 d3d0 default
8086 10c6 d3d0 default
8086 10c7 d3d0 default
8086 10c8 d3d0 default
8086 10dd d3d0 default
# Broadcom 57710/57711/57712 10Gig cards are not shareable
14e4 164e default false
14e4 164f default false
14e4 1650 default false
14e4 1662 link false
# Qlogic 8Gb FC card can not be shared
1077 2532 default false
# LSILogic 1068 based SAS controllers
1000 0056 d3d0 default
1000 0058 d3d0 default
# NVIDIA
10de ffff bridge false
# AMD FCH SATA Controller [AHCI mode]
1022 7901 d3d0 default
Click to expand...
Run lspci -n and look for 0000:84:00.0 , 0000:84:00.1 , 0000:84:00.2 and 0000:84:00.3.

Copy the last 8 digits for the read out of each of the above.

In the passthru.map file, edit and insert

#P1000
Xxxx Xxx0 d3d0 false

Xxxx Xxx1 d3d0 false

Xxxx Xxx2 d3d0 false

Xxxx Xxx3 d3d0 false

Save and exit. Reboot.

I normally pass thru everything with no issues.
 
R

rhyzar

New Member
Sep 2, 2020
10
0
1
Ran lspci -n - found 0000:83:00.0 & 0000:83:00.0

0000:83:00.0 Class 0300: 10de:1cb1
0000:83:00.1 Class 0403: 10de:0fb9

modified passthrough.map:

# Intel 82579LM Gig NIC can be reset with d3d0
8086 1502 d3d0 default
# Intel 82598 10Gig cards can be reset with d3d0
8086 10b6 d3d0 default
8086 10c6 d3d0 default
8086 10c7 d3d0 default
8086 10c8 d3d0 default
8086 10dd d3d0 default
# Broadcom 57710/57711/57712 10Gig cards are not shareable
14e4 164e default false
14e4 164f default false
14e4 1650 default false
14e4 1662 link false
# Qlogic 8Gb FC card can not be shared
1077 2532 default false
# LSILogic 1068 based SAS controllers
1000 0056 d3d0 default
1000 0058 d3d0 default
# NVIDIA
#10de ffff bridge false
# AMD FCH SATA Controller [AHCI mode]
1022 7901 d3d0 default
#P1000
10de 1cb1 d3d0 false
10de 0fb9 d3d0 false

rebooted

P1000 did not show up as available under PCI devices
Went ahead and toggled PCI device for passthrough

rebooted

same boat:
1613071616458.png

Any other ideas?:(
 
Z

zack$

Well-Known Member
Aug 16, 2018
708
338
63
Is the 9361-8i on the bifurcated x32 ? Maybe try the x16 right-side pcie slot?

This shouldn't normally cause this type issue but worth a try.
 
R

rhyzar

New Member
Sep 2, 2020
10
0
1
That is what I thought as well initially so I purchased the RSC-R2UW-2E8R right side riser card so that I can put it and the USB controller on the two 8x ports (before I just had a riser card with one 8x port).
So, I have the 9361-8i on the right with the USB, and the P1000 with the Broadcom 57810 NIC on the left side on the RSC-R2UW-2E8E16.
 
Z

zack$

Well-Known Member
Aug 16, 2018
708
338
63
Try booting esxi with only the p1000 and 9361-8i?

If that doesn't work, try a previous version of ESXI.
 
R

rhyzar

New Member
Sep 2, 2020
10
0
1
I know it has been a little while, but I am getting a bit further...
After removing the Dell N20KJ Broadcom 57810 NIC and StarTech PEXUSB312EIC USB 3.1 controller, I am able to select the P1000 card without the 9361-8i being selected as well.
I am thinking it had something to do with the NIC card.

Now the P1000 is by itself on the bifurcated x32 left side and the 9361-8i is on the x16 side by itself.

But...after rebooting the server, The P1000 High Definition Audio Controller still shows 'Enabled / Needs reboot'
1614543353690.png

From what I understand they both need to be active for passthrough to work correctly.

Either way, I tried to add just the P1000 to the VM anyways:
1614543462339.png

On power on I get the following failure:
1614543510127.png

Not sure where it is getting the 'Device 130:0.0" from....

So I am at another hault, and not sure where to go from here.

Other things I have tried:
modifying passthru.map:

#P1000
10de 1cb1 d3d0 false
10de 0fb9 d3d0 false

trying
#P1000
10de 1cb1 bridge false
10de 0fb9 bridge false

Updating to 7.0 Update 1d

Anyone have any other ideas on what to do?
 
G

gb00s

Well-Known Member
Jul 25, 2018
1,189
600
113
Poland
First of all i am still convinced you shall not be forced to passthrough audio with your P1000. Proxmox for instance doesn't force you to share Nvidia audio with graphics and Nvidia passthrough just works. Still considering this as bug.

You mentioned you have to take off other devices to be able to passthrough other periphery independently. Are you maybe hitting limitations on your motherboard (pci lanes shares/bridges).

I am not a ESXi user by any means. So it is just sharing my thoughts.
 
R

rhyzar

New Member
Sep 2, 2020
10
0
1
Honestly I do not need the audio portion, so if that doesn't work I do not care - I just need the card to passthrough for video encode/decode.

I shouldn't be hitting any limitations - right now I only have 2 devices on the PCI bus - I think the problem was the Broadcom NIC. I saw other issues with other models with similar chipset sharing on the bus.

Any ideas why the P1000 under passthrough shows 0000:82:00.0 for a hardware address, but the VM says that 130:0.0 is not a passthrough device when booting?
 
G

gb00s

Well-Known Member
Jul 25, 2018
1,189
600
113
Poland
If you see “Enabled/Needs reboot”, did you just set both of the devices GPU & Audio to off and then off again and then reboot the vm? A colleague mentioned it is/was a VMware issue in late 2020 and should have been fixed, but still seems unresolved on some systems.

It confirms my first impression when you described the issue first. The whole behavior is not normal at all.

But why dont you try the same setup under Proxmox to see how your setup runs in there? System image backup and install Proxmox and test. If you dont like revert back from the backed up image.
 
You must log in or register to reply here.
Top Bottom