ESXI 7.0.3 - NVMe (Asus Hyper) passthrough stuck on enabled/needs reboot

[MartinM]

Hi,
I have a Supermicro X10SRi-F, 2780v4, with the ASUS Hyper M.2 X16 PCIe 4.0 X4 with two NVMe drives (Samsung 983 + Samsung 970EVO). I have the Samsung 970EVO passed through to VM and works flawlessly. However i cannot get the Samsung 983 NVMe drive to pass through.

I have set PCIE bifurcation in BIOS, have ACS control on, I see both drives correctly in BIOS and can see them both in ESXi.

However, and here is the issue, I can passthrough the Samsun970EVO without any issues (passthrough "Active"), however the Samsung 983 is stuck (Passthrough "Enabled / Needs reboot"). I can create datastore on the Samsung 983 without any issue, copy data to it and run VMs of it. Through some googling I had arrived at the following tip (Enable PCI Passthrough for Intel 82575GB NIC in VMWare ESXi) to disable ACS checking in ESXI, even though I have ACS on in BIOS, but even changing this in ESXi has no impact and the drive is still stuck on "enabled/needs reboot". Multiple reboots have not helped.

I've read that 7.0.3 is picky with drives, however I would expect the drive either not work at all, but only failling pass through seems a bit odd.

Thanks for any suggestions!

 

[DavidWJohnston]

I assisted another member a few weeks back about a nearly identical passthrough issue, also with 7.0.3.

The only thing that worked was upgrading to ESXi 8, then it worked first try. I would recommend trying this. Even if your hardware is not officially supported, ESXi 8 will run on a lot of systems.

Upgrading in general can help fix a lot of issues. I'm still on 6.7 and SecureBoot in Windows Server 2022 VMs just stopped working and vmware will never fix it. SR-IOV doesn't work right, etc... If I had kept up with the upgrades it would have saved time in the end, now I have a "deficit" that needs to be repaid with time!

 

[MartinM]

Thanks for the tip, I was a bit worried this may be the case.. At least I will move the install from USB flashdisk to a SSD, finally (per VMware recommendation).. Will report if it works, when I am physically next to the server ;-)

 

[MartinM]

@DavidWJohnston thank you for the tip, update to esxi 8 solved the issue immediately.

Thank you!

 

[MartinM]

To anyone who may find this, even with ESXi8 it is stoll wonky. Adding another HBA (lsi3008) into the mix resulted in all passthrough getting broken, three reinstalls of ESXi8 later, with identical hardware/bios/software settings made it work.

 

[vangoose]

I haven't touched my lab for a while, couple days ago I updated my ESXi from 7.0.3b to 7.0.3u and passthrough for 970 pro stopped working.
Here is the devices passthrough that still works.

 

[MartinM]

I haven't touched my lab for a while, couple days ago I updated my ESXi from 7.0.3b to 7.0.3u and passthrough for 970 pro stopped working.
Here is the devices passthrough that still works.

View attachment 30414

Is there a question? I would suggest trying what i wrote above - play with config, but if it worked it shouldnt be BIOS related, reinstall otherwise

 

[nickwalt]

FWIW, I have a new homelab based on an AMD Epyc 7452 on a Supermicro H12SSL-i motherboard with 2 x Kingston KC3000 1TB SSDs mounted on an ASUS Hyper M.2 Gen 4 Card. Also on this system are 2 x Intel 670p 2TB SSDs mounted directly on the motherboard. The 670p SSDs are passed through to a Truenas Core VM hosted on a bare-metal ESXi 8.0 installation managed by a vCenter VM. It is working flawlessly, however...

The ASUS Hyper is built for ThreadRipper and works fine on the Epyc ROME platform but the second KC3000 SSD would not get detected when Bifurcation on the x16 Slot (7) was set to the AUTO default. I had to manually set the Slot to 4x4 in the BIOS.