proxmox cannot boot if VT-d is enabled (F8 SSD Plus)

[kobi0815]

Hi,
I recently purchased a Terramaster F8 SSD Plus and added 8 NVMe drives, some of them are behind an onboard ASM2806.
I plan to use it as a home server and need pcie-passthrough, therefore VT-d is a must.
Unfortunately, I am not able to start the install process of proxmox if VT-d is enabled in bios.
When starting the installer it freezes at Waiting for /dev to be fully populated ... (see logs below)

Proxmox 8.3.2, Kernel 6.8.12-6-pve

What else I have tested:

• Installing with VT-d disabled works perfectly and everything beside pcie-passthrough works great. If I then enable VT-d the system freezes during boot.
• Starting other linux distros also does not work with VT-D enabled
• Installing Truenas Core (BSD), Windows 11 and also the preinstalled OS TOS6 work with VT-d enabled
• In Truenas Scale forums people describe the same issue. They suspect it has to do with the ASM2806 chip.
  I have also tried several of the recommended grub kernel parameters mentioned there but was not successful.

Does anyone have an idea how I can make VT-d work with this device?
Thanks in advance!


Proxmox installer boot log:

Welcome to the Proxmox VE 8.3 installer
initial setup startup
mounting proc filesystem
mounting sys filesystem
EFI boot mode detected, mounting efivars filesystem
boot comandline: BOOT_IMAGE=/boot/linux26 ro ramdisk_size=16777216 rw quiet splash=silent
loading drivers: nume wmi intel_pmc_core mac_hid acpi_pad video acpi_tad pinctrl_alderlake spi_intel_pci xhci_pci i2c_i801 acpi_cpufreq rtc_cmos efi_pstore serio_raw psmouse pcspkr intel_cstate rapl aesni_intel sha1_ssse3 sha256_ssse3 ghash_clmulni_intel polyval_clmulni crc32_pclmul crct10dif_pclmul kum_intel intel_powerclamp x86_pkg_temp_thermal intel_tcc_cooling intel_rapl_common
modprobe: ERROR: could not insert 'intel_tcc_cooling': No such device
searching for block device containing the ISO proxmox-ve-8.3-1
with ISO ID '20c7412e-a788-11ef-9ee4-4f61d2460cfb'
testing again in 1 seconds
testing device '/dev/sda' for ISO
found Proxmox VE ISO
preparing installer mount points and working environment
switching root from initrd to actual installation system
Starting Proxmox installation
EFI boot mode detected, mounting efivars filesystem
Installing additional hardware drivers
Starting hotplug events dispatcher: systemd-udeud.
Synthesizing the initial hotplug events (subsystems) ... done.
Synthesizing the initial hotplug events (devices) ... done.
Waiting for /dev to be fully populated ...

Proxmox installer detailed boot log:

nume 0000:02:00.0: platform quirk: setting simple suspend
nume 0000:03:00.0: platform quirk: setting simple suspend
nume nume2: pci function 0000:07:00.0
nume nume3: pci function 0000:08:00.0
nume nume0: pci function 0000:06:00.0
nume nume4: pci function 0000:09:00.0
nume nume1: pci function 0000:03:00.0
nume nume5: pci function 0000:02:00.0
nume 0000:0a:00.0: platform quirk: setting simple suspend
nume 0000:0b:00.0: platform quirk: setting simple suspend
nume nume6: pci function 0000:0b:00.0
nume nume7: pci function 0000:0a:00.0
nume nume5: 8/0/0 default/read/poll queues
nume nume4: 8/0/0 default/read/poll queues
nume nume1: 8/0/0 default/read/poll queues
nume nume3: 8/0/0 default/read/poll queues
nume nume2: 8/0/0 default/read/poll queues
mume nume0: 8/0/0 default/read/poll queues
mume nume6: 8/0/0 default/read/poll queues
nume nume7: 8/0/0 default/read/poll queues
nume1n1: p1 p2 p3 p4
nume4n1: p1 p2 p3 p4
nume3n1: p1 p2 p3 p4
nume2n1: p1 p2 p3 p4
nume0n1: p1 p2 p3 p4
nume5n1: p1 p2 p3
nume7n1: p1 p2 p3 p4
nume6n1: p1 p2 p3 p4

lspci:

00:00.0 Host bridge: Intel Corporation Device 4617
00:02.0 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
00:14.0 USB controller: Intel Corporation Alder Lake-N PCH USB 3.2 xHCI Host Controller
00:14.2 RAM memory: Intel Corporation Alder Lake-N PCH Shared SRAM
00:16.0 Communication controller: Intel Corporation Alder Lake-N PCH HECI Controller
00:1c.0 PCI bridge: Intel Corporation Device 54b8
00:1c.3 PCI bridge: Intel Corporation Device 54bb
00:1c.6 PCI bridge: Intel Corporation Device 54be
00:1d.0 PCI bridge: Intel Corporation Alder Lake-N PCI Express Root Port
00:1d.2 PCI bridge: Intel Corporation Alder Lake-N PCI Express Root Port
00:1d.3 PCI bridge: Intel Corporation Alder Lake-N PCI Express Root Port
00:1f.0 ISA bridge: Intel Corporation Alder Lake-N PCH eSPI Controller
00:1f.3 Audio device: Intel Corporation Alder Lake-N PCH High Definition Audio Controller
00:1f.4 SMBus: Intel Corporation Alder Lake-N SMBus
00:1f.5 Serial bus controller: Intel Corporation Alder Lake-N SPI (flash) Controller
01:00.0 Ethernet controller: Aquantia Corp. AQC113C NBase-T/IEEE 802.3an Ethernet Controller [Marvell Scalable mGig] (rev 03)
02:00.0 Non-Volatile memory controller: Sandisk Corp Device 5049 (rev 01)
03:00.0 Non-Volatile memory controller: Sandisk Corp Device 5049 (rev 01)
04:00.0 PCI bridge: ASMedia Technology Inc. ASM2806 4-Port PCIe x2 Gen3 Packet Switch (rev 01)
05:00.0 PCI bridge: ASMedia Technology Inc. ASM2806 4-Port PCIe x2 Gen3 Packet Switch (rev 01)
05:02.0 PCI bridge: ASMedia Technology Inc. ASM2806 4-Port PCIe x2 Gen3 Packet Switch (rev 01)
05:06.0 PCI bridge: ASMedia Technology Inc. ASM2806 4-Port PCIe x2 Gen3 Packet Switch (rev 01)
05:0e.0 PCI bridge: ASMedia Technology Inc. ASM2806 4-Port PCIe x2 Gen3 Packet Switch (rev 01)
06:00.0 Non-Volatile memory controller: Sandisk Corp Device 5049 (rev 01)
07:00.0 Non-Volatile memory controller: Sandisk Corp Device 5049 (rev 01)
08:00.0 Non-Volatile memory controller: Sandisk Corp Device 5049 (rev 01)
09:00.0 Non-Volatile memory controller: Sandisk Corp Device 5049 (rev 01)
0a:00.0 Non-Volatile memory controller: Sandisk Corp Device 5049 (rev 01)
0b:00.0 Non-Volatile memory controller: Sandisk Corp Device 5049 (rev 01)

ASM2806: lspci -s 04:00.0 -vvv

04:00.0 PCI bridge: ASMedia Technology Inc. ASM2806 4-Port PCIe x2 Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        Bus: primary=04, secondary=05, subordinate=09, sec-latency=0
        I/O behind bridge: 0000f000-00000fff [disabled] [32-bit]
        Memory behind bridge: 80a00000-80dfffff [size=4M] [32-bit]
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled] [64-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [80] Express (v2) Upstream Port, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 25W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; Disabled- CommClk+
                        ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x2 (downgraded)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-
                         AtomicOpsCap: Routing-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: EgressBlck-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis+
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [c0] Subsystem: Device 0000:0000
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 14, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 05000001 0000000f 05080005 00000000
        Capabilities: [140 v1] Power Budgeting <?>
        Capabilities: [160 v1] Latency Tolerance Reporting
                Max snoop latency: 3145728ns
                Max no snoop latency: 3145728ns
        Capabilities: [180 v1] Multicast
                McastCap: MaxGroups 64, ECRCRegen-
                McastCtl: NumGroups 1, Enable-
                McastBAR: IndexPos 0, BaseAddr 0000000000000000
                McastReceiveVec:      0000000000000000
                McastBlockAllVec:     0000000000000000
                McastBlockUntransVec: 0000000000000000
                McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000
        Capabilities: [1c0 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [200 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+ L1_PM_Substates+
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                L1SubCtl2:
        Kernel driver in use: pcieport

 

[Captain Lukey]

I would recommend resetting / upgrading the BIOS and setting it to its default settings, as I have frequently encountered issues with VT-d and certain NVMe devices. Additionally, consider disabling the Render C-State (RC6) in the kernel, as it can cause problems when VT-d is enabled. I also agree that there are numerous discussions regarding the ASM2806 chip." Have you tried the device on its own? We had the same message on "Waiting for /dev to be fully populated" on some SM motherboards

Question - BIOS hangs during boot if Intel Virtualization is turned on

 

[Apachez]

Also verify that UEFI is enabled (you can disable CSM all together) and have secure boot disabled.

 

[kobi0815]

Hi guys,
I think i found a solution for the VT-D problem. If you add the boot option pci=nommconf to grub you can boot every OS including TrueNAS Scale.
I tried with TrueNAS, Ubuntu, Debian and Proxmox.
I was not sure if it is stable so I tested it for about 3 weeks now and i can report that so far everything works perfect.

I am now running proxmox with a TrueNAS vm on top. The NVMe’s are handed over to TrueNAS via PCIe passthrough (VT-D, iommu) so truenas has access to the whole disks and not just virtual disks. So also SMART checks are possible.
By the way this solution is not limitted to 4x m.2 nvme’s, I run it fully populated with 8.

This really opened the device for me to use it as a homeserver/nas as i planned for. I was even able to passthrough the GPU to Jellyfin, immich and nextcloud for transcoding and basic AI workloads.

It took me a long time to find the issue and solve it and I hope it helps some of you!

 

[Stef68]

Hello,

Thank you for your message which gives me great hope

I've installed UNRAID and, of course, by turning “off” the “VT-D” option in the BIOS, everything works.

Now that I've read your message, I absolutely must test this.

My question: do you still have a positive opinion of your discovery?

And, if you have an idea, where should I put your parameter? in the "go" file (which is on the boot USB key)? and should I reactivate the "VT-D" option in the BIOS (which would seem logical to me)

Thanks for your advice

EDIT: my "FS8 plus" is currently at 6 NVME

EDIT 2 : by reactivating "VT-D" in the BIOS -> it no longer works

 

[kobi0815]

Hello,

Thank you for your message which gives me great hope

I've installed UNRAID and, of course, by turning “off” the “VT-D” option in the BIOS, everything works.

Now that I've read your message, I absolutely must test this.

My question: do you still have a positive opinion of your discovery?

And, if you have an idea, where should I put your parameter? in the "go" file (which is on the boot USB key)? and should I reactivate the "VT-D" option in the BIOS (which would seem logical to me)

Thanks for your advice

EDIT: my "FS8 plus" is currently at 6 NVME

EDIT 2 : by reactivating "VT-D" in the BIOS -> it no longer works

View attachment 43203

Hi,
sorry i have missed your message. I still have a very positive opinion about this. I still have the same proxmox install running, I am very satisfied with the device and had no issues so far.

The option has to be put in your boot loader. I personally don't use unraid but I just checked online and unraid seems to use Syslinux instead of Grub. You will have to configure it there. Yes, after this is configured you should be able to boot with VT-D enabled.

You can also test this option on your device by booting into a live image of a linux distro and adding the setting to grub while booting.

 

[Stef68]

Thank you for your message. I tried to put it in what I think is the UNRAID “grub” (but not sure), and once put, UNRAID would not boot, removing VT-D again in BIOS, it would boot again.

So my experience with VT-D enabled in the BIOS is not very positive, unfortunately.

 

[kobi0815]

Thank you for your message. I tried to put it in what I think is the UNRAID “grub” (but not sure), and once put, UNRAID would not boot, removing VT-D again in BIOS, it would boot again.

So my experience with VT-D enabled in the BIOS is not very positive, unfortunately.

Sorry to hear that...
As mentioned before I am not using unraid so it would be best if you ask how to do this in the unraid forum.

Please be warned that I am not an expert (just used some google) and that the steps below could potentially prevent your unraid from booting.
That said, I think this is how it should be done:

1. open the syslinux config file /boot/syslinux/syslinux.cfg
2. Find the boot entry you usually use (probably labeled as Unraid OS) and add pci=nommconf to the append line.
   example:

   label Unraid OS
   menu default
   kernel /bzimage
       append initrd=/bzroot pci=nommconf

3. save the file and reboot

 

[IceFyre]

Thanks for your help on this kobi0815

You mentioned that have this fix working on Proxmox. Have you got the exact code for this?

I did try adding it to grub but it was a no go with VT-D enabled.

 

[Stef68]

Sorry to hear that...
As mentioned before I am not using unraid so it would be best if you ask how to do this in the unraid forum.

Please be warned that I am not an expert (just used some google) and that the steps below could potentially prevent your unraid from booting.
That said, I think this is how it should be done:

1. open the syslinux config file /boot/syslinux/syslinux.cfg
2. Find the boot entry you usually use (probably labeled as Unraid OS) and add pci=nommconf to the append line.
   example:

   label Unraid OS
   menu default
   kernel /bzimage
       append initrd=/bzroot pci=nommconf

3. save the file and reboot

Hi "kobi0815",

Thank you very much for your help and your research to help me. My English is not good, which limits my understanding, unfortunately.

I put this in my “Syslinux Configuration” .

kernel /bzimage
append initrd=/bzroot
pci=nommconf

This way, my UNRAID starts up without a hitch, but if I also activate “VT-D” in the BIOS, that's it, it doesn't start up anymore and reboots endlessly.

If you have an idea, I'll take it

EDIT :
Would putting rather be the key ? :

kernel /bzimage
append initrd=/bzroot pci=nommconf

To do this new test, I'd have to stop my server and move it to put a screen on it...

EDIT 2 :
I couldn't wait to do this test, so I turned off my server, modified the BIOS (thus enabling “VT-D”), rebooted and this time it went all the way and I can now connect without any problem with this last change in the UNRAID “Syslinux Configuration” parameters.

My mistake was putting this parameter on a new line! -> solution -> "append initrd=/bzroot pci=nommconf"

THANK YOU !!!!!!!!!! for your help and valuable advice. I hope other people will be able to read this exchange and find a solution.

 

[kobi0815]

Thanks for your help on this kobi0815

You mentioned that have this fix working on Proxmox. Have you got the exact code for this?

I did try adding it to grub but it was a no go with VT-D enabled.

Here is my /etc/default/grub:

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="pci=nommconf pcie_acs_override=downstream intel_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX=""

Don't forget to use update-grub before you reboot.

pci=nommconf -> necessary to be able to boot with VT-D enabled
pcie_acs_override=downstream, intel_iommu=on, iommu=pt -> for IOMMU and IOMMU isolation to work.
see: PCI Passthrough - Proxmox VE

 

[IceFyre]

Here is my /etc/default/grub:

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="pci=nommconf pcie_acs_override=downstream intel_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX=""

Don't forget to use update-grub before you reboot.

pci=nommconf -> necessary to be able to boot with VT-D enabled
pcie_acs_override=downstream, intel_iommu=on, iommu=pt -> for IOMMU and IOMMU isolation to work.
see: PCI Passthrough - Proxmox VE

You are a champion thanks. I will try this and report back.