Enabling IOMMU for GPU passthrough in Proxmox of Nvidia M10 GPUs on Dell R730

[TheMohawkNinja]

Hello,

I have recently gotten my hands on a decommissioned Dell R730 equipped with two Nvidia M10 GPUs. This machine was used as a VMWare host that passed the vGPUs through for CAD work, so I know the hardware stack is capable, but I have been failing to get IOMMU enabled to the extent that Proxmox needs for PCI passthrough to work. When attempting to pass the vGPUs through to a VM as a raw device, I get an TASK ERROR: cannot prepare PCI pass-through, IOMMU not present error.

So far, I have done the following:

• Enabled SR-IOV, Virtualization Technology, and x2apic mode in the BIOS
• Updated /etc/default/grub with GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt" and pcie_acs_override=downstream before running update-grub
• Updated /etc/modules with the following:
  

  vfio
  vfio_iommu_type1
  vfio_pci
  vfio_virqfd

• Updated /etc/modprobe.d/kvm.conf with options kvm ignore_msrs=1
• Updated /etc/modprobe.d/blacklist.conf with blacklist nouveau and blacklist nvidia
• Updated /etc/modprobe.d/vfio.conf with options vfio-pci ids=10de:13bd disable_vga=1
• Ran update-initramfs -u -k all and rebooted

dmesg | grep -e IOMMU

[    2.068410] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffc000 IOMMU 0
[    2.068412] DMAR-IR: IOAPIC id 8 under DRHD base  0xc7ffc000 IOMMU 1
[    2.068413] DMAR-IR: IOAPIC id 9 under DRHD base  0xc7ffc000 IOMMU 1

dmesg | grep 'remapping'

[    2.068416] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    2.068680] DMAR-IR: IRQ remapping was enabled on dmar0 but we are not in kdump mode
[    2.068937] DMAR-IR: IRQ remapping was enabled on dmar1 but we are not in kdump mode
[    2.069060] DMAR-IR: Enabled IRQ remapping in x2apic mode

pvesh get /nodes/mynode/hardware/pci --pci-class-blacklist "" shows -1 for all entries in the iommugroup column.

UPDATE: Also enabled Alternate RTID (Requestor Transaction ID), as the description mentions enabling virtualization features and support for DMAR. This did not resolve the issue, but it is almost certainly a specific requirement on the R730.

 

[ttabbal]

Look in the BIOS for "vt-d". That's what intel likes to call it and I've seen it that way in some BIOS.

 

[TheMohawkNinja]

Look in the BIOS for "vt-d". That's what intel likes to call it and I've seen it that way in some BIOS.

There is no "vt-d" in the R730 BIOS (I double-checked the owner's manual as well). I'm guessing that's what the "Virtualization Technology" setting I enabled is.

 

[6c352e2e-f287-445a-9cd3-d]

maybe [SOLVED] - Enabling IOMMU on PVE6 (zfs) compared to PVE5.4 (ext4)?

Are you sure that the kernel parameters are recognized?

Virtualization Technology should enable vt-d when there's no vt-d option.

I don't know about Proxmox. Fedora shows the kernel command line in dmesg or 'journalctl -b'. I have intel_iommu=on with a Supermircro board, that causes messages like

kernel: pci 0000:ff:0b.0: Adding to iommu group 0
kernel: pci 0000:ff:0b.1: Adding to iommu group 0
kernel: pci 0000:ff:0b.2: Adding to iommu group 0
kernel: pci 0000:ff:0c.0: Adding to iommu group 1
kernel: pci 0000:ff:0c.1: Adding to iommu group 1
[...]
kernel: DMAR: Intel(R) Virtualization Technology for Directed I/O

I haven't tried it out, but I guess that means that it's working.

 

[TheMohawkNinja]

maybe [SOLVED] - Enabling IOMMU on PVE6 (zfs) compared to PVE5.4 (ext4)?

Are you sure that the kernel parameters are recognized?

Virtualization Technology is at least something like that

From what I am reading, I need to also put the intel_iommu=on iommu=pt line for GRUB_CMDLINE_LINUX. I did that and ran update-grub and rebooted, but no change.

As far as I can tell, I shouldn't need to do anything systemd-boot related given GRUB is clearly the bootloader (I can see the GRUB menu appear during boot).

I can't say I know how to confirm if the kernel parameters are being recognized.

 

[6c352e2e-f287-445a-9cd3-d]

What happens without iommu=pt? [1] says it's for Intel boards only, but that was 4 years ago. That parameter doesn't seem to be in [2]; perhaps it's specific to Proxmox.

When the gub menu appears, you should be able to press e (or F10, IIRC) to edit the command line. That would show you what's actually used. If it's missing you can add it, though it won't be persistent.

Did you make sure the modules are loaded like this guide[3] suggests?

[1]: https://www.reddit.com/r/Proxmox/comments/hhx77k [2]:The kernel’s command-line parameters — The Linux Kernel documentation
[3]: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough

 

[TheMohawkNinja]

What happens without iommu=pt? [1] says it's for Intel boards only, but that was 4 years ago. That parameter doesn't seem to be in [2]; perhaps it's specific to Proxmox.

When the gub menu appears, you should be able to press e (or F10, IIRC) to edit the command line. That would show you what's actually used. If it's missing you can add it, though it won't be persistent.

Did you make sure the modules are loaded like this guide[3] suggests?

[1]: https://www.reddit.com/r/Proxmox/comments/hhx77k [2]:The kernel’s command-line parameters — The Linux Kernel documentation
[3]: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough

That seems to have gotten me past the IOMMU error I was getting (if only the documentation would get updated...). Now it's throwing TASK ERROR: Cannot find vfio-pci module! when I try to start a VM with the M10 vGPUs attached.

That seems odd given I did make sure to add it to /etc/modules

Running modprobe vfio-pci returns:
modprobe: FATAL: Module vfio_pci not found in directory /lib/modules/6.8.12-2-pve

Running ls /lib/modules/6.8.12-2-pve returns:
kernel modules.alias.bin modules.builtin.alias.bin modules.builtin.modinfo modules.dep.bin modules.order modules.symbols zfs
modules.alias modules.builtin modules.builtin.bin modules.dep modules.devname modules.softdep modules.symbols.bin

Doing some additional research shows that the vfio_virqfd module is no longer available, so I've removed that to see if the modules actually load. Maybe that unneeded line is throwing an error that prevents any of them from loading?

After running update-initramfs -u -k all and rebooting, IOMMU was disabled again so I reverted the change and rebooted.

... and somehow I still have IOMMU disabled despite reverted the one change I made.

Also I'm back on Linux 6.5 even though it was on 6.8 when IOMMU was enabled. No idea why it loaded a different kernel all of a sudden?

EDIT: Noticed that when booting, the first volume group it finds and makes active is "pve-OLD-F760F21D". I'm guessing that's got the old Linux kernel and needs purged. Any ideas how to remove volume groups?

 

[6c352e2e-f287-445a-9cd3-d]

Yeah the guide is probably outdated. I would expect such modules being loaded automatically on demand and iommu being enabled by default in an OS solely intended to host VMs.

Volume groups from LVM? Avoid them, they are a total nightmare for they are the worst and most inflexible way to provide storage for anything.[1] I never used Proxmoxx, so I don't know how to delete them there. When you delete a volume, you may have to overwrite the start of it with something like dd. Otherwise it may show up as deleted or unused or something, but the data is still on the disk.

But why would you boot from a logical volume? Is that some special feature UEFI BIOSs have? It may help to set the right boot option without deleting volumes.


[1]: I just use files (qcow2) for VMs. Maybe it's not the fastest way to run a VM, but it's the most practical way. A file you can copy/move anywhere, compress it for backup, keep it on your NFS server and shrink or enlarge it as needed. With a VG, you're stuck to the physical disks the VG resides on, and each volume is stuck in its VG. If Proxmoxx insists on VGs simply avoid it altogether.

 

[TheMohawkNinja]

Yeah the guide is probably outdated. I would expect such modules being loaded automatically on demand and iommu being enabled by default in an OS solely intended to host VMs.

Volume groups from LVM? Avoid them, they are a total nightmare for they are the worst and most inflexible way to provide storage for anything.[1] I never used Proxmoxx, so I don't know how to delete them there. When you delete a volume, you may have to overwrite the start of it with something like dd. Otherwise it may show up as deleted or unused or something, but the data is still on the disk.

But why would you boot from a logical volume? Is that some special feature UEFI BIOSs have? It may help to set the right boot option without deleting volumes.


[1]: I just use files (qcow2) for VMs. Maybe it's not the fastest way to run a VM, but it's the most practical way. A file you can copy/move anywhere, compress it for backup, keep it on your NFS server and shrink or enlarge it as needed. With a VG, you're stuck to the physical disks the VG resides on, and each volume is stuck in its VG. If Proxmoxx insists on VGs simply avoid it altogether.

I didn't choose to put volume groups on anything, Proxmox just did it on its' own. I attached a screenshot for more context if that helps.

I think Grub is still on my IDSDM, as rearranging the boot order with my RAID1 boot first results in a failure to find bootable media on the host, even through Proxmox is installed on the RAID1. Once I finish work today, I'll disable or at least wipe the IDSDM and reinstall Proxmox to hopefully force Grub to install on the RAID1 proper.

 

[6c352e2e-f287-445a-9cd3-d]

SD cards for storage? How reliable is that?

Otherwise wiping and reinstalling seems like a good idea, though I'd just put Fedora Server on it instead of Proxmoxx. It may take some learning to set up the VMs while I get the feeling that you would not have so many issues/erros.

If you do that, keep using the hardware RAID at least for the boot drives. Enable intel_iommu in /etc/sysconfig/grub, update grub and reboot. You may want to install tuned, numad if you have 2 CPUs, and irqbalance and set the desired profile with tuned-adm. Then set up/import the VMs and you should be good to go. If you can, convert them to files instead of using LVM.

 

[TheMohawkNinja]

Now that I've got the IDSDM disabled, the RAID card booting the correct disk, and all the Proxmox specific settings set, I can actually launch VMs with the GPUs.

Granted, I have now discovered that Proxmox just outright doesn't support hardware RAID cards, so I'll need to get an HBA, but that's a different issue.

 

[ttabbal]

Hardware RAID isn't a good idea with ZFS or other software RAID setups. HBAs are cheap, that's what I would recommend. You can probably pick up an old Dell H310 cheaply. There are a bunch on ebay now.

 

[TheMohawkNinja]

Hardware RAID isn't a good idea with ZFS or other software RAID setups. HBAs are cheap, that's what I would recommend. You can probably pick up an old Dell H310 cheaply. There are a bunch on ebay now.

Yeah, I just bought a HBA330 Mini for $18, so I should be all good once that arrives and gets installed.