Proxmox passthrough to Freenas

Jon

Member
Feb 28, 2016
55
16
8
38
Hi I was wondering if any experts could tell me what I was doing wrong?

Note: I am aware that passthrough is an experimental feature in Proxmox VE and that some purist will not like the FREENAS running in a VM. But trying to make a all in one test system ala Patrick.

Following this I managed to passthrough one device successfully:
Pci passthrough - Proxmox VE

But as soon as I add a second device the VM will not start? Running into this issue on two systhems so preaty sure I am doing something wrong?

Code:
oot@pveD1518:~# lspci
00:00.0 Host bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 (rev 03)
00:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 03)
00:01.1 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 03)
00:02.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 03)
00:02.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 03)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 03)
00:03.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 03)
00:05.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management (rev 03)
00:05.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug (rev 03)
00:05.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors (rev 03)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
00:16.1 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #2 (rev 04)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d5)
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C224 Series Chipset Family Server Standard SKU LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)
01:00.0 Non-Volatile memory controller: Intel Corporation Device 2522
02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] (rev 02)
03:00.0 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 0
03:00.1 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 1
03:00.2 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 2
03:00.3 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 3
04:00.0 Ethernet controller: Intel Corporation Ethernet Connection X552 10 GbE SFP+
04:00.1 Ethernet controller: Intel Corporation Ethernet Connection X552 10 GbE SFP+
06:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
07:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
08:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
09:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
0a:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
ff:0b.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 03)
ff:0b.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 03)
ff:0b.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 03)
ff:0b.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link Debug (rev 03)
ff:0c.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0c.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0c.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0c.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0f.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0f.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0f.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:10.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 03)
ff:10.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 03)
ff:10.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 03)
ff:10.6 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 03)
ff:10.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 03)
ff:12.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 (rev 03)
ff:12.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 (rev 03)
ff:13.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 03)
ff:13.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 03)
ff:13.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target AddressDecoder (rev 03)
ff:13.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target AddressDecoder (rev 03)
ff:13.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target AddressDecoder (rev 03)
ff:13.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target AddressDecoder (rev 03)
ff:13.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Broadcast (rev 03)
ff:13.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast (rev 03)
ff:14.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Thermal Control (rev 03)
ff:14.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Thermal Control (rev 03)
ff:14.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Error (rev 03)
ff:14.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Error (rev 03)
ff:14.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 03)
ff:14.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 03)
ff:14.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 03)
ff:14.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 03)
ff:15.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Thermal Control (rev 03)
ff:15.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Thermal Control (rev 03)
ff:15.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Error (rev 03)
ff:15.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Error (rev 03)
ff:1e.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
ff:1e.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
ff:1e.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
ff:1e.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
ff:1e.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
ff:1f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
ff:1f.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
root@pveD1518:~#
Trying to pass 3 device.
  • Sas controller hostpci0: 02:00
  • Intel Optane NVME hostpci1: 01:00
  • Mellanox Technologies MT26448 [ConnectX EN 10GigE hostpci2: 06:00.0
Using this config bellow I can attach and use all 3 devices but just one at a time? If I attach more then one the VM will not start
Code:
bootdisk: sata0
cores: 4
ide2: local:iso/FreeNAS-11.1-U5.iso,media=cdrom
memory: 32768
name: FreenasD1518
net0: e1000=5E:EA:64:3D:6A:DD,bridge=vmbr0
numa: 0
ostype: other
sata0: local-zfs:vm-101-disk-1,size=16G
scsihw: virtio-scsi-pci
smbios1: uuid=6df66d93-fc4f-476f-bf99-b2c1abe2fd79
sockets: 1
hostpci0: 02:00
Code:
kvm: -device vfio-pci,host=06:00.0,id=hostpci1,bus=pci.0,addr=0x11: vfio error: 0000:06:00.0: failed to setup INTx fd: Operation not permitted
TASK ERROR: start failed: command '/usr/bin/kvm -id 101 -name FreenasD1518 -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=6df66d93-fc4f-476f-bf99-b2c1abe2fd79' -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga std -vnc unix:/var/run/qemu-server/101.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 32768 -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'vfio-pci,host=02:00.0,id=hostpci0,bus=pci.0,addr=0x10' -device 'vfio-pci,host=06:00.0,id=hostpci1,bus=pci.0,addr=0x11' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:5c9bbd5791d6' -drive 'file=/var/lib/vz/template/iso/FreeNAS-11.1-U5.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7' -drive 'file=/dev/zvol/rpool/data/vm-101-disk-1,if=none,id=drive-sata0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'ide-drive,bus=ahci0.0,drive=drive-sata0,id=sata0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=5E:EA:64:3D:6A:DD,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: exit code 1
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,935
4,891
113
Being fair, I would never do FreeNAS for storage inside Proxmox VE. Proxmox has ZFS support so adding the complexity of the VM and passthrough is scarier to me.

On the passthrough, is VT-d enabled in BIOS? It often is turned off by default so good to start with that.
 
  • Like
Reactions: MikeWebb

Jon

Member
Feb 28, 2016
55
16
8
38
Thanks for the reply Patrick

VT-d is enabled I can pass one device at a time just have issues if I try more then one in the same VM?

About the ZFS and Freenas thing would you say that this is no longuer a thing or there is now a better way with newer versions of promox?(ZFS with a GUI)?
https://www.servethehome.com/the-proxmox-ve-kvm-based-all-in-one-freenas/

Will look into what firmware and OEM I actually have.
 
Last edited:

brianmc

New Member
Jun 25, 2018
29
7
3
@Jon
It should be noted that running FreeNAS or other ZFS storage in a VM is far from a leading practice, and there are many reasons why you would not do this in a production cluster. Still, for learning purposes, we thought this may be an interesting idea. The eventual goal is to simply use it as a backup target. Again, it would be much better to simply create a zpool with the two WD Red drives in the base Debian OS then create an iSCSI target to access the drives.
 

Jon

Member
Feb 28, 2016
55
16
8
38
Yes this is what I am doing playing around for fun with passthrough.

So on the subject :
  • after a app-get update
  • playing around in the VM conf file some more
  • deactivating int 19 catch in the bios for some of my card
  • Also added (hostpci0: 02:00.0,rombar=off) on some card
  • I can now pass the nvme drive + sas controller withough error
  • Can even pass a 3rd device in a onboard nic that is not used by promox
  • Only the mellanox card is now giving me issues.

Found this on the forum that could maiby help on the mellanox card issue:
https://forums.servethehome.com/index.php?threads/mellanox-unhappy-w-vtd-passthru-in-vsphere.12173/

So for now will move the mellanox card to windows PC and flash it. Seam to be lucky and have original mellanox card.

Code:
\Program Files\Mellanox\WinMFT>flint -d mt26448_pci_cr0 -i fw291200.bin burn

    Note: Both the image file and the flash contain a ROM image.
          Select "yes" to use the ROM from the given image file.
          Select "no" to keep the existing ROM in the flash
    Current ROM info on flash: type=PXE version=3.3.400 proto=VPI
    ROM info from image file : type=PXE version=3.3.400 proto=VPI

 Use the ROM from the image file ? (y/n) [n] : y

    Current FW version on flash:  2.9.1000
    New FW version:               2.9.1200

Burning FS2 FW image without signatures - OK
Restoring signature                     - OK
 
Last edited:

Rand__

Well-Known Member
Mar 6, 2014
4,551
901
113
What Optane are you trying to pass through? 900p is still not working with Freenas (or NappIt) in a virtualized setup; have you tried without using it?
 

Jon

Member
Feb 28, 2016
55
16
8
38
What Optane are you trying to pass through? 900p is still not working with Freenas (or NappIt) in a virtualized setup; have you tried without using it?
Just more info

The optane drive now works. Main issue is from the Mellanox NIC and the error in question was in promox prior to even VM Bios startup so VM OS probably not related. It will do the same thing if I pass it to PFSense or Windows VM.
 

vl1969

Active Member
Feb 5, 2014
611
68
28
was never able to do this setup on my hardware. not 100% supported I guess.

but I am currently running an all-in-one single server Proxmox VE setup using containers and bind-mounts of ZFS.
so far love it. but YMMV
 

Zedicus

New Member
Jul 12, 2018
8
5
3
the amount of devices you can pass through is a combined limit of 2 things, the physical host CPU and the selected Vcpu of the VM.

i ran a freenas VM with 2 passed through controllers for a long time with no issue. but there is a max amount of things you can pass through per VM and it is usually around 2 or 3 devices.

for hba passtrough, one of the best things to do is use the firmware flash tool for the card to 0 out the cards BIOS, (leave the firmware section alone though). then proxmox will not attempt to import the bios to the VM, and for freenas and an hba the bios is not needed.

use intel nics, they make life on linux easier.

also of note, i had a freenas VM running nice for years, and one day a proxmox update significantly altered how passthrough was handled and overnight i started writing garbage to the freenas VMs zfs arrays. so while it can be very stable, it can be a severe trouble shooting headache when things go wrong. and the freenas board will at best ignore your virtualization questions, and at worst delete them entirely.
 
  • Like
Reactions: leebo_28

MikeWebb

Member
Jan 28, 2018
87
21
8
50
Does that Mellanox card do SR-IOV? Would be easier to just conna VF to the VM then pass a whole device through