Proxmox passthrough to Freenas

Discussion in 'Linux Admins, Storage and Virtualization' started by Jon, Jul 14, 2018.

  1. Jon

    Jon New Member

    Joined:
    Feb 28, 2016
    Messages:
    18
    Likes Received:
    0
    Hi I was wondering if any experts could tell me what I was doing wrong?

    Note: I am aware that passthrough is an experimental feature in Proxmox VE and that some purist will not like the FREENAS running in a VM. But trying to make a all in one test system ala Patrick.

    Following this I managed to passthrough one device successfully:
    Pci passthrough - Proxmox VE

    But as soon as I add a second device the VM will not start? Running into this issue on two systhems so preaty sure I am doing something wrong?

    Code:
    oot@pveD1518:~# lspci
    00:00.0 Host bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 (rev 03)
    00:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 03)
    00:01.1 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 03)
    00:02.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 03)
    00:02.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 03)
    00:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 03)
    00:03.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 03)
    00:05.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management (rev 03)
    00:05.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug (rev 03)
    00:05.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors (rev 03)
    00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
    00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
    00:16.1 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #2 (rev 04)
    00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
    00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
    00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d5)
    00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
    00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
    00:1f.0 ISA bridge: Intel Corporation C224 Series Chipset Family Server Standard SKU LPC Controller (rev 05)
    00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
    00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
    00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)
    01:00.0 Non-Volatile memory controller: Intel Corporation Device 2522
    02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] (rev 02)
    03:00.0 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 0
    03:00.1 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 1
    03:00.2 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 2
    03:00.3 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 3
    04:00.0 Ethernet controller: Intel Corporation Ethernet Connection X552 10 GbE SFP+
    04:00.1 Ethernet controller: Intel Corporation Ethernet Connection X552 10 GbE SFP+
    06:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
    07:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
    08:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
    09:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
    0a:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
    ff:0b.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 03)
    ff:0b.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 03)
    ff:0b.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 03)
    ff:0b.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link Debug (rev 03)
    ff:0c.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
    ff:0c.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
    ff:0c.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
    ff:0c.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
    ff:0f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
    ff:0f.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
    ff:0f.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
    ff:0f.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
    ff:10.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 03)
    ff:10.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 03)
    ff:10.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 03)
    ff:10.6 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 03)
    ff:10.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 03)
    ff:12.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 (rev 03)
    ff:12.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 (rev 03)
    ff:13.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 03)
    ff:13.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 03)
    ff:13.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target AddressDecoder (rev 03)
    ff:13.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target AddressDecoder (rev 03)
    ff:13.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target AddressDecoder (rev 03)
    ff:13.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target AddressDecoder (rev 03)
    ff:13.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Broadcast (rev 03)
    ff:13.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast (rev 03)
    ff:14.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Thermal Control (rev 03)
    ff:14.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Thermal Control (rev 03)
    ff:14.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Error (rev 03)
    ff:14.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Error (rev 03)
    ff:14.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 03)
    ff:14.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 03)
    ff:14.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 03)
    ff:14.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 03)
    ff:15.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Thermal Control (rev 03)
    ff:15.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Thermal Control (rev 03)
    ff:15.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Error (rev 03)
    ff:15.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Error (rev 03)
    ff:1e.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
    ff:1e.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
    ff:1e.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
    ff:1e.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
    ff:1e.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
    ff:1f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
    ff:1f.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
    root@pveD1518:~#
    
    Trying to pass 3 device.
    • Sas controller hostpci0: 02:00
    • Intel Optane NVME hostpci1: 01:00
    • Mellanox Technologies MT26448 [ConnectX EN 10GigE hostpci2: 06:00.0
    Using this config bellow I can attach and use all 3 devices but just one at a time? If I attach more then one the VM will not start
    Code:
    bootdisk: sata0
    cores: 4
    ide2: local:iso/FreeNAS-11.1-U5.iso,media=cdrom
    memory: 32768
    name: FreenasD1518
    net0: e1000=5E:EA:64:3D:6A:DD,bridge=vmbr0
    numa: 0
    ostype: other
    sata0: local-zfs:vm-101-disk-1,size=16G
    scsihw: virtio-scsi-pci
    smbios1: uuid=6df66d93-fc4f-476f-bf99-b2c1abe2fd79
    sockets: 1
    hostpci0: 02:00
    Code:
    kvm: -device vfio-pci,host=06:00.0,id=hostpci1,bus=pci.0,addr=0x11: vfio error: 0000:06:00.0: failed to setup INTx fd: Operation not permitted
    TASK ERROR: start failed: command '/usr/bin/kvm -id 101 -name FreenasD1518 -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=6df66d93-fc4f-476f-bf99-b2c1abe2fd79' -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga std -vnc unix:/var/run/qemu-server/101.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 32768 -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'vfio-pci,host=02:00.0,id=hostpci0,bus=pci.0,addr=0x10' -device 'vfio-pci,host=06:00.0,id=hostpci1,bus=pci.0,addr=0x11' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:5c9bbd5791d6' -drive 'file=/var/lib/vz/template/iso/FreeNAS-11.1-U5.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7' -drive 'file=/dev/zvol/rpool/data/vm-101-disk-1,if=none,id=drive-sata0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'ide-drive,bus=ahci0.0,drive=drive-sata0,id=sata0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=5E:EA:64:3D:6A:DD,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: exit code 1
     
    #1
  2. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,112
    Likes Received:
    4,069
    Being fair, I would never do FreeNAS for storage inside Proxmox VE. Proxmox has ZFS support so adding the complexity of the VM and passthrough is scarier to me.

    On the passthrough, is VT-d enabled in BIOS? It often is turned off by default so good to start with that.
     
    #2
    MikeWebb likes this.
  3. Jon

    Jon New Member

    Joined:
    Feb 28, 2016
    Messages:
    18
    Likes Received:
    0
    Thanks for the reply Patrick

    VT-d is enabled I can pass one device at a time just have issues if I try more then one in the same VM?

    About the ZFS and Freenas thing would you say that this is no longuer a thing or there is now a better way with newer versions of promox?(ZFS with a GUI)?
    https://www.servethehome.com/the-proxmox-ve-kvm-based-all-in-one-freenas/

    Will look into what firmware and OEM I actually have.
     
    #3
    Last edited: Jul 15, 2018
  4. brianmc

    brianmc New Member

    Joined:
    Jun 25, 2018
    Messages:
    29
    Likes Received:
    7
    @Jon
     
    #4
  5. Jon

    Jon New Member

    Joined:
    Feb 28, 2016
    Messages:
    18
    Likes Received:
    0
    Yes this is what I am doing playing around for fun with passthrough.

    So on the subject :
    • after a app-get update
    • playing around in the VM conf file some more
    • deactivating int 19 catch in the bios for some of my card
    • Also added (hostpci0: 02:00.0,rombar=off) on some card
    • I can now pass the nvme drive + sas controller withough error
    • Can even pass a 3rd device in a onboard nic that is not used by promox
    • Only the mellanox card is now giving me issues.

    Found this on the forum that could maiby help on the mellanox card issue:
    https://forums.servethehome.com/index.php?threads/mellanox-unhappy-w-vtd-passthru-in-vsphere.12173/

    So for now will move the mellanox card to windows PC and flash it. Seam to be lucky and have original mellanox card.

    Code:
    \Program Files\Mellanox\WinMFT>flint -d mt26448_pci_cr0 -i fw291200.bin burn
    
        Note: Both the image file and the flash contain a ROM image.
              Select "yes" to use the ROM from the given image file.
              Select "no" to keep the existing ROM in the flash
        Current ROM info on flash: type=PXE version=3.3.400 proto=VPI
        ROM info from image file : type=PXE version=3.3.400 proto=VPI
    
     Use the ROM from the image file ? (y/n) [n] : y
    
        Current FW version on flash:  2.9.1000
        New FW version:               2.9.1200
    
    Burning FS2 FW image without signatures - OK
    Restoring signature                     - OK
    
     
    #5
    Last edited: Jul 15, 2018
  6. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,490
    Likes Received:
    325
    What Optane are you trying to pass through? 900p is still not working with Freenas (or NappIt) in a virtualized setup; have you tried without using it?
     
    #6
  7. Jon

    Jon New Member

    Joined:
    Feb 28, 2016
    Messages:
    18
    Likes Received:
    0
    Just more info

    The optane drive now works. Main issue is from the Mellanox NIC and the error in question was in promox prior to even VM Bios startup so VM OS probably not related. It will do the same thing if I pass it to PFSense or Windows VM.
     
    #7
  8. vl1969

    vl1969 Active Member

    Joined:
    Feb 5, 2014
    Messages:
    518
    Likes Received:
    53
    was never able to do this setup on my hardware. not 100% supported I guess.

    but I am currently running an all-in-one single server Proxmox VE setup using containers and bind-mounts of ZFS.
    so far love it. but YMMV
     
    #8
    MikeWebb and nthu9280 like this.
  9. Zedicus

    Zedicus New Member

    Joined:
    Jul 12, 2018
    Messages:
    7
    Likes Received:
    5
    the amount of devices you can pass through is a combined limit of 2 things, the physical host CPU and the selected Vcpu of the VM.

    i ran a freenas VM with 2 passed through controllers for a long time with no issue. but there is a max amount of things you can pass through per VM and it is usually around 2 or 3 devices.

    for hba passtrough, one of the best things to do is use the firmware flash tool for the card to 0 out the cards BIOS, (leave the firmware section alone though). then proxmox will not attempt to import the bios to the VM, and for freenas and an hba the bios is not needed.

    use intel nics, they make life on linux easier.

    also of note, i had a freenas VM running nice for years, and one day a proxmox update significantly altered how passthrough was handled and overnight i started writing garbage to the freenas VMs zfs arrays. so while it can be very stable, it can be a severe trouble shooting headache when things go wrong. and the freenas board will at best ignore your virtualization questions, and at worst delete them entirely.
     
    #9
    leebo_28 likes this.
  10. MikeWebb

    MikeWebb Member

    Joined:
    Jan 28, 2018
    Messages:
    65
    Likes Received:
    15
    Does that Mellanox card do SR-IOV? Would be easier to just conna VF to the VM then pass a whole device through
     
    #10
Similar Threads: Proxmox passthrough
Forum Title Date
Linux Admins, Storage and Virtualization OpenIndiana Hipster under Proxmox with SAS HBA PCI passthrough Jul 1, 2018
Linux Admins, Storage and Virtualization ESXi under Proxmox + pci-passthrough CX3 VF Dec 18, 2017
Linux Admins, Storage and Virtualization Proxmox - Single nic passthrough to vm from quad nic Apr 28, 2016
Linux Admins, Storage and Virtualization Advice needed: Proxmox vs vSphere Oct 15, 2018
Linux Admins, Storage and Virtualization Proxmox ZFS Pool GUI Sep 29, 2018

Share This Page