Proxmox - I350 and sr-iov issues

Discussion in 'Linux Admins, Storage and Virtualization' started by xavierd, Mar 20, 2018.

  1. xavierd

    xavierd New Member

    Joined:
    Jun 6, 2016
    Messages:
    6
    Likes Received:
    1
    Hi all,

    I've updated Proxmox and seems with the new kernels I'm having issues to start the SR-IOV on my Intel I-350T as I'm used too.

    For what I see in the logs, the igb module starts properly and enable the VF that I've set up. It complains about "PF still in reset state. Is the PF interface up?" and seems after enabling all the VF, iommu removes all of them.

    Any suggestion of what I'm doing wrong and how I could fix this?

    Thanks



    igb config
    root@proxmox:~# cat /etc/modprobe.d/igb.conf
    options igb max_vfs=0,7,0,0 MDD=0

    modules config
    root@proxmox:~# cat /etc/modules
    vfio
    vfio_iommu_type1
    vfio_pci
    vfio_virqfd


    igb start on boot
    Mar 21 07:52:11 proxmox kernel: [ 1.508333] igb: 0000:04:00.1: igb_validate_option: max_vfs - SR-IOV VF devices set to 7
    Mar 21 07:52:11 proxmox kernel: [ 1.616960] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
    Mar 21 07:52:11 proxmox kernel: [ 1.616961] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
    Mar 21 07:52:11 proxmox kernel: [ 1.617096] igbvf 0000:04:12.1: enabling device (0000 -> 0002)
    Mar 21 07:52:11 proxmox kernel: [ 1.618308] igbvf 0000:04:12.1: PF still in reset state. Is the PF interface up?
    Mar 21 07:52:11 proxmox kernel: [ 1.618310] igbvf 0000:04:12.1: Assigning random MAC address.
    Mar 21 07:52:11 proxmox kernel: [ 1.619635] igbvf 0000:04:12.1: Intel(R) I350 Virtual Function
    Mar 21 07:52:11 proxmox kernel: [ 1.619637] igbvf 0000:04:12.1: Address: 8e:58:17:16:4b:99
    Mar 21 07:52:11 proxmox kernel: [ 1.619873] igbvf 0000:04:12.5: enabling device (0000 -> 0002)
    Mar 21 07:52:11 proxmox kernel: [ 1.621092] igbvf 0000:04:12.5: PF still in reset state. Is the PF interface up?
    Mar 21 07:52:11 proxmox kernel: [ 1.621093] igbvf 0000:04:12.5: Assigning random MAC address.
    Mar 21 07:52:11 proxmox kernel: [ 1.622398] igbvf 0000:04:12.5: Intel(R) I350 Virtual Function
    Mar 21 07:52:11 proxmox kernel: [ 1.622399] igbvf 0000:04:12.5: Address: de:32:e5:2c:0a:a8
    Mar 21 07:52:11 proxmox kernel: [ 1.622628] igbvf 0000:04:13.1: enabling device (0000 -> 0002)
    Mar 21 07:52:11 proxmox kernel: [ 1.623834] igbvf 0000:04:13.1: PF still in reset state. Is the PF interface up?
    Mar 21 07:52:11 proxmox kernel: [ 1.623835] igbvf 0000:04:13.1: Assigning random MAC address.
    Mar 21 07:52:11 proxmox kernel: [ 1.625144] igbvf 0000:04:13.1: Intel(R) I350 Virtual Function
    Mar 21 07:52:11 proxmox kernel: [ 1.625145] igbvf 0000:04:13.1: Address: e2:fe:4f:86:4f:90
    Mar 21 07:52:11 proxmox kernel: [ 1.625173] igb 0000:04:00.1: SR-IOV has been enabled: configure port VLANs to keep your VFs secure
    Mar 21 07:52:11 proxmox kernel: [ 1.625230] igb 0000:04:00.1: Failed to initialize MSI-X interrupts. Falling back to MSI interrupts.
    Mar 21 07:52:11 proxmox kernel: [ 3.292047] igb 0000:04:00.1: IOV Disabled
    Mar 21 07:52:11 proxmox kernel: [ 3.516464] igb 0000:04:00.1: added PHC on eth1
    Mar 21 07:52:11 proxmox kernel: [ 3.516465] igb 0000:04:00.1: Intel(R) Gigabit Ethernet Linux Driver
    Mar 21 07:52:11 proxmox kernel: [ 3.516467] igb 0000:04:00.1: eth1: (PCIe:5.0GT/s:Width x4)
    Mar 21 07:52:11 proxmox kernel: [ 3.516468] igb 0000:04:00.1 eth1: MAC: 0c:c4:7a:ac:8d:bf
    Mar 21 07:52:11 proxmox kernel: [ 3.516540] igb 0000:04:00.1: eth1: PBA No: 050B00-000
    Mar 21 07:52:11 proxmox kernel: [ 3.528053] igb 0000:04:00.1: LRO is disabled
    Mar 21 07:52:11 proxmox kernel: [ 3.528056] igb 0000:04:00.1: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)

    iommu adding and removing the VF
    Mar 21 07:52:11 proxmox kernel: [ 1.616205] iommu: Adding device 0000:04:10.1 to group 35
    Mar 21 07:52:11 proxmox kernel: [ 1.616427] iommu: Adding device 0000:04:10.5 to group 36
    Mar 21 07:52:11 proxmox kernel: [ 1.616614] iommu: Adding device 0000:04:11.1 to group 37
    Mar 21 07:52:11 proxmox kernel: [ 1.616846] iommu: Adding device 0000:04:11.5 to group 38
    Mar 21 07:52:11 proxmox kernel: [ 1.617064] iommu: Adding device 0000:04:12.1 to group 39
    Mar 21 07:52:11 proxmox kernel: [ 1.619837] iommu: Adding device 0000:04:12.5 to group 40
    Mar 21 07:52:11 proxmox kernel: [ 1.622595] iommu: Adding device 0000:04:13.1 to group 41
    Mar 21 07:52:11 proxmox kernel: [ 1.625329] iommu: Removing device 0000:04:10.1 from group 35
    Mar 21 07:52:11 proxmox kernel: [ 1.625378] iommu: Removing device 0000:04:10.5 from group 36
    Mar 21 07:52:11 proxmox kernel: [ 1.625422] iommu: Removing device 0000:04:11.1 from group 37
    Mar 21 07:52:11 proxmox kernel: [ 1.625463] iommu: Removing device 0000:04:11.5 from group 38
    Mar 21 07:52:11 proxmox kernel: [ 1.680027] iommu: Removing device 0000:04:12.1 from group 39
    Mar 21 07:52:11 proxmox kernel: [ 1.728092] iommu: Removing device 0000:04:12.5 from group 40
    Mar 21 07:52:11 proxmox kernel: [ 1.768123] iommu: Removing device 0000:04:13.1 from group 41
     
    #1
  2. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,633
    Likes Received:
    4,586
  3. MiniKnight

    MiniKnight Well-Known Member

    Joined:
    Mar 30, 2012
    Messages:
    2,956
    Likes Received:
    864
    Is SRIOV enabled in BIOS? I'd normally troubleshoot this as being a BIOS type or driver issue.
     
    #3
  4. xavierd

    xavierd New Member

    Joined:
    Jun 6, 2016
    Messages:
    6
    Likes Received:
    1
    Thanks for the help !!

    So I checked the Bios and indeed SR-IOV wasn't enable. I have enable SR-IOV and set ASPM on auto, wondering I'f this option didn't changed while I upgraded my bios.

    However after booting the same issue, I've tried to reload the driver but however same thing. I can see that it does seems something disables IOV... could it be related to the failure with MSI-X ?

    Mar 22 21:50:42 proxmox kernel: [ 1.694286] igbvf 0000:04:13.1: PF still in reset state. Is the PF interface up?
    Mar 22 21:50:42 proxmox kernel: [ 1.694287] igbvf 0000:04:13.1: Assigning random MAC address.
    Mar 22 21:50:42 proxmox kernel: [ 1.699419] igbvf 0000:04:13.1: Intel(R) I350 Virtual Function
    Mar 22 21:50:42 proxmox kernel: [ 1.699421] igbvf 0000:04:13.1: Address: 6a:0e:42:01:7d:f3
    Mar 22 21:50:42 proxmox kernel: [ 1.699467] igb 0000:04:00.1: SR-IOV has been enabled: configure port VLANs to keep your VFs secure
    Mar 22 21:50:42 proxmox kernel: [ 1.699914] igb 0000:04:00.1: Failed to initialize MSI-X interrupts. Falling back to MSI interrupts.
    Mar 22 21:50:42 proxmox kernel: [ 3.516019] igb 0000:04:00.1: IOV Disabled
    Mar 22 21:50:42 proxmox kernel: [ 3.740425] igb 0000:04:00.1: added PHC on eth1
     
    #4
  5. xavierd

    xavierd New Member

    Joined:
    Jun 6, 2016
    Messages:
    6
    Likes Received:
    1
    Been working on this and found what seems a bug on the current kernel.

    Updating on the new kernel 4.15.3, seems to solve the issue
    proxmox-ve: 5.1-42 (running kernel: 4.15.3-1-pve)

    4.15 based test kernel for PVE 5.x available

    Looking at the boot it does seems I don't have more problems with a failure to initialize MSI-X which my understanding is needed for sr-iov on igb?

    Thanks for all the help
     
    #5
    Patrick likes this.
Similar Threads: Proxmox I350
Forum Title Date
Linux Admins, Storage and Virtualization Networking Services don’t auto-start on boot up - Dell R720 w/ proxmox (Warning: Beginner User) Dec 6, 2019
Linux Admins, Storage and Virtualization Proxmox environment feedback Nov 4, 2019
Linux Admins, Storage and Virtualization Selecting a Proxmox Master Node for Cluster Oct 3, 2019
Linux Admins, Storage and Virtualization Running pfSense virtualized but serving the Proxmox as well Sep 20, 2019
Linux Admins, Storage and Virtualization Proxmox server with separate unraid storage box, is it possible? Aug 19, 2019

Share This Page