Hi all,
I've updated Proxmox and seems with the new kernels I'm having issues to start the SR-IOV on my Intel I-350T as I'm used too.
For what I see in the logs, the igb module starts properly and enable the VF that I've set up. It complains about "PF still in reset state. Is the PF interface up?" and seems after enabling all the VF, iommu removes all of them.
Any suggestion of what I'm doing wrong and how I could fix this?
Thanks
igb config
root@proxmox:~# cat /etc/modprobe.d/igb.conf
options igb max_vfs=0,7,0,0 MDD=0
modules config
root@proxmox:~# cat /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
igb start on boot
Mar 21 07:52:11 proxmox kernel: [ 1.508333] igb: 0000:04:00.1: igb_validate_option: max_vfs - SR-IOV VF devices set to 7
Mar 21 07:52:11 proxmox kernel: [ 1.616960] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
Mar 21 07:52:11 proxmox kernel: [ 1.616961] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
Mar 21 07:52:11 proxmox kernel: [ 1.617096] igbvf 0000:04:12.1: enabling device (0000 -> 0002)
Mar 21 07:52:11 proxmox kernel: [ 1.618308] igbvf 0000:04:12.1: PF still in reset state. Is the PF interface up?
Mar 21 07:52:11 proxmox kernel: [ 1.618310] igbvf 0000:04:12.1: Assigning random MAC address.
Mar 21 07:52:11 proxmox kernel: [ 1.619635] igbvf 0000:04:12.1: Intel(R) I350 Virtual Function
Mar 21 07:52:11 proxmox kernel: [ 1.619637] igbvf 0000:04:12.1: Address: 8e:58:17:16:4b:99
Mar 21 07:52:11 proxmox kernel: [ 1.619873] igbvf 0000:04:12.5: enabling device (0000 -> 0002)
Mar 21 07:52:11 proxmox kernel: [ 1.621092] igbvf 0000:04:12.5: PF still in reset state. Is the PF interface up?
Mar 21 07:52:11 proxmox kernel: [ 1.621093] igbvf 0000:04:12.5: Assigning random MAC address.
Mar 21 07:52:11 proxmox kernel: [ 1.622398] igbvf 0000:04:12.5: Intel(R) I350 Virtual Function
Mar 21 07:52:11 proxmox kernel: [ 1.622399] igbvf 0000:04:12.5: Address: de:32:e5:2c:0a:a8
Mar 21 07:52:11 proxmox kernel: [ 1.622628] igbvf 0000:04:13.1: enabling device (0000 -> 0002)
Mar 21 07:52:11 proxmox kernel: [ 1.623834] igbvf 0000:04:13.1: PF still in reset state. Is the PF interface up?
Mar 21 07:52:11 proxmox kernel: [ 1.623835] igbvf 0000:04:13.1: Assigning random MAC address.
Mar 21 07:52:11 proxmox kernel: [ 1.625144] igbvf 0000:04:13.1: Intel(R) I350 Virtual Function
Mar 21 07:52:11 proxmox kernel: [ 1.625145] igbvf 0000:04:13.1: Address: e2:fe:4f:86:4f:90
Mar 21 07:52:11 proxmox kernel: [ 1.625173] igb 0000:04:00.1: SR-IOV has been enabled: configure port VLANs to keep your VFs secure
Mar 21 07:52:11 proxmox kernel: [ 1.625230] igb 0000:04:00.1: Failed to initialize MSI-X interrupts. Falling back to MSI interrupts.
Mar 21 07:52:11 proxmox kernel: [ 3.292047] igb 0000:04:00.1: IOV Disabled
Mar 21 07:52:11 proxmox kernel: [ 3.516464] igb 0000:04:00.1: added PHC on eth1
Mar 21 07:52:11 proxmox kernel: [ 3.516465] igb 0000:04:00.1: Intel(R) Gigabit Ethernet Linux Driver
Mar 21 07:52:11 proxmox kernel: [ 3.516467] igb 0000:04:00.1: eth1: (PCIe:5.0GT/s:Width x4)
Mar 21 07:52:11 proxmox kernel: [ 3.516468] igb 0000:04:00.1 eth1: MAC: 0c:c4:7a:ac:8d:bf
Mar 21 07:52:11 proxmox kernel: [ 3.516540] igb 0000:04:00.1: eth1: PBA No: 050B00-000
Mar 21 07:52:11 proxmox kernel: [ 3.528053] igb 0000:04:00.1: LRO is disabled
Mar 21 07:52:11 proxmox kernel: [ 3.528056] igb 0000:04:00.1: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)
iommu adding and removing the VF
Mar 21 07:52:11 proxmox kernel: [ 1.616205] iommu: Adding device 0000:04:10.1 to group 35
Mar 21 07:52:11 proxmox kernel: [ 1.616427] iommu: Adding device 0000:04:10.5 to group 36
Mar 21 07:52:11 proxmox kernel: [ 1.616614] iommu: Adding device 0000:04:11.1 to group 37
Mar 21 07:52:11 proxmox kernel: [ 1.616846] iommu: Adding device 0000:04:11.5 to group 38
Mar 21 07:52:11 proxmox kernel: [ 1.617064] iommu: Adding device 0000:04:12.1 to group 39
Mar 21 07:52:11 proxmox kernel: [ 1.619837] iommu: Adding device 0000:04:12.5 to group 40
Mar 21 07:52:11 proxmox kernel: [ 1.622595] iommu: Adding device 0000:04:13.1 to group 41
Mar 21 07:52:11 proxmox kernel: [ 1.625329] iommu: Removing device 0000:04:10.1 from group 35
Mar 21 07:52:11 proxmox kernel: [ 1.625378] iommu: Removing device 0000:04:10.5 from group 36
Mar 21 07:52:11 proxmox kernel: [ 1.625422] iommu: Removing device 0000:04:11.1 from group 37
Mar 21 07:52:11 proxmox kernel: [ 1.625463] iommu: Removing device 0000:04:11.5 from group 38
Mar 21 07:52:11 proxmox kernel: [ 1.680027] iommu: Removing device 0000:04:12.1 from group 39
Mar 21 07:52:11 proxmox kernel: [ 1.728092] iommu: Removing device 0000:04:12.5 from group 40
Mar 21 07:52:11 proxmox kernel: [ 1.768123] iommu: Removing device 0000:04:13.1 from group 41
I've updated Proxmox and seems with the new kernels I'm having issues to start the SR-IOV on my Intel I-350T as I'm used too.
For what I see in the logs, the igb module starts properly and enable the VF that I've set up. It complains about "PF still in reset state. Is the PF interface up?" and seems after enabling all the VF, iommu removes all of them.
Any suggestion of what I'm doing wrong and how I could fix this?
Thanks
igb config
root@proxmox:~# cat /etc/modprobe.d/igb.conf
options igb max_vfs=0,7,0,0 MDD=0
modules config
root@proxmox:~# cat /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
igb start on boot
Mar 21 07:52:11 proxmox kernel: [ 1.508333] igb: 0000:04:00.1: igb_validate_option: max_vfs - SR-IOV VF devices set to 7
Mar 21 07:52:11 proxmox kernel: [ 1.616960] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
Mar 21 07:52:11 proxmox kernel: [ 1.616961] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
Mar 21 07:52:11 proxmox kernel: [ 1.617096] igbvf 0000:04:12.1: enabling device (0000 -> 0002)
Mar 21 07:52:11 proxmox kernel: [ 1.618308] igbvf 0000:04:12.1: PF still in reset state. Is the PF interface up?
Mar 21 07:52:11 proxmox kernel: [ 1.618310] igbvf 0000:04:12.1: Assigning random MAC address.
Mar 21 07:52:11 proxmox kernel: [ 1.619635] igbvf 0000:04:12.1: Intel(R) I350 Virtual Function
Mar 21 07:52:11 proxmox kernel: [ 1.619637] igbvf 0000:04:12.1: Address: 8e:58:17:16:4b:99
Mar 21 07:52:11 proxmox kernel: [ 1.619873] igbvf 0000:04:12.5: enabling device (0000 -> 0002)
Mar 21 07:52:11 proxmox kernel: [ 1.621092] igbvf 0000:04:12.5: PF still in reset state. Is the PF interface up?
Mar 21 07:52:11 proxmox kernel: [ 1.621093] igbvf 0000:04:12.5: Assigning random MAC address.
Mar 21 07:52:11 proxmox kernel: [ 1.622398] igbvf 0000:04:12.5: Intel(R) I350 Virtual Function
Mar 21 07:52:11 proxmox kernel: [ 1.622399] igbvf 0000:04:12.5: Address: de:32:e5:2c:0a:a8
Mar 21 07:52:11 proxmox kernel: [ 1.622628] igbvf 0000:04:13.1: enabling device (0000 -> 0002)
Mar 21 07:52:11 proxmox kernel: [ 1.623834] igbvf 0000:04:13.1: PF still in reset state. Is the PF interface up?
Mar 21 07:52:11 proxmox kernel: [ 1.623835] igbvf 0000:04:13.1: Assigning random MAC address.
Mar 21 07:52:11 proxmox kernel: [ 1.625144] igbvf 0000:04:13.1: Intel(R) I350 Virtual Function
Mar 21 07:52:11 proxmox kernel: [ 1.625145] igbvf 0000:04:13.1: Address: e2:fe:4f:86:4f:90
Mar 21 07:52:11 proxmox kernel: [ 1.625173] igb 0000:04:00.1: SR-IOV has been enabled: configure port VLANs to keep your VFs secure
Mar 21 07:52:11 proxmox kernel: [ 1.625230] igb 0000:04:00.1: Failed to initialize MSI-X interrupts. Falling back to MSI interrupts.
Mar 21 07:52:11 proxmox kernel: [ 3.292047] igb 0000:04:00.1: IOV Disabled
Mar 21 07:52:11 proxmox kernel: [ 3.516464] igb 0000:04:00.1: added PHC on eth1
Mar 21 07:52:11 proxmox kernel: [ 3.516465] igb 0000:04:00.1: Intel(R) Gigabit Ethernet Linux Driver
Mar 21 07:52:11 proxmox kernel: [ 3.516467] igb 0000:04:00.1: eth1: (PCIe:5.0GT/s:Width x4)
Mar 21 07:52:11 proxmox kernel: [ 3.516468] igb 0000:04:00.1 eth1: MAC: 0c:c4:7a:ac:8d:bf
Mar 21 07:52:11 proxmox kernel: [ 3.516540] igb 0000:04:00.1: eth1: PBA No: 050B00-000
Mar 21 07:52:11 proxmox kernel: [ 3.528053] igb 0000:04:00.1: LRO is disabled
Mar 21 07:52:11 proxmox kernel: [ 3.528056] igb 0000:04:00.1: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)
iommu adding and removing the VF
Mar 21 07:52:11 proxmox kernel: [ 1.616205] iommu: Adding device 0000:04:10.1 to group 35
Mar 21 07:52:11 proxmox kernel: [ 1.616427] iommu: Adding device 0000:04:10.5 to group 36
Mar 21 07:52:11 proxmox kernel: [ 1.616614] iommu: Adding device 0000:04:11.1 to group 37
Mar 21 07:52:11 proxmox kernel: [ 1.616846] iommu: Adding device 0000:04:11.5 to group 38
Mar 21 07:52:11 proxmox kernel: [ 1.617064] iommu: Adding device 0000:04:12.1 to group 39
Mar 21 07:52:11 proxmox kernel: [ 1.619837] iommu: Adding device 0000:04:12.5 to group 40
Mar 21 07:52:11 proxmox kernel: [ 1.622595] iommu: Adding device 0000:04:13.1 to group 41
Mar 21 07:52:11 proxmox kernel: [ 1.625329] iommu: Removing device 0000:04:10.1 from group 35
Mar 21 07:52:11 proxmox kernel: [ 1.625378] iommu: Removing device 0000:04:10.5 from group 36
Mar 21 07:52:11 proxmox kernel: [ 1.625422] iommu: Removing device 0000:04:11.1 from group 37
Mar 21 07:52:11 proxmox kernel: [ 1.625463] iommu: Removing device 0000:04:11.5 from group 38
Mar 21 07:52:11 proxmox kernel: [ 1.680027] iommu: Removing device 0000:04:12.1 from group 39
Mar 21 07:52:11 proxmox kernel: [ 1.728092] iommu: Removing device 0000:04:12.5 from group 40
Mar 21 07:52:11 proxmox kernel: [ 1.768123] iommu: Removing device 0000:04:13.1 from group 41