Proxmox - I350 and sr-iov issues

xavierd

New Member
Jun 6, 2016
6
1
3
36
Hi all,

I've updated Proxmox and seems with the new kernels I'm having issues to start the SR-IOV on my Intel I-350T as I'm used too.

For what I see in the logs, the igb module starts properly and enable the VF that I've set up. It complains about "PF still in reset state. Is the PF interface up?" and seems after enabling all the VF, iommu removes all of them.

Any suggestion of what I'm doing wrong and how I could fix this?

Thanks



igb config
root@proxmox:~# cat /etc/modprobe.d/igb.conf
options igb max_vfs=0,7,0,0 MDD=0

modules config
root@proxmox:~# cat /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd


igb start on boot
Mar 21 07:52:11 proxmox kernel: [ 1.508333] igb: 0000:04:00.1: igb_validate_option: max_vfs - SR-IOV VF devices set to 7
Mar 21 07:52:11 proxmox kernel: [ 1.616960] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
Mar 21 07:52:11 proxmox kernel: [ 1.616961] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
Mar 21 07:52:11 proxmox kernel: [ 1.617096] igbvf 0000:04:12.1: enabling device (0000 -> 0002)
Mar 21 07:52:11 proxmox kernel: [ 1.618308] igbvf 0000:04:12.1: PF still in reset state. Is the PF interface up?
Mar 21 07:52:11 proxmox kernel: [ 1.618310] igbvf 0000:04:12.1: Assigning random MAC address.
Mar 21 07:52:11 proxmox kernel: [ 1.619635] igbvf 0000:04:12.1: Intel(R) I350 Virtual Function
Mar 21 07:52:11 proxmox kernel: [ 1.619637] igbvf 0000:04:12.1: Address: 8e:58:17:16:4b:99
Mar 21 07:52:11 proxmox kernel: [ 1.619873] igbvf 0000:04:12.5: enabling device (0000 -> 0002)
Mar 21 07:52:11 proxmox kernel: [ 1.621092] igbvf 0000:04:12.5: PF still in reset state. Is the PF interface up?
Mar 21 07:52:11 proxmox kernel: [ 1.621093] igbvf 0000:04:12.5: Assigning random MAC address.
Mar 21 07:52:11 proxmox kernel: [ 1.622398] igbvf 0000:04:12.5: Intel(R) I350 Virtual Function
Mar 21 07:52:11 proxmox kernel: [ 1.622399] igbvf 0000:04:12.5: Address: de:32:e5:2c:0a:a8
Mar 21 07:52:11 proxmox kernel: [ 1.622628] igbvf 0000:04:13.1: enabling device (0000 -> 0002)
Mar 21 07:52:11 proxmox kernel: [ 1.623834] igbvf 0000:04:13.1: PF still in reset state. Is the PF interface up?
Mar 21 07:52:11 proxmox kernel: [ 1.623835] igbvf 0000:04:13.1: Assigning random MAC address.
Mar 21 07:52:11 proxmox kernel: [ 1.625144] igbvf 0000:04:13.1: Intel(R) I350 Virtual Function
Mar 21 07:52:11 proxmox kernel: [ 1.625145] igbvf 0000:04:13.1: Address: e2:fe:4f:86:4f:90
Mar 21 07:52:11 proxmox kernel: [ 1.625173] igb 0000:04:00.1: SR-IOV has been enabled: configure port VLANs to keep your VFs secure
Mar 21 07:52:11 proxmox kernel: [ 1.625230] igb 0000:04:00.1: Failed to initialize MSI-X interrupts. Falling back to MSI interrupts.
Mar 21 07:52:11 proxmox kernel: [ 3.292047] igb 0000:04:00.1: IOV Disabled
Mar 21 07:52:11 proxmox kernel: [ 3.516464] igb 0000:04:00.1: added PHC on eth1
Mar 21 07:52:11 proxmox kernel: [ 3.516465] igb 0000:04:00.1: Intel(R) Gigabit Ethernet Linux Driver
Mar 21 07:52:11 proxmox kernel: [ 3.516467] igb 0000:04:00.1: eth1: (PCIe:5.0GT/s:Width x4)
Mar 21 07:52:11 proxmox kernel: [ 3.516468] igb 0000:04:00.1 eth1: MAC: 0c:c4:7a:ac:8d:bf
Mar 21 07:52:11 proxmox kernel: [ 3.516540] igb 0000:04:00.1: eth1: PBA No: 050B00-000
Mar 21 07:52:11 proxmox kernel: [ 3.528053] igb 0000:04:00.1: LRO is disabled
Mar 21 07:52:11 proxmox kernel: [ 3.528056] igb 0000:04:00.1: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)

iommu adding and removing the VF
Mar 21 07:52:11 proxmox kernel: [ 1.616205] iommu: Adding device 0000:04:10.1 to group 35
Mar 21 07:52:11 proxmox kernel: [ 1.616427] iommu: Adding device 0000:04:10.5 to group 36
Mar 21 07:52:11 proxmox kernel: [ 1.616614] iommu: Adding device 0000:04:11.1 to group 37
Mar 21 07:52:11 proxmox kernel: [ 1.616846] iommu: Adding device 0000:04:11.5 to group 38
Mar 21 07:52:11 proxmox kernel: [ 1.617064] iommu: Adding device 0000:04:12.1 to group 39
Mar 21 07:52:11 proxmox kernel: [ 1.619837] iommu: Adding device 0000:04:12.5 to group 40
Mar 21 07:52:11 proxmox kernel: [ 1.622595] iommu: Adding device 0000:04:13.1 to group 41
Mar 21 07:52:11 proxmox kernel: [ 1.625329] iommu: Removing device 0000:04:10.1 from group 35
Mar 21 07:52:11 proxmox kernel: [ 1.625378] iommu: Removing device 0000:04:10.5 from group 36
Mar 21 07:52:11 proxmox kernel: [ 1.625422] iommu: Removing device 0000:04:11.1 from group 37
Mar 21 07:52:11 proxmox kernel: [ 1.625463] iommu: Removing device 0000:04:11.5 from group 38
Mar 21 07:52:11 proxmox kernel: [ 1.680027] iommu: Removing device 0000:04:12.1 from group 39
Mar 21 07:52:11 proxmox kernel: [ 1.728092] iommu: Removing device 0000:04:12.5 from group 40
Mar 21 07:52:11 proxmox kernel: [ 1.768123] iommu: Removing device 0000:04:13.1 from group 41
 

MiniKnight

Well-Known Member
Mar 30, 2012
2,987
892
113
NYC
Is SRIOV enabled in BIOS? I'd normally troubleshoot this as being a BIOS type or driver issue.
 

xavierd

New Member
Jun 6, 2016
6
1
3
36
Thanks for the help !!

So I checked the Bios and indeed SR-IOV wasn't enable. I have enable SR-IOV and set ASPM on auto, wondering I'f this option didn't changed while I upgraded my bios.

However after booting the same issue, I've tried to reload the driver but however same thing. I can see that it does seems something disables IOV... could it be related to the failure with MSI-X ?

Mar 22 21:50:42 proxmox kernel: [ 1.694286] igbvf 0000:04:13.1: PF still in reset state. Is the PF interface up?
Mar 22 21:50:42 proxmox kernel: [ 1.694287] igbvf 0000:04:13.1: Assigning random MAC address.
Mar 22 21:50:42 proxmox kernel: [ 1.699419] igbvf 0000:04:13.1: Intel(R) I350 Virtual Function
Mar 22 21:50:42 proxmox kernel: [ 1.699421] igbvf 0000:04:13.1: Address: 6a:0e:42:01:7d:f3
Mar 22 21:50:42 proxmox kernel: [ 1.699467] igb 0000:04:00.1: SR-IOV has been enabled: configure port VLANs to keep your VFs secure
Mar 22 21:50:42 proxmox kernel: [ 1.699914] igb 0000:04:00.1: Failed to initialize MSI-X interrupts. Falling back to MSI interrupts.
Mar 22 21:50:42 proxmox kernel: [ 3.516019] igb 0000:04:00.1: IOV Disabled
Mar 22 21:50:42 proxmox kernel: [ 3.740425] igb 0000:04:00.1: added PHC on eth1
 

xavierd

New Member
Jun 6, 2016
6
1
3
36
Been working on this and found what seems a bug on the current kernel.

Updating on the new kernel 4.15.3, seems to solve the issue
proxmox-ve: 5.1-42 (running kernel: 4.15.3-1-pve)

4.15 based test kernel for PVE 5.x available

Looking at the boot it does seems I don't have more problems with a failure to initialize MSI-X which my understanding is needed for sr-iov on igb?

Thanks for all the help
 
  • Like
Reactions: Patrick