I have a few ConnectX-3 (MCX 311, ethernet) and am looking to activate SR-IOV on one of them on my Asrock Z370 Extreme 4 motherboard.
I have a few questions for which I cannot find answers (or my Google-FU is lacking):
1- When I try to check the configuration I get:
[root@lair ~]# mlxconfig -d /dev/mst/mt4099_pciconf0 q
Device #1:
----------
Device type: ConnectX3
Device: /dev/mst/mt4099_pciconf0
Configurations: Next Boot
-E- Failed to query device current configuration
The Bios configration tool cannot set SRIOV either.
When I dump the ini file, it does not contain the sriov_en setting.
Is it possible to get the card to support SR-IOV?
2- What is the probability of my motherboard supporting SR-IOV at all?
Centos 7.6 reports the following for the IOMMU settings:
[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-957.10.1.el7.x86_64 root=/dev/mapper/lair1-root ro crashkernel=auto rd.lvm.lv=lair1/root rd.lvm.lv=lair1/swap LANG=en_US.UTF-8 intel_iommu=on iommu=pt
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-957.10.1.el7.x86_64 root=/dev/mapper/lair1-root ro crashkernel=auto rd.lvm.lv=lair1/root rd.lvm.lv=lair1/swap LANG=en_US.UTF-8 intel_iommu=on iommu=pt
[ 0.689043] iommu: Adding device 0000:00:00.0 to group 0
[ 0.689058] iommu: Adding device 0000:00:01.0 to group 1
[ 0.689067] iommu: Adding device 0000:00:02.0 to group 2
[ 0.689081] iommu: Adding device 0000:00:14.0 to group 3
[ 0.689089] iommu: Adding device 0000:00:14.2 to group 3
[ 0.689100] iommu: Adding device 0000:00:16.0 to group 4
[ 0.689109] iommu: Adding device 0000:00:17.0 to group 5
[ 0.689122] iommu: Adding device 0000:00:1b.0 to group 6
[ 0.689134] iommu: Adding device 0000:00:1b.4 to group 7
[ 0.689145] iommu: Adding device 0000:00:1c.0 to group 8
[ 0.689157] iommu: Adding device 0000:00:1c.1 to group 9
[ 0.689170] iommu: Adding device 0000:00:1c.4 to group 10
[ 0.689182] iommu: Adding device 0000:00:1c.7 to group 11
[ 0.689193] iommu: Adding device 0000:00:1d.0 to group 12
[ 0.689211] iommu: Adding device 0000:00:1f.0 to group 13
[ 0.689220] iommu: Adding device 0000:00:1f.2 to group 13
[ 0.689229] iommu: Adding device 0000:00:1f.3 to group 13
[ 0.689238] iommu: Adding device 0000:00:1f.4 to group 13
[ 0.689247] iommu: Adding device 0000:00:1f.6 to group 14
[ 0.689253] iommu: Adding device 0000:01:00.0 to group 1
[ 0.689266] iommu: Adding device 0000:02:00.0 to group 15
[ 0.689277] iommu: Adding device 0000:6f:00.0 to group 16
[ 0.689289] iommu: Adding device 0000:70:00.0 to group 17
and lspci reports:
00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 07)
00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 630 (Desktop)
00:14.0 USB controller: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller
00:14.2 Signal processing controller: Intel Corporation 200 Series PCH Thermal Subsystem
00:16.0 Communication controller: Intel Corporation 200 Series PCH CSME HECI #1
00:17.0 SATA controller: Intel Corporation 200 Series PCH SATA controller [AHCI mode]
00:1b.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #17 (rev f0)
00:1b.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #21 (rev f0)
00:1c.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #1 (rev f0)
00:1c.1 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #2 (rev f0)
00:1c.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5 (rev f0)
00:1c.7 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #8 (rev f0)
00:1d.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #9 (rev f0)
00:1f.0 ISA bridge: Intel Corporation Z370 Chipset LPC/eSPI Controller
00:1f.2 Memory controller: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller
00:1f.3 Audio device: Intel Corporation 200 Series PCH HD Audio
00:1f.4 SMBus: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V
01:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
6f:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
70:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller
I am pretty new to this stuff, so please be gentle....
/Louis
I have a few questions for which I cannot find answers (or my Google-FU is lacking):
1- When I try to check the configuration I get:
[root@lair ~]# mlxconfig -d /dev/mst/mt4099_pciconf0 q
Device #1:
----------
Device type: ConnectX3
Device: /dev/mst/mt4099_pciconf0
Configurations: Next Boot
-E- Failed to query device current configuration
The Bios configration tool cannot set SRIOV either.
When I dump the ini file, it does not contain the sriov_en setting.
Is it possible to get the card to support SR-IOV?
2- What is the probability of my motherboard supporting SR-IOV at all?
Centos 7.6 reports the following for the IOMMU settings:
[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-957.10.1.el7.x86_64 root=/dev/mapper/lair1-root ro crashkernel=auto rd.lvm.lv=lair1/root rd.lvm.lv=lair1/swap LANG=en_US.UTF-8 intel_iommu=on iommu=pt
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-957.10.1.el7.x86_64 root=/dev/mapper/lair1-root ro crashkernel=auto rd.lvm.lv=lair1/root rd.lvm.lv=lair1/swap LANG=en_US.UTF-8 intel_iommu=on iommu=pt
[ 0.689043] iommu: Adding device 0000:00:00.0 to group 0
[ 0.689058] iommu: Adding device 0000:00:01.0 to group 1
[ 0.689067] iommu: Adding device 0000:00:02.0 to group 2
[ 0.689081] iommu: Adding device 0000:00:14.0 to group 3
[ 0.689089] iommu: Adding device 0000:00:14.2 to group 3
[ 0.689100] iommu: Adding device 0000:00:16.0 to group 4
[ 0.689109] iommu: Adding device 0000:00:17.0 to group 5
[ 0.689122] iommu: Adding device 0000:00:1b.0 to group 6
[ 0.689134] iommu: Adding device 0000:00:1b.4 to group 7
[ 0.689145] iommu: Adding device 0000:00:1c.0 to group 8
[ 0.689157] iommu: Adding device 0000:00:1c.1 to group 9
[ 0.689170] iommu: Adding device 0000:00:1c.4 to group 10
[ 0.689182] iommu: Adding device 0000:00:1c.7 to group 11
[ 0.689193] iommu: Adding device 0000:00:1d.0 to group 12
[ 0.689211] iommu: Adding device 0000:00:1f.0 to group 13
[ 0.689220] iommu: Adding device 0000:00:1f.2 to group 13
[ 0.689229] iommu: Adding device 0000:00:1f.3 to group 13
[ 0.689238] iommu: Adding device 0000:00:1f.4 to group 13
[ 0.689247] iommu: Adding device 0000:00:1f.6 to group 14
[ 0.689253] iommu: Adding device 0000:01:00.0 to group 1
[ 0.689266] iommu: Adding device 0000:02:00.0 to group 15
[ 0.689277] iommu: Adding device 0000:6f:00.0 to group 16
[ 0.689289] iommu: Adding device 0000:70:00.0 to group 17
and lspci reports:
00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 07)
00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 630 (Desktop)
00:14.0 USB controller: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller
00:14.2 Signal processing controller: Intel Corporation 200 Series PCH Thermal Subsystem
00:16.0 Communication controller: Intel Corporation 200 Series PCH CSME HECI #1
00:17.0 SATA controller: Intel Corporation 200 Series PCH SATA controller [AHCI mode]
00:1b.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #17 (rev f0)
00:1b.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #21 (rev f0)
00:1c.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #1 (rev f0)
00:1c.1 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #2 (rev f0)
00:1c.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5 (rev f0)
00:1c.7 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #8 (rev f0)
00:1d.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #9 (rev f0)
00:1f.0 ISA bridge: Intel Corporation Z370 Chipset LPC/eSPI Controller
00:1f.2 Memory controller: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller
00:1f.3 Audio device: Intel Corporation 200 Series PCH HD Audio
00:1f.4 SMBus: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V
01:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
6f:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
70:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller
I am pretty new to this stuff, so please be gentle....
/Louis