Mellanox ConnectX-4 LX EN SR-IOV on ESXi 6.7U3

vangoose

Active Member
May 21, 2019
263
69
28
Canada
Very strange. Motherboard is SM X9SRL-F, BIOS setting is right. I'm going to try a different slot.

SR-IOV is working
Code:
esxcli network sriovnic list
Name    PCI Device    Driver   Link  Speed  Duplex  MAC Address         MTU  Description
------  ------------  -------  ----  -----  ------  -----------------  ----  ---------------------------------------------------------------------
vmnic4  0000:06:00.2  qedentv  Up    10000  Full    34:80:0d:0d:8f:c2  9000  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter
vmnic5  0000:06:00.3  qedentv  Up    10000  Full    34:80:0d:0d:8f:c3  9000  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter
vmnic6  0000:06:00.4  qedentv  Up    10000  Full    34:80:0d:0d:8f:c4  9000  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter
vmnic7  0000:06:00.5  qedentv  Up    10000  Full    34:80:0d:0d:8f:c5  9000  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter
vmnic8  0000:06:00.6  qedentv  Up    10000  Full    34:80:0d:0d:8f:c6  1500  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter
vmnic9  0000:06:00.7  qedentv  Up    10000  Full    34:80:0d:0d:8f:c7  1500  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter

SR-IOV is turned on in firmware
Code:
/opt/mellanox/bin/mlxconfig -d mt4117_pciconf0  q
Device #1:
----------

Device type:    ConnectX4LX
Name:           MCX4121A-ACA_Ax
Description:    ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
Device:         mt4117_pciconf0

Configurations:                              Next Boot
         MEMIC_BAR_SIZE                      0
         MEMIC_SIZE_LIMIT                    _256KB(1)
         FLEX_PARSER_PROFILE_ENABLE          0
         FLEX_IPV4_OVER_VXLAN_PORT           0
         ROCE_NEXT_PROTOCOL                  254
         NON_PREFETCHABLE_PF_BAR             False(0)
         VF_VPD_ENABLE                       False(0)
         STRICT_VF_MSIX_NUM                  False(0)
         VF_NODNIC_ENABLE                    False(0)
         NUM_OF_VFS                          8
         SRIOV_EN                            True(1)

Same card is working in H11 board.
Code:
esxcli network sriovnic list
Name    PCI Device    Driver      Link  Speed  Duplex  MAC Address         MTU  Description
------  ------------  ----------  ----  -----  ------  -----------------  ----  -----------
vmnic4  0000:85:00.4  qedentv     Up    10000  Full    34:80:0d:0d:5b:08  1500  QLogic Corp. QLogic FastLinQ QL41xxx 1/10/25 GbE Ethernet Adapter
vmnic5  0000:85:00.5  qedentv     Up    10000  Full    34:80:0d:0d:5b:09  1500  QLogic Corp. QLogic FastLinQ QL41xxx 1/10/25 GbE Ethernet Adapter
vmnic6  0000:85:00.6  qedentv     Up    10000  Full    34:80:0d:0d:5b:0a  1500  QLogic Corp. QLogic FastLinQ QL41xxx 1/10/25 GbE Ethernet Adapter
vmnic7  0000:85:00.7  qedentv     Up    10000  Full    34:80:0d:0d:5b:0b  1500  QLogic Corp. QLogic FastLinQ QL41xxx 1/10/25 GbE Ethernet Adapter
vmnic8  0000:43:00.0  nmlx5_core  Up    10000  Full    9c:dc:71:4c:22:50  9000  Mellanox Technologies ConnectX-4 Lx EN NIC; 25GbE; dual-port SFP28; (MCX4121A-ACA)
vmnic9  0000:43:00.1  nmlx5_core  Up    10000  Full    9c:dc:71:4c:22:51  9000  Mellanox Technologies ConnectX-4 Lx EN NIC; 25GbE; dual-port SFP28; (MCX4121A-ACA)
 

Attachments

vangoose

Active Member
May 21, 2019
263
69
28
Canada
hmmm

In vmkernel.log
Code:
2020-06-11T01:53:54.830Z cpu10:2097622)WARNING: PCI: 1614: Enable VFs failed for device @0000:07:00.0, Please make sure the system has proper BIOS installed and enabled for SRIOV.
2020-06-11T01:53:55.475Z cpu10:2097622)WARNING: PCI: 1614: Enable VFs failed for device @0000:07:00.1, Please make sure the system has proper BIOS installed and enabled for SRIOV.
But it's turned on in firmware. Another issue, firmware BIOS utility is not available. Maybe compatibility issues? Hope not.