Mellanox ConnectX-4 LX EN SR-IOV on ESXi 6.7U3

vangoose

Active Member
May 21, 2019
268
73
28
Canada
Very strange. Motherboard is SM X9SRL-F, BIOS setting is right. I'm going to try a different slot.

SR-IOV is working
Code:
esxcli network sriovnic list
Name    PCI Device    Driver   Link  Speed  Duplex  MAC Address         MTU  Description
------  ------------  -------  ----  -----  ------  -----------------  ----  ---------------------------------------------------------------------
vmnic4  0000:06:00.2  qedentv  Up    10000  Full    34:80:0d:0d:8f:c2  9000  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter
vmnic5  0000:06:00.3  qedentv  Up    10000  Full    34:80:0d:0d:8f:c3  9000  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter
vmnic6  0000:06:00.4  qedentv  Up    10000  Full    34:80:0d:0d:8f:c4  9000  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter
vmnic7  0000:06:00.5  qedentv  Up    10000  Full    34:80:0d:0d:8f:c5  9000  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter
vmnic8  0000:06:00.6  qedentv  Up    10000  Full    34:80:0d:0d:8f:c6  1500  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter
vmnic9  0000:06:00.7  qedentv  Up    10000  Full    34:80:0d:0d:8f:c7  1500  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter

SR-IOV is turned on in firmware
Code:
/opt/mellanox/bin/mlxconfig -d mt4117_pciconf0  q
Device #1:
----------

Device type:    ConnectX4LX
Name:           MCX4121A-ACA_Ax
Description:    ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
Device:         mt4117_pciconf0

Configurations:                              Next Boot
         MEMIC_BAR_SIZE                      0
         MEMIC_SIZE_LIMIT                    _256KB(1)
         FLEX_PARSER_PROFILE_ENABLE          0
         FLEX_IPV4_OVER_VXLAN_PORT           0
         ROCE_NEXT_PROTOCOL                  254
         NON_PREFETCHABLE_PF_BAR             False(0)
         VF_VPD_ENABLE                       False(0)
         STRICT_VF_MSIX_NUM                  False(0)
         VF_NODNIC_ENABLE                    False(0)
         NUM_OF_VFS                          8
         SRIOV_EN                            True(1)

Same card is working in H11 board.
Code:
esxcli network sriovnic list
Name    PCI Device    Driver      Link  Speed  Duplex  MAC Address         MTU  Description
------  ------------  ----------  ----  -----  ------  -----------------  ----  -----------
vmnic4  0000:85:00.4  qedentv     Up    10000  Full    34:80:0d:0d:5b:08  1500  QLogic Corp. QLogic FastLinQ QL41xxx 1/10/25 GbE Ethernet Adapter
vmnic5  0000:85:00.5  qedentv     Up    10000  Full    34:80:0d:0d:5b:09  1500  QLogic Corp. QLogic FastLinQ QL41xxx 1/10/25 GbE Ethernet Adapter
vmnic6  0000:85:00.6  qedentv     Up    10000  Full    34:80:0d:0d:5b:0a  1500  QLogic Corp. QLogic FastLinQ QL41xxx 1/10/25 GbE Ethernet Adapter
vmnic7  0000:85:00.7  qedentv     Up    10000  Full    34:80:0d:0d:5b:0b  1500  QLogic Corp. QLogic FastLinQ QL41xxx 1/10/25 GbE Ethernet Adapter
vmnic8  0000:43:00.0  nmlx5_core  Up    10000  Full    9c:dc:71:4c:22:50  9000  Mellanox Technologies ConnectX-4 Lx EN NIC; 25GbE; dual-port SFP28; (MCX4121A-ACA)
vmnic9  0000:43:00.1  nmlx5_core  Up    10000  Full    9c:dc:71:4c:22:51  9000  Mellanox Technologies ConnectX-4 Lx EN NIC; 25GbE; dual-port SFP28; (MCX4121A-ACA)
 

Attachments

vangoose

Active Member
May 21, 2019
268
73
28
Canada
hmmm

In vmkernel.log
Code:
2020-06-11T01:53:54.830Z cpu10:2097622)WARNING: PCI: 1614: Enable VFs failed for device @0000:07:00.0, Please make sure the system has proper BIOS installed and enabled for SRIOV.
2020-06-11T01:53:55.475Z cpu10:2097622)WARNING: PCI: 1614: Enable VFs failed for device @0000:07:00.1, Please make sure the system has proper BIOS installed and enabled for SRIOV.
But it's turned on in firmware. Another issue, firmware BIOS utility is not available. Maybe compatibility issues? Hope not.
 

johngillespie

New Member
Jun 20, 2012
13
0
1
Paris, France
Hi, did you ever get this working ?

I seem to be facing the same issue with the same motherboard and also using a ConnectX-4 LX card (AOC-S25G-m2S).

I'm running ESXi 7.0u2
SR-IOV settings are enabled in BIOS
AOC-S25G-m2S Card Firmware Version: 14.28.2006
nmlx5_core driver version: 4.19.70.1
SR-IOV settings are enabled in NIC firmware config:
NUM_OF_VFS 8
SRIOV_EN True(1)

I am able to use SR-IOV in the same server with an intel X520-DA2 card.
I've tried moving the AOC-S25G-m2S card to the same slot as the Intel one which works with SR-IOV but no change.

Any input would be welcome :)