Mellanox ConnectX-4 LX EN SR-IOV on ESXi 6.7U3

[vangoose]

Very strange. Motherboard is SM X9SRL-F, BIOS setting is right. I'm going to try a different slot.

SR-IOV is working

esxcli network sriovnic list
Name    PCI Device    Driver   Link  Speed  Duplex  MAC Address         MTU  Description
------  ------------  -------  ----  -----  ------  -----------------  ----  ---------------------------------------------------------------------
vmnic4  0000:06:00.2  qedentv  Up    10000  Full    34:80:0d:0d:8f:c2  9000  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter
vmnic5  0000:06:00.3  qedentv  Up    10000  Full    34:80:0d:0d:8f:c3  9000  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter
vmnic6  0000:06:00.4  qedentv  Up    10000  Full    34:80:0d:0d:8f:c4  9000  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter
vmnic7  0000:06:00.5  qedentv  Up    10000  Full    34:80:0d:0d:8f:c5  9000  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter
vmnic8  0000:06:00.6  qedentv  Up    10000  Full    34:80:0d:0d:8f:c6  1500  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter
vmnic9  0000:06:00.7  qedentv  Up    10000  Full    34:80:0d:0d:8f:c7  1500  QLogic Corp. QLogic FastLinQ QL41xxx 10/25/40/50 GbE Ethernet Adapter

SR-IOV is turned on in firmware

/opt/mellanox/bin/mlxconfig -d mt4117_pciconf0  q
Device #1:
----------

Device type:    ConnectX4LX
Name:           MCX4121A-ACA_Ax
Description:    ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
Device:         mt4117_pciconf0

Configurations:                              Next Boot
         MEMIC_BAR_SIZE                      0
         MEMIC_SIZE_LIMIT                    _256KB(1)
         FLEX_PARSER_PROFILE_ENABLE          0
         FLEX_IPV4_OVER_VXLAN_PORT           0
         ROCE_NEXT_PROTOCOL                  254
         NON_PREFETCHABLE_PF_BAR             False(0)
         VF_VPD_ENABLE                       False(0)
         STRICT_VF_MSIX_NUM                  False(0)
         VF_NODNIC_ENABLE                    False(0)
         NUM_OF_VFS                          8
         SRIOV_EN                            True(1)

Same card is working in H11 board.

esxcli network sriovnic list
Name    PCI Device    Driver      Link  Speed  Duplex  MAC Address         MTU  Description
------  ------------  ----------  ----  -----  ------  -----------------  ----  -----------
vmnic4  0000:85:00.4  qedentv     Up    10000  Full    34:80:0d:0d:5b:08  1500  QLogic Corp. QLogic FastLinQ QL41xxx 1/10/25 GbE Ethernet Adapter
vmnic5  0000:85:00.5  qedentv     Up    10000  Full    34:80:0d:0d:5b:09  1500  QLogic Corp. QLogic FastLinQ QL41xxx 1/10/25 GbE Ethernet Adapter
vmnic6  0000:85:00.6  qedentv     Up    10000  Full    34:80:0d:0d:5b:0a  1500  QLogic Corp. QLogic FastLinQ QL41xxx 1/10/25 GbE Ethernet Adapter
vmnic7  0000:85:00.7  qedentv     Up    10000  Full    34:80:0d:0d:5b:0b  1500  QLogic Corp. QLogic FastLinQ QL41xxx 1/10/25 GbE Ethernet Adapter
vmnic8  0000:43:00.0  nmlx5_core  Up    10000  Full    9c:dc:71:4c:22:50  9000  Mellanox Technologies ConnectX-4 Lx EN NIC; 25GbE; dual-port SFP28; (MCX4121A-ACA)
vmnic9  0000:43:00.1  nmlx5_core  Up    10000  Full    9c:dc:71:4c:22:51  9000  Mellanox Technologies ConnectX-4 Lx EN NIC; 25GbE; dual-port SFP28; (MCX4121A-ACA)

 

[vangoose]

hmmm

In vmkernel.log

2020-06-11T01:53:54.830Z cpu10:2097622)WARNING: PCI: 1614: Enable VFs failed for device @0000:07:00.0, Please make sure the system has proper BIOS installed and enabled for SRIOV.
2020-06-11T01:53:55.475Z cpu10:2097622)WARNING: PCI: 1614: Enable VFs failed for device @0000:07:00.1, Please make sure the system has proper BIOS installed and enabled for SRIOV.

But it's turned on in firmware. Another issue, firmware BIOS utility is not available. Maybe compatibility issues? Hope not.

 

[johngillespie]

Hi, did you ever get this working ?

I seem to be facing the same issue with the same motherboard and also using a ConnectX-4 LX card (AOC-S25G-m2S).

I'm running ESXi 7.0u2
SR-IOV settings are enabled in BIOS
AOC-S25G-m2S Card Firmware Version: 14.28.2006
nmlx5_core driver version: 4.19.70.1
SR-IOV settings are enabled in NIC firmware config:
NUM_OF_VFS 8
SRIOV_EN True(1)

I am able to use SR-IOV in the same server with an intel X520-DA2 card.
I've tried moving the AOC-S25G-m2S card to the same slot as the Intel one which works with SR-IOV but no change.

Any input would be welcome

 

[bandit8623]

Hi, did you ever get this working ?

I seem to be facing the same issue with the same motherboard and also using a ConnectX-4 LX card (AOC-S25G-m2S).

I'm running ESXi 7.0u2
SR-IOV settings are enabled in BIOS
AOC-S25G-m2S Card Firmware Version: 14.28.2006
nmlx5_core driver version: 4.19.70.1
SR-IOV settings are enabled in NIC firmware config:
NUM_OF_VFS 8
SRIOV_EN True(1)

I am able to use SR-IOV in the same server with an intel X520-DA2 card.
I've tried moving the AOC-S25G-m2S card to the same slot as the Intel one which works with SR-IOV but no change.

Any input would be welcome

for me on the hypervisor the newest driver that passed sriov through was 1.70.18120.0 dated 8/14/2017

my cards firmware is on 14.17.2020 -actually a netapp rebranded from supermicro.

this allowed sriov to work on all my vms. any newer driver than that one and only vmq worked v.
inside the vms i can use the latest greats virtual function driver.

if its a card you could force to a newer firmware. i believe its this model MCX4121A-ACAT. my sys has 2 onboard so i didnt try to update

as i stand everything is working flawlessly sriov on the driver listed above

sorry late to the game on this one but wanted to share