Connectx-4 lx virtual function devices not working

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
B

bbqdt

Member
Sep 15, 2019
93
64
18
I have enabled sr-iov and added vfs in the driver properly per the instructions

The problem is that after I add a virtual functions via `sriov_numvfs` I cannot seem to configure them nor use them in KVM.

Here is my current state -

Code: 
# lspci | grep X-4
83:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
83:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
# ip link show eth1
15: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 98:03:9b:94:8d:30 brd ff:ff:ff:ff:ff:ff
Note that the VF does not show under eth1, it shows as eth2:

Code: 
# ip link show eth2
79: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 56:17:3a:18:05:d6 brd ff:ff:ff:ff:ff:ff

If I unbind the vf via `/sys/bus/pci/drivers/mlx5_core/unbind` it does not show at all under `ip link`.

With the vf bound _or_ unbound I get the following when trying to do _any_ ip link set operations on it -

Code: 
# ip link set eth4 vf 0 mac 52:54:00:96:19:c1
RTNETLINK answers: Operation not supported
Another interesting point is that if I try to use it in kvm with an `<interface>` element I get this error when trying to start the vm - `error: internal error: missing IFLA_VF_INFO in netlink response`.


If I use a `<hostdev>` element I can start the vm and it seems to recognize it but network traffic does not work.

Am I missing some config somewhere? Do I need to set some param somewhere that I am missing?

System info -

Code: 
# uname -a
Linux staton-nas 5.15.43-Unraid #1 SMP Wed May 25 11:54:05 PDT 2022 x86_64 Intel(R) Xeon(R) CPU E5-2648L v2 @ 1.90GHz GenuineIntel GNU/Linux
# ip -V
ip utility, iproute2-5.16.0
# virsh -V
Virsh command line tool of libvirt 8.2.0
See web site at https://libvirt.org/

Compiled with support for:
 Hypervisors: QEMU/KVM Cloud-Hypervisor
 Networking: Remote Network Bridging Interface udev Nwfilter
 Storage: Dir
 Miscellaneous: Daemon Nodedev Secrets Debug
 # mstconfig -d 83:00.0 q

Device #1:
----------

Device type:    ConnectX4LX     
Name:           MCX4431A-GCA_Bx 
Description:    ConnectX-4 Lx EN network interface card for OCP; 50GbE single-port QSFP28; PCIe3.0 x8; ROHS R6
Device:         83:00.0         

Configurations:                              Next Boot
         MEMIC_BAR_SIZE                      0               
         MEMIC_SIZE_LIMIT                    _256KB(1)       
         FLEX_PARSER_PROFILE_ENABLE          0               
         FLEX_IPV4_OVER_VXLAN_PORT           0               
         ROCE_NEXT_PROTOCOL                  254             
         PF_NUM_OF_VF_VALID                  False(0)        
         NON_PREFETCHABLE_PF_BAR             False(0)        
         VF_VPD_ENABLE                       False(0)        
         STRICT_VF_MSIX_NUM                  False(0)        
         VF_NODNIC_ENABLE                    False(0)        
         NUM_PF_MSIX_VALID                   True(1)         
         NUM_OF_VFS                          20              
         NUM_OF_PF                           1               
         SRIOV_EN                            True(1)         
         PF_LOG_BAR_SIZE                     5               
         VF_LOG_BAR_SIZE                     0               
         NUM_PF_MSIX                         63              
         NUM_VF_MSIX                         11              
         INT_LOG_MAX_PAYLOAD_SIZE            AUTOMATIC(0)    
         PCIE_CREDIT_TOKEN_TIMEOUT           0               
         ACCURATE_TX_SCHEDULER               False(0)        
         PARTIAL_RESET_EN                    False(0)        
         SW_RECOVERY_ON_ERRORS               False(0)        
         RESET_WITH_HOST_ON_ERRORS           False(0)        
         PCI_BUS0_RESTRICT_SPEED             PCI_GEN_1(0)    
         PCI_BUS0_RESTRICT_ASPM              False(0)        
         PCI_BUS0_RESTRICT_WIDTH             PCI_X1(0)       
         PCI_BUS0_RESTRICT                   False(0)        
         PCI_DOWNSTREAM_PORT_OWNER           Array[0..15]    
         CQE_COMPRESSION                     BALANCED(0)     
         IP_OVER_VXLAN_EN                    False(0)        
         MKEY_BY_NAME                        False(0)        
         UCTX_EN                             True(1)         
         PCI_ATOMIC_MODE                     PCI_ATOMIC_DISABLED_EXT_ATOMIC_ENABLED(0)
         TUNNEL_ECN_COPY_DISABLE             False(0)        
         LRO_LOG_TIMEOUT0                    6               
         LRO_LOG_TIMEOUT1                    7               
         LRO_LOG_TIMEOUT2                    8               
         LRO_LOG_TIMEOUT3                    13              
         ICM_CACHE_MODE                      DEVICE_DEFAULT(0)
         TX_SCHEDULER_BURST                  0               
         LOG_MAX_QUEUE                       17              
         LOG_DCR_HASH_TABLE_SIZE             14              
         DCR_LIFO_SIZE                       16384           
         ROCE_CC_PRIO_MASK_P1                255             
         CLAMP_TGT_RATE_AFTER_TIME_INC_P1    True(1)         
         CLAMP_TGT_RATE_P1                   False(0)        
         RPG_TIME_RESET_P1                   300             
         RPG_BYTE_RESET_P1                   32767           
         RPG_THRESHOLD_P1                    1               
         RPG_MAX_RATE_P1                     0               
         RPG_AI_RATE_P1                      5               
         RPG_HAI_RATE_P1                     50              
         RPG_GD_P1                           11              
         RPG_MIN_DEC_FAC_P1                  50              
         RPG_MIN_RATE_P1                     1               
         RATE_TO_SET_ON_FIRST_CNP_P1         0               
         DCE_TCP_G_P1                        1019            
         DCE_TCP_RTT_P1                      1               
         RATE_REDUCE_MONITOR_PERIOD_P1       4               
         INITIAL_ALPHA_VALUE_P1              1023            
         MIN_TIME_BETWEEN_CNPS_P1            4               
         CNP_802P_PRIO_P1                    6               
         CNP_DSCP_P1                         48              
         LLDP_NB_DCBX_P1                     False(0)        
         LLDP_NB_RX_MODE_P1                  OFF(0)          
         LLDP_NB_TX_MODE_P1                  OFF(0)          
         DCBX_IEEE_P1                        True(1)         
         DCBX_CEE_P1                         True(1)         
         DCBX_WILLING_P1                     True(1)         
         KEEP_ETH_LINK_UP_P1                 True(1)         
         KEEP_IB_LINK_UP_P1                  False(0)        
         KEEP_LINK_UP_ON_BOOT_P1             False(0)        
         KEEP_LINK_UP_ON_STANDBY_P1          False(0)        
         DO_NOT_CLEAR_PORT_STATS_P1          False(0)        
         AUTO_POWER_SAVE_LINK_DOWN_P1        False(0)        
         NUM_OF_VL_P1                        _4_VLs(3)       
         NUM_OF_TC_P1                        _8_TCs(0)       
         NUM_OF_PFC_P1                       8               
         VL15_BUFFER_SIZE_P1                 0               
         DUP_MAC_ACTION_P1                   LAST_CFG(0)     
         SRIOV_IB_ROUTING_MODE_P1            LID(1)          
         IB_ROUTING_MODE_P1                  LID(1)          
         PHY_FEC_OVERRIDE_P1                 DEVICE_DEFAULT(0)
         WOL_MAGIC_EN                        True(1)         
         ROCE_CONTROL                        ROCE_ENABLE(2)  
         PCI_WR_ORDERING                     per_mkey(0)     
         MULTI_PORT_VHCA_EN                  False(0)        
         PORT_OWNER                          True(1)         
         ALLOW_RD_COUNTERS                   True(1)         
         RENEG_ON_CHANGE                     True(1)         
         TRACER_ENABLE                       True(1)         
         IP_VER                              IPv4(0)         
         BOOT_UNDI_NETWORK_WAIT              0               
         UEFI_HII_EN                         True(1)         
         BOOT_DBG_LOG                        False(0)        
         UEFI_LOGS                           DISABLED(0)     
         BOOT_VLAN                           1               
         LEGACY_BOOT_PROTOCOL                PXE(1)          
         BOOT_RETRY_CNT                      NONE(0)         
         BOOT_INTERRUPT_DIS                  False(0)        
         BOOT_LACP_DIS                       True(1)         
         BOOT_VLAN_EN                        False(0)        
         BOOT_PKEY                           0               
         DYNAMIC_VF_MSIX_TABLE               False(0)        
         EXP_ROM_UEFI_ARM_ENABLE             False(0)        
         EXP_ROM_UEFI_x86_ENABLE             False(0)        
         EXP_ROM_PXE_ENABLE                  True(1)         
         ADVANCED_PCI_SETTINGS               False(0)        
         SAFE_MODE_THRESHOLD                 10              
         SAFE_MODE_ENABLE                    True(1)   
# mstflint -d 83:00.0 q
Image type:            FS3
FW Version:            14.32.1010
FW Release Date:       1.12.2021
Product Version:       14.32.1010
Rom Info:              type=UEFI version=14.25.17 cpu=AMD64,AARCH64
                       type=PXE version=3.6.502 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             98039b0300948d30        4
Base MAC:              98039b948d30            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  MT_2490111032
Security Attributes:   N/A

Also note that my setup works fine with my connectx-3 card.
 
B

bbqdt

Member
Sep 15, 2019
93
64
18
I will add that the card works. It works in the host just fine and have tested up to 10gbps.

It seems like a compatibility problem with a new-ish kernel (5.15.43) and the toolchains. Although I am having a very hard time finding people with the same issue via google.
 
L

llowrey

Active Member
Feb 26, 2018
168
140
43
SRIOV for ConnectX-4 is very different from ConnectX-3. I recommend saving the content below to /etc/udev/rules.d/99-sriov.rules:

Code: 
ACTION=="add", SUBSYSTEM=="net", ATTRS{vendor}=="0x15b3", ATTRS{device}=="0x1013", ATTR{device/sriov_drivers_autoprobe}="0", ATTR{device/sriov_numvfs}="16"
Change the device ID to that of your card. You can find the id by running lspci -nn | grep ConnectX. Also change the sriov_numvfs to your desired value.

Now, on reboot, you will have the desired number of VFs and the OS won't try to bind to them so they will be free for KVM to use. You'll be able to confirm via ip link.

I also recommend creating a libvirt network so it can manage the VFs for you. Here's mine:

Code: 
<network>
  <name>Mellanox</name>
  <uuid>a4bcc942-67c9-4942-8e3a-f21863ba7c41</uuid>
  <forward mode='hostdev' managed='yes'>
    <pf dev='enp194s0'/>
  </forward>
</network>
Set the pf dev to match your physical interface.
 
B

bbqdt

Member
Sep 15, 2019
93
64
18
Tried that.

Starting vm still reports `error: internal error: missing IFLA_VF_INFO in netlink response`.
 
B

bbqdt

Member
Sep 15, 2019
93
64
18
Some progress -

I booted ubuntu from a flash drive and it works fine -

Screen Shot 2022-06-15 at 10.48.07 AM.png

So it seems to be a problem with the unraid kernel/config/toolchain.
 
You must log in or register to reply here.
Top Bottom