Help with ConnectX 3 SR-IOV with Linux host and windows guest via kvm

bbqdt · Jun 3, 2020

I’m having trouble getting my ConnectX 3 to work via SR-IOV in windows 10 or server 2019 guests in kvm.

I get error 43 in both from the latest Mellanox or Microsoft drivers.

Literally the exact same xml config files work fine with Linux and FreeBSD guests. I have tried several different configs with no success.

Can anyone post their kvm xml file that does work in this situation? Any suggestions?

mod - feel free to move if this is the wrong forum

Here is one of my config files -

XML:

<domain type='kvm'>
  <name>Windows Server 2019 q35</name>
  <uuid>64d6cd06-33c6-5fc4-f4b8-eb65091b17e7</uuid>
  <memory unit='KiB'>20971520</memory>
  <currentMemory unit='KiB'>20971520</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>14</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='20'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='21'/>
    <vcpupin vcpu='4' cpuset='2'/>
    <vcpupin vcpu='5' cpuset='22'/>
    <vcpupin vcpu='6' cpuset='3'/>
    <vcpupin vcpu='7' cpuset='23'/>
    <vcpupin vcpu='8' cpuset='4'/>
    <vcpupin vcpu='9' cpuset='24'/>
    <vcpupin vcpu='10' cpuset='5'/>
    <vcpupin vcpu='11' cpuset='25'/>
    <vcpupin vcpu='12' cpuset='6'/>
    <vcpupin vcpu='13' cpuset='26'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/64d6cd06-33c6-5fc4-f4b8-eb65091b17e7_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='7' threads='2'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/mnt/user/domains/Windows Server 2019 q35/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/17763.737.190906-2324.rs5_release_svc_refresh_SERVER_EVAL_x64FRE_en-us_1.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:fd:44:fe'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:c5:86:48'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x83' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>

llowrey · Jun 4, 2020

My configuration is a bit different.

Code:

<interface type='network'>
      <mac address='52:54:00:76:9a:89'/>
      <source network='Mellanox'/>
      <model type='e1000e'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>

Here's my network definition:

Code:

<network>
  <name>Mellanox</name>
  <uuid>a4bcc942-67c9-4942-8e3a-f21863ba7c41</uuid>
  <forward mode='hostdev' managed='yes'>
    <pf dev='eth0p0'/>
  </forward>
</network>

I am using the network above to treat the VFs as a pool and libvirt will issue the next available VF to the guest. It looks like you are statically assigning VFs. Is that right?

I'm using the inbox drivers provided by MS and am running Windows 10 Professional x86_64. I have not tried the OFED drivers

bbqdt · Jun 5, 2020

Yes, directly assigning them. Changing to a pool like you have didn’t work.

Can you post your whole xml file? I have a feeling it’s some random flag or something.

llowrey · Jun 5, 2020

Host: Epyc 7502 (32-core)
OS: CentOS 8.1.1911
Kernel: 4.18.0-147.8.1.el8_1.x86_64
CX-3: MCX354A-FCB_A2-A5 (MT_1090120019)
FW: 2.42.5000
OFED: v5.0-2.1.8

Code:

Device #1:
----------

Device type:    ConnectX3
Device:         /dev/mst/mt4099_pci_cr0

Configurations:                              Next Boot
         SRIOV_EN                            True(1)
         NUM_OF_VFS                          16
         LINK_TYPE_P1                        ETH(2)
         LINK_TYPE_P2                        ETH(2)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               True(1)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               True(1)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             PXE(1)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)

Code:

<domain type='kvm'>
  <name>windows_10-1</name>
  <uuid>9d724977-62e1-4d16-bd41-6096707bad27</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel7.6.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <topology sockets='1' cores='8' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='native' discard='unmap'/>
      <source file='/var/lib/libvirt/images/flash/windows_10-1.qcow2'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/isos/virtio-win-1.9.9.iso'/>
      <target dev='sdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <driver queues='8'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:76:9a:89'/>
      <source network='Mellanox'/>
      <model type='e1000e'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>

llowrey · Jun 5, 2020

Also...

/etc/modprobe.d/mlnx.conf

Code:

blacklist mlx4_core
blacklist mlx4_en
blacklist mlx5_core
blacklist mlx5_ib

options mlx4_core num_vfs=16,0,0 port_type_array=2,2 probe_vf=1,0,0

/etc/modprobe.d/vfio.conf

Code:

options vfio-pci ids=1022:7901,9005:028c,8086:1521,15b3:1004

... and the following kernel args:

Code:

rd.driver.pre=vfio-pci pci=realloc iommu=pti

I had to pass pci=realloc to get past an issue with the CX-3. I can't recall now what it was.

efschu3 · Jun 2, 2021

I run into the similar problem. JFYI:
The problem is the in-kernel driver version v4.0.0, this driver has a bug, which was resolved with v4.3.x OFED. Latest LTS driver for cx3(pro) will only build for kernel up to 5.4, acutal it should work on 5.8 and 5.10 kernels too, but it does not. For now I go with kernel 5.4 until Mellanox fixes their LTS driver.

okrasit · Jun 3, 2021

Here's a dirty patch for the old (5.5.15) mainstream kernel driver.
It might apply successfully to later kernels as well.

Code:

From d4fbe94b4adb16a5cc094a50215dc39496c9fc66 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 05:35:13 +0000
Subject: [PATCH] MLX4 patch for vf

---
drivers/net/ethernet/mellanox/mlx4/fw.c | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/fw.c b/drivers/net/ethernet/mellanox/mlx4/fw.c
index 6e501af0e..2d52e29fd 100644
--- a/drivers/net/ethernet/mellanox/mlx4/fw.c
+++ b/drivers/net/ethernet/mellanox/mlx4/fw.c
@@ -1408,7 +1408,7 @@ int mlx4_QUERY_PORT_wrapper(struct mlx4_dev *dev, int slave,
{
     struct mlx4_priv *priv = mlx4_priv(dev);
     u64 def_mac;
-    u8 port_type;
+    u8 port_type, vlan, field;
     u16 short_field;
     int err;
     int admin_link_state;
@@ -1476,6 +1476,16 @@ int mlx4_QUERY_PORT_wrapper(struct mlx4_dev *dev, int slave,
         short_field = dev->caps.pkey_table_len[vhcr->in_modifier];
         MLX4_PUT(outbox->buf, short_field,
              QUERY_PORT_CUR_MAX_PKEY_OFFSET);
+
+        /* Change the mac table size for the VF */
+        MLX4_GET(field, outbox, QUERY_PORT_MAX_MACVLAN_OFFSET);
+        /* keep the origin vlan of the VF */
+        vlan = field >> 4;
+        /* set the field with the prev vlan and the mac defined quota */
+        field = vlan << 4;
+        field |= ilog2(mlx4_get_port_free_macs(dev,
+            priv->port->port + 1));
+        MLX4_PUT(outbox->buf, field, QUERY_PORT_MAX_MACVLAN_OFFSET);
     }
out:
     return err;
--
2.27.0

From 98cc63cd92b23d8545ad1832d55bcae935fb2f24 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 06:27:20 +0000
Subject: [PATCH 2/2] MLX4 get macs

---
drivers/net/ethernet/mellanox/mlx4/mlx4.h |  1 +
drivers/net/ethernet/mellanox/mlx4/port.c | 57 +++++++++++++++++++++++
2 files changed, 58 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4.h b/drivers/net/ethernet/mellanox/mlx4/mlx4.h
index 527b52e48..71ab05246 100644
--- a/drivers/net/ethernet/mellanox/mlx4/mlx4.h
+++ b/drivers/net/ethernet/mellanox/mlx4/mlx4.h
@@ -740,6 +740,7 @@ struct mlx4_catas_err {

#define MLX4_MAX_MAC_NUM    128
#define MLX4_MAC_TABLE_SIZE    (MLX4_MAX_MAC_NUM << 3)
+#define MLX4_VF_MAC_QUOTA   2

struct mlx4_mac_table {
     __be64            entries[MLX4_MAX_MAC_NUM];
diff --git a/drivers/net/ethernet/mellanox/mlx4/port.c b/drivers/net/ethernet/mellanox/mlx4/port.c
index ba6ac31a3..7ea7b9db9 100644
--- a/drivers/net/ethernet/mellanox/mlx4/port.c
+++ b/drivers/net/ethernet/mellanox/mlx4/port.c
@@ -2128,3 +2128,60 @@ int mlx4_max_tc(struct mlx4_dev *dev)
     return num_tc;
}
EXPORT_SYMBOL(mlx4_max_tc);
+
+static int mlx4_get_port_reserved_mac_num(struct mlx4_dev *mdev, int port)
+{
+    struct mlx4_priv *priv = mlx4_priv(mdev);
+    struct resource_allocator *res_alloc;
+    int reserved;
+
+    if (mlx4_is_slave(mdev))
+        return 0;
+
+    res_alloc = &priv->mfunc.master.res_tracker.res_alloc[RES_MAC];
+
+    reserved = (port > 0) ? res_alloc->res_port_rsvd[port - 1] :
+        res_alloc->res_reserved;
+
+    return reserved;
+}
+
+int mlx4_get_port_max_macs(struct mlx4_dev *mdev, int port)
+{
+    struct mlx4_port_info *info = &mlx4_priv(mdev)->port[port];
+
+    /* The maximum value should considers the reserved macs for the vfs */
+    return info->mac_table.max - mlx4_get_port_reserved_mac_num(mdev, port);
+}
+EXPORT_SYMBOL(mlx4_get_port_max_macs);
+
+int mlx4_get_port_total_macs(struct mlx4_dev *mdev, int port)
+{
+    struct mlx4_port_info *info = &mlx4_priv(mdev)->port[port];
+
+    return info->mac_table.total;
+}
+EXPORT_SYMBOL(mlx4_get_port_total_macs);
+
+int mlx4_get_port_free_macs(struct mlx4_dev *mdev, int port)
+{
+    /* slave will get the free macs (log2) from its master */
+    if (mlx4_is_slave(mdev)) {
+        struct mlx4_port_cap port_cap;
+
+        mlx4_QUERY_PORT(mdev, port, &port_cap);
+        return (1 << port_cap.log_max_macs);
+    }
+
+    return (mlx4_get_port_max_macs(mdev, port) -
+        mlx4_get_port_total_macs(mdev, port));
+}
+EXPORT_SYMBOL(mlx4_get_port_free_macs);
+
+bool mlx4_is_available_mac(struct mlx4_dev *mdev, int port)
+{
+    int free_macs = mlx4_get_port_free_macs(mdev, port);
+
+    return free_macs >= MLX4_VF_MAC_QUOTA;
+}
+EXPORT_SYMBOL(mlx4_is_available_mac);
--
2.27.0

From fd2e45a02d177e71f2b2d6ea509d25bc3757ac1b Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 06:42:41 +0000
Subject: [PATCH 3/3] MLX4 get macs headers

---
include/linux/mlx4/device.h | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h
index 36e412c3d..c5ef47a5f 100644
--- a/include/linux/mlx4/device.h
+++ b/include/linux/mlx4/device.h
@@ -1515,6 +1515,10 @@ int mlx4_phys_to_slave_port(struct mlx4_dev *dev, int slave, int port);
int mlx4_get_base_gid_ix(struct mlx4_dev *dev, int slave, int port);

int mlx4_config_vxlan_port(struct mlx4_dev *dev, __be16 udp_port);
+int mlx4_get_port_max_macs(struct mlx4_dev *mdev, int port);
+int mlx4_get_port_total_macs(struct mlx4_dev *mdev, int port);
+int mlx4_get_port_free_macs(struct mlx4_dev *mdev, int port);
+bool mlx4_is_available_mac(struct mlx4_dev *mdev, int port);
int mlx4_disable_rx_port_check(struct mlx4_dev *dev, bool dis);
int mlx4_config_roce_v2_port(struct mlx4_dev *dev, u16 udp_port);
int mlx4_virt2phy_port_map(struct mlx4_dev *dev, u32 port1, u32 port2);
--
2.27.0

From 77037f4e53732f69001634359086b369d51cf6c0 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 07:16:40 +0000
Subject: [PATCH 4/4] MLX4 fix header

---
drivers/net/ethernet/mellanox/mlx4/port.c | 1 +
1 file changed, 1 insertion(+)

diff --git a/drivers/net/ethernet/mellanox/mlx4/port.c b/drivers/net/ethernet/mellanox/mlx4/port.c
index 7ea7b9db9..616a3b73b 100644
--- a/drivers/net/ethernet/mellanox/mlx4/port.c
+++ b/drivers/net/ethernet/mellanox/mlx4/port.c
@@ -39,6 +39,7 @@

#include "mlx4.h"
#include "mlx4_stats.h"
+#include "fw.h"

#define MLX4_MAC_VALID        (1ull << 63)

--
2.27.0

From 3011ace4fd230fec377e62aa1e9386cf0b1ab9e8 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 12:04:37 +0000
Subject: [PATCH 5/5] MLX4 testing

---
drivers/net/ethernet/mellanox/mlx4/fw.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/fw.c b/drivers/net/ethernet/mellanox/mlx4/fw.c
index 2d52e29fd..90d4bac3c 100644
--- a/drivers/net/ethernet/mellanox/mlx4/fw.c
+++ b/drivers/net/ethernet/mellanox/mlx4/fw.c
@@ -820,6 +820,7 @@ int mlx4_QUERY_DEV_CAP(struct mlx4_dev *dev, struct mlx4_dev_cap *dev_cap)
#define QUERY_DEV_CAP_DIAG_RPRT_PER_PORT    0x9c
#define QUERY_DEV_CAP_FW_REASSIGN_MAC        0x9d
#define QUERY_DEV_CAP_VXLAN            0x9e
+#define QUERY_DEV_CAP_ADD_MAC            0x9f
#define QUERY_DEV_CAP_MAD_DEMUX_OFFSET        0xb0
#define QUERY_DEV_CAP_DMFS_HIGH_RATE_QPN_BASE_OFFSET    0xa8
#define QUERY_DEV_CAP_DMFS_HIGH_RATE_QPN_RANGE_OFFSET    0xac
@@ -1327,6 +1328,11 @@ int mlx4_QUERY_DEV_CAP_wrapper(struct mlx4_dev *dev, int slave,
     field &= 0xd7;
     MLX4_PUT(outbox->buf, field, QUERY_DEV_CAP_VXLAN);

+    /* For guests report additional-mac query not available */
+    MLX4_GET(field, outbox->buf, QUERY_DEV_CAP_ADD_MAC);
+    field &= 0xfb;
+    MLX4_PUT(outbox->buf, field, QUERY_DEV_CAP_ADD_MAC);
+
     /* For guests, disable port BEACON */
     MLX4_GET(field, outbox->buf, QUERY_DEV_CAP_PORT_BEACON_OFFSET);
     field &= 0x7f;
@@ -1427,7 +1433,7 @@ int mlx4_QUERY_PORT_wrapper(struct mlx4_dev *dev, int slave,
      * QUERY_PORT general query.
      */
     if (vhcr->op_modifier || vhcr->in_modifier & ~0xFF)
-        return -EINVAL;
+        return -23;

     vhcr->in_modifier = port;

--
2.27.0

From 5193521e6e26ece2b4a9a69e3031dbf9a46a9a5d Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 13:07:20 +0000
Subject: [PATCH 6/6] MLX4 stats

---
.../net/ethernet/mellanox/mlx4/resource_tracker.c | 15 +++++++++++++++
1 file changed, 15 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
index 1187ef137..9bcaa1517 100644
--- a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
+++ b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
@@ -4560,6 +4560,16 @@ enum {
     BUSY_MAX_RETRIES = 10
};

+static void mlx4_if_stat_zeroes(struct mlx4_counter *if_cnt)
+{
+    u8 temp;
+    temp = if_cnt->counter_mode;
+
+    memset(if_cnt, 0, sizeof(*if_cnt));
+    if_cnt->counter_mode = temp;
+    if_cnt->num_ifc = cpu_to_be32(1);
+}
+
int mlx4_QUERY_IF_STAT_wrapper(struct mlx4_dev *dev, int slave,
                    struct mlx4_vhcr *vhcr,
                    struct mlx4_cmd_mailbox *inbox,
@@ -4569,6 +4579,11 @@ int mlx4_QUERY_IF_STAT_wrapper(struct mlx4_dev *dev, int slave,
     int err;
     int index = vhcr->in_modifier & 0xffff;

+    if (index == MLX4_SINK_COUNTER_INDEX(dev)) {
+        mlx4_if_stat_zeroes(outbox->buf);
+        return 0;
+    }
+
     err = get_res(dev, slave, index, RES_COUNTER, NULL);
     if (err)
         return err;
--
2.27.0

From 14f69590cb72ba3cdadfdad43585e95931db86b7 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Wed, 29 Jul 2020 13:25:12 +0000
Subject: [PATCH 7/7] MLX4 stat wrapper

---
drivers/net/ethernet/mellanox/mlx4/resource_tracker.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
index 9bcaa1517..cc0cb76cd 100644
--- a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
+++ b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
@@ -4576,7 +4576,7 @@ int mlx4_QUERY_IF_STAT_wrapper(struct mlx4_dev *dev, int slave,
                    struct mlx4_cmd_mailbox *outbox,
                    struct mlx4_cmd_info *cmd)
{
-    int err;
+    int err = 0;
     int index = vhcr->in_modifier & 0xffff;

     if (index == MLX4_SINK_COUNTER_INDEX(dev)) {
@@ -4585,8 +4585,10 @@ int mlx4_QUERY_IF_STAT_wrapper(struct mlx4_dev *dev, int slave,
     }

     err = get_res(dev, slave, index, RES_COUNTER, NULL);
-    if (err)
-        return err;
+    if (err) {
+        mlx4_if_stat_zeroes(outbox->buf);
+        err = 0;
+    }

     err = mlx4_DMA_wrapper(dev, slave, vhcr, inbox, outbox, cmd);
     put_res(dev, slave, index, RES_COUNTER);
--
2.27.0

mimino · Jun 6, 2021

@efschu3 It didn't work for me for some reason. I'm still getting "This device cannot start. (Code 10)" from windoze.
I installed it from mlnx-en-4.9-3.1.5.0-debian10.0-x86_64.iso, would that be the correct (fixed) version?

Code:

# modinfo mlx4_en
filename:       /lib/modules/5.4.106-1-pve/updates/dkms/mlx4_en.ko
version:        4.9-3.1.5

efschu3 · Jun 6, 2021

Code:

 mlx4_en
filename:       /lib/modules/5.4.114-1-pve/updates/dkms/mlx4_en.ko
version:        4.9-3.1.5

for proxmox and latest WinOF (not WinOF2) for winVM.
I did not have a Code10, I had Code43

What does your dmesg say when the driver tries to start?

Btw you need mlx4_core, mlx4_en is the ethernet part for the host (4.3 and later does work)

mimino · Jun 6, 2021

efschu3 said:
for proxmox and latest WinOF (not WinOF2) for winVM.
I did not have a Code10, I had Code43

I had Code 43 with kernel driver

efschu3 said:
What does your dmesg say when the driver tries to start?

What dmesg in windows guest? There is no issues with any of the Linux/BSD guests nor LXC containers, this driver works as well as the kernel one.

efschu3 said:
Btw you need mlx4_core, mlx4_en is the ethernet part for the host (4.3 and later does work)

Of course.

Code:

# lsmod | grep mlx
mlx4_en               139264  0
mlx4_core             352256  1 mlx4_en
mlx_compat             65536  2 mlx4_core,mlx4_en

efschu3 · Jun 6, 2021

dmesg of host when windows tries to start the vf.

mimino · Jun 7, 2021

efschu3 said:
dmesg of host when windows tries to start the vf.

There's only one line:

Code:

vfio-pci 0000:02:01.4: enabling device (0000 -> 0002)

okrasit · Jun 7, 2021

mimino said:
There's only one line:

Code:

vfio-pci 0000:02:01.4: enabling device (0000 -> 0002)

Did you assign mac-address to the vf?

bbqdt · Jun 7, 2021

okrasit said:

Here's a dirty patch for the old (5.5.15) mainstream kernel driver.
It might apply successfully to later kernels as well.

Code:

From d4fbe94b4adb16a5cc094a50215dc39496c9fc66 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 05:35:13 +0000
Subject: [PATCH] MLX4 patch for vf

---
drivers/net/ethernet/mellanox/mlx4/fw.c | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/fw.c b/drivers/net/ethernet/mellanox/mlx4/fw.c
index 6e501af0e..2d52e29fd 100644
--- a/drivers/net/ethernet/mellanox/mlx4/fw.c
+++ b/drivers/net/ethernet/mellanox/mlx4/fw.c
@@ -1408,7 +1408,7 @@ int mlx4_QUERY_PORT_wrapper(struct mlx4_dev *dev, int slave,
{
     struct mlx4_priv *priv = mlx4_priv(dev);
     u64 def_mac;
-    u8 port_type;
+    u8 port_type, vlan, field;
     u16 short_field;
     int err;
     int admin_link_state;
@@ -1476,6 +1476,16 @@ int mlx4_QUERY_PORT_wrapper(struct mlx4_dev *dev, int slave,
         short_field = dev->caps.pkey_table_len[vhcr->in_modifier];
         MLX4_PUT(outbox->buf, short_field,
              QUERY_PORT_CUR_MAX_PKEY_OFFSET);
+
+        /* Change the mac table size for the VF */
+        MLX4_GET(field, outbox, QUERY_PORT_MAX_MACVLAN_OFFSET);
+        /* keep the origin vlan of the VF */
+        vlan = field >> 4;
+        /* set the field with the prev vlan and the mac defined quota */
+        field = vlan << 4;
+        field |= ilog2(mlx4_get_port_free_macs(dev,
+            priv->port->port + 1));
+        MLX4_PUT(outbox->buf, field, QUERY_PORT_MAX_MACVLAN_OFFSET);
     }
out:
     return err;
--
2.27.0

From 98cc63cd92b23d8545ad1832d55bcae935fb2f24 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 06:27:20 +0000
Subject: [PATCH 2/2] MLX4 get macs

---
drivers/net/ethernet/mellanox/mlx4/mlx4.h |  1 +
drivers/net/ethernet/mellanox/mlx4/port.c | 57 +++++++++++++++++++++++
2 files changed, 58 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4.h b/drivers/net/ethernet/mellanox/mlx4/mlx4.h
index 527b52e48..71ab05246 100644
--- a/drivers/net/ethernet/mellanox/mlx4/mlx4.h
+++ b/drivers/net/ethernet/mellanox/mlx4/mlx4.h
@@ -740,6 +740,7 @@ struct mlx4_catas_err {

#define MLX4_MAX_MAC_NUM    128
#define MLX4_MAC_TABLE_SIZE    (MLX4_MAX_MAC_NUM << 3)
+#define MLX4_VF_MAC_QUOTA   2

struct mlx4_mac_table {
     __be64            entries[MLX4_MAX_MAC_NUM];
diff --git a/drivers/net/ethernet/mellanox/mlx4/port.c b/drivers/net/ethernet/mellanox/mlx4/port.c
index ba6ac31a3..7ea7b9db9 100644
--- a/drivers/net/ethernet/mellanox/mlx4/port.c
+++ b/drivers/net/ethernet/mellanox/mlx4/port.c
@@ -2128,3 +2128,60 @@ int mlx4_max_tc(struct mlx4_dev *dev)
     return num_tc;
}
EXPORT_SYMBOL(mlx4_max_tc);
+
+static int mlx4_get_port_reserved_mac_num(struct mlx4_dev *mdev, int port)
+{
+    struct mlx4_priv *priv = mlx4_priv(mdev);
+    struct resource_allocator *res_alloc;
+    int reserved;
+
+    if (mlx4_is_slave(mdev))
+        return 0;
+
+    res_alloc = &priv->mfunc.master.res_tracker.res_alloc[RES_MAC];
+
+    reserved = (port > 0) ? res_alloc->res_port_rsvd[port - 1] :
+        res_alloc->res_reserved;
+
+    return reserved;
+}
+
+int mlx4_get_port_max_macs(struct mlx4_dev *mdev, int port)
+{
+    struct mlx4_port_info *info = &mlx4_priv(mdev)->port[port];
+
+    /* The maximum value should considers the reserved macs for the vfs */
+    return info->mac_table.max - mlx4_get_port_reserved_mac_num(mdev, port);
+}
+EXPORT_SYMBOL(mlx4_get_port_max_macs);
+
+int mlx4_get_port_total_macs(struct mlx4_dev *mdev, int port)
+{
+    struct mlx4_port_info *info = &mlx4_priv(mdev)->port[port];
+
+    return info->mac_table.total;
+}
+EXPORT_SYMBOL(mlx4_get_port_total_macs);
+
+int mlx4_get_port_free_macs(struct mlx4_dev *mdev, int port)
+{
+    /* slave will get the free macs (log2) from its master */
+    if (mlx4_is_slave(mdev)) {
+        struct mlx4_port_cap port_cap;
+
+        mlx4_QUERY_PORT(mdev, port, &port_cap);
+        return (1 << port_cap.log_max_macs);
+    }
+
+    return (mlx4_get_port_max_macs(mdev, port) -
+        mlx4_get_port_total_macs(mdev, port));
+}
+EXPORT_SYMBOL(mlx4_get_port_free_macs);
+
+bool mlx4_is_available_mac(struct mlx4_dev *mdev, int port)
+{
+    int free_macs = mlx4_get_port_free_macs(mdev, port);
+
+    return free_macs >= MLX4_VF_MAC_QUOTA;
+}
+EXPORT_SYMBOL(mlx4_is_available_mac);
--
2.27.0

From fd2e45a02d177e71f2b2d6ea509d25bc3757ac1b Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 06:42:41 +0000
Subject: [PATCH 3/3] MLX4 get macs headers

---
include/linux/mlx4/device.h | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h
index 36e412c3d..c5ef47a5f 100644
--- a/include/linux/mlx4/device.h
+++ b/include/linux/mlx4/device.h
@@ -1515,6 +1515,10 @@ int mlx4_phys_to_slave_port(struct mlx4_dev *dev, int slave, int port);
int mlx4_get_base_gid_ix(struct mlx4_dev *dev, int slave, int port);

int mlx4_config_vxlan_port(struct mlx4_dev *dev, __be16 udp_port);
+int mlx4_get_port_max_macs(struct mlx4_dev *mdev, int port);
+int mlx4_get_port_total_macs(struct mlx4_dev *mdev, int port);
+int mlx4_get_port_free_macs(struct mlx4_dev *mdev, int port);
+bool mlx4_is_available_mac(struct mlx4_dev *mdev, int port);
int mlx4_disable_rx_port_check(struct mlx4_dev *dev, bool dis);
int mlx4_config_roce_v2_port(struct mlx4_dev *dev, u16 udp_port);
int mlx4_virt2phy_port_map(struct mlx4_dev *dev, u32 port1, u32 port2);
--
2.27.0

From 77037f4e53732f69001634359086b369d51cf6c0 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 07:16:40 +0000
Subject: [PATCH 4/4] MLX4 fix header

---
drivers/net/ethernet/mellanox/mlx4/port.c | 1 +
1 file changed, 1 insertion(+)

diff --git a/drivers/net/ethernet/mellanox/mlx4/port.c b/drivers/net/ethernet/mellanox/mlx4/port.c
index 7ea7b9db9..616a3b73b 100644
--- a/drivers/net/ethernet/mellanox/mlx4/port.c
+++ b/drivers/net/ethernet/mellanox/mlx4/port.c
@@ -39,6 +39,7 @@

#include "mlx4.h"
#include "mlx4_stats.h"
+#include "fw.h"

#define MLX4_MAC_VALID        (1ull << 63)

--
2.27.0

From 3011ace4fd230fec377e62aa1e9386cf0b1ab9e8 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 12:04:37 +0000
Subject: [PATCH 5/5] MLX4 testing

---
drivers/net/ethernet/mellanox/mlx4/fw.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/fw.c b/drivers/net/ethernet/mellanox/mlx4/fw.c
index 2d52e29fd..90d4bac3c 100644
--- a/drivers/net/ethernet/mellanox/mlx4/fw.c
+++ b/drivers/net/ethernet/mellanox/mlx4/fw.c
@@ -820,6 +820,7 @@ int mlx4_QUERY_DEV_CAP(struct mlx4_dev *dev, struct mlx4_dev_cap *dev_cap)
#define QUERY_DEV_CAP_DIAG_RPRT_PER_PORT    0x9c
#define QUERY_DEV_CAP_FW_REASSIGN_MAC        0x9d
#define QUERY_DEV_CAP_VXLAN            0x9e
+#define QUERY_DEV_CAP_ADD_MAC            0x9f
#define QUERY_DEV_CAP_MAD_DEMUX_OFFSET        0xb0
#define QUERY_DEV_CAP_DMFS_HIGH_RATE_QPN_BASE_OFFSET    0xa8
#define QUERY_DEV_CAP_DMFS_HIGH_RATE_QPN_RANGE_OFFSET    0xac
@@ -1327,6 +1328,11 @@ int mlx4_QUERY_DEV_CAP_wrapper(struct mlx4_dev *dev, int slave,
     field &= 0xd7;
     MLX4_PUT(outbox->buf, field, QUERY_DEV_CAP_VXLAN);

+    /* For guests report additional-mac query not available */
+    MLX4_GET(field, outbox->buf, QUERY_DEV_CAP_ADD_MAC);
+    field &= 0xfb;
+    MLX4_PUT(outbox->buf, field, QUERY_DEV_CAP_ADD_MAC);
+
     /* For guests, disable port BEACON */
     MLX4_GET(field, outbox->buf, QUERY_DEV_CAP_PORT_BEACON_OFFSET);
     field &= 0x7f;
@@ -1427,7 +1433,7 @@ int mlx4_QUERY_PORT_wrapper(struct mlx4_dev *dev, int slave,
      * QUERY_PORT general query.
      */
     if (vhcr->op_modifier || vhcr->in_modifier & ~0xFF)
-        return -EINVAL;
+        return -23;

     vhcr->in_modifier = port;

--
2.27.0

From 5193521e6e26ece2b4a9a69e3031dbf9a46a9a5d Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 13:07:20 +0000
Subject: [PATCH 6/6] MLX4 stats

---
.../net/ethernet/mellanox/mlx4/resource_tracker.c | 15 +++++++++++++++
1 file changed, 15 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
index 1187ef137..9bcaa1517 100644
--- a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
+++ b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
@@ -4560,6 +4560,16 @@ enum {
     BUSY_MAX_RETRIES = 10
};

+static void mlx4_if_stat_zeroes(struct mlx4_counter *if_cnt)
+{
+    u8 temp;
+    temp = if_cnt->counter_mode;
+
+    memset(if_cnt, 0, sizeof(*if_cnt));
+    if_cnt->counter_mode = temp;
+    if_cnt->num_ifc = cpu_to_be32(1);
+}
+
int mlx4_QUERY_IF_STAT_wrapper(struct mlx4_dev *dev, int slave,
                    struct mlx4_vhcr *vhcr,
                    struct mlx4_cmd_mailbox *inbox,
@@ -4569,6 +4579,11 @@ int mlx4_QUERY_IF_STAT_wrapper(struct mlx4_dev *dev, int slave,
     int err;
     int index = vhcr->in_modifier & 0xffff;

+    if (index == MLX4_SINK_COUNTER_INDEX(dev)) {
+        mlx4_if_stat_zeroes(outbox->buf);
+        return 0;
+    }
+
     err = get_res(dev, slave, index, RES_COUNTER, NULL);
     if (err)
         return err;
--
2.27.0

From 14f69590cb72ba3cdadfdad43585e95931db86b7 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Wed, 29 Jul 2020 13:25:12 +0000
Subject: [PATCH 7/7] MLX4 stat wrapper

---
drivers/net/ethernet/mellanox/mlx4/resource_tracker.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
index 9bcaa1517..cc0cb76cd 100644
--- a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
+++ b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
@@ -4576,7 +4576,7 @@ int mlx4_QUERY_IF_STAT_wrapper(struct mlx4_dev *dev, int slave,
                    struct mlx4_cmd_mailbox *outbox,
                    struct mlx4_cmd_info *cmd)
{
-    int err;
+    int err = 0;
     int index = vhcr->in_modifier & 0xffff;

     if (index == MLX4_SINK_COUNTER_INDEX(dev)) {
@@ -4585,8 +4585,10 @@ int mlx4_QUERY_IF_STAT_wrapper(struct mlx4_dev *dev, int slave,
     }

     err = get_res(dev, slave, index, RES_COUNTER, NULL);
-    if (err)
-        return err;
+    if (err) {
+        mlx4_if_stat_zeroes(outbox->buf);
+        err = 0;
+    }

     err = mlx4_DMA_wrapper(dev, slave, vhcr, inbox, outbox, cmd);
     put_res(dev, slave, index, RES_COUNTER);
--
2.27.0

Where did you get his patch? I'm curious if it will be (or already has been) included in an official kernel release.

okrasit · Jun 7, 2021

bbqdt said:
Where did you get his patch? I'm curious if it will be (or already has been) included in an official kernel release.

I did it last summer so, no. It was just a quick way to get it to work. It'd require extensive testing & cleaning up.
For me, it's been working fine.

Code:

<interface type='hostdev' managed='yes'>
    <source>
        <address type='pci' domain='0x0000' bus='0x83' slot='0x00' function='0x1'/>
    </source>
    <mac address='38:d5:48:00:00:01' />
</interface>

Code:

2: enp131s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether f4:52:14:8e:a5:41 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 38:d5:48:00:00:01 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 4     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 5     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 6     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 7     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto

The patch seems ok on 5.12.9 too.

Code:

patching file drivers/net/ethernet/mellanox/mlx4/fw.c
patching file drivers/net/ethernet/mellanox/mlx4/mlx4.h
patching file drivers/net/ethernet/mellanox/mlx4/port.c
Hunk #1 succeeded at 2227 (offset 99 lines).
patching file include/linux/mlx4/device.h
Hunk #1 succeeded at 1494 (offset -21 lines).
patching file drivers/net/ethernet/mellanox/mlx4/port.c
patching file drivers/net/ethernet/mellanox/mlx4/fw.c
patching file drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
patching file drivers/net/ethernet/mellanox/mlx4/resource_tracker.c

mimino · Jun 7, 2021

okrasit said:
Did you assign mac-address to the vf?

I did not, but setting it with 'ip link set dev enp2s0 vf 11 mac 01:02:03:04:05:06' did not make any difference.

okrasit · Jun 7, 2021

mimino said:
I did not, but setting it with 'ip link set dev enp2s0 vf 11 mac 01:02:03:04:05:06' did not make any difference.

does mlx4_core spit anything out in the kernel output, when the vm tries to bring the device online?

mimino · Jun 7, 2021

okrasit said:
does mlx4_core spit anything out in the kernel output, when the vm tries to bring the device online?

Absolutely nothing. Disabling/enabling device in windows device manager produces these two lines however:

Code:

[172949.657691] mlx4_core 0000:02:00.0: Received reset from slave:12
[172950.304146] mlx4_core 0000:02:00.0: vhcr command 0xf00 slave:12 in_param 0x0 in_mod=0x107 op_mod=0x1 failed with error:0, status -28

Mellanox explains it like the following (not sure how it applies here):

When creating more than 125 VLANs and SR-IOV mode is enabled, a kernel warning message will be printed indicating that the native VLAN is created but will not work with RoCE traffic.

okrasit · Jun 7, 2021

mimino said:
Absolutely nothing. Disabling/enabling device in windows device manager produces these two lines however:

Code:

[172949.657691] mlx4_core 0000:02:00.0: Received reset from slave:12 [172950.304146] mlx4_core 0000:02:00.0: vhcr command 0xf00 slave:12 in_param 0x0 in_mod=0x107 op_mod=0x1 failed with error:0, status -28

Mellanox explains it like the following (not sure how it applies here):

Are you using some non-default parameters for the mlx4_core module?

Here's mine:

Code:

msi_x=1 enable_4k_uar=1 enable_qos=1 log_num_mac=7 log_num_mgm_entry_size=-1 log_mtts_per_seg=4

mimino · Jun 7, 2021

okrasit said:
Are you using some non-default parameters for the mlx4_core module?

Here's mine:

Code:

msi_x=1 enable_4k_uar=1 enable_qos=1 log_num_mac=7 log_num_mgm_entry_size=-1 log_mtts_per_seg=4

I have 8 probed vf's and 8 not probed. Probed are passed through to the LXC containers, not probed - to the VM's. Like I said, linux VM's have zero issues with this.

Code:

num_vfs=16,0,0 port_type_array=2,2 probe_vf=8,0,0

Help with ConnectX 3 SR-IOV with Linux host and windows guest via kvm

Member

Active Member

Member

Active Member

Active Member

Active Member

Member

Active Member

Active Member

Active Member

Active Member

Active Member

Member

Member

Member

Active Member

Member

Active Member

Member

Active Member