Help with ConnectX 3 SR-IOV with Linux host and windows guest via kvm

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

bbqdt

Member
Sep 15, 2019
93
64
18
I’m having trouble getting my ConnectX 3 to work via SR-IOV in windows 10 or server 2019 guests in kvm.

I get error 43 in both from the latest Mellanox or Microsoft drivers.

Literally the exact same xml config files work fine with Linux and FreeBSD guests. I have tried several different configs with no success.

Can anyone post their kvm xml file that does work in this situation? Any suggestions?

mod - feel free to move if this is the wrong forum

Here is one of my config files -

XML:
<domain type='kvm'>
  <name>Windows Server 2019 q35</name>
  <uuid>64d6cd06-33c6-5fc4-f4b8-eb65091b17e7</uuid>
  <memory unit='KiB'>20971520</memory>
  <currentMemory unit='KiB'>20971520</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>14</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='20'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='21'/>
    <vcpupin vcpu='4' cpuset='2'/>
    <vcpupin vcpu='5' cpuset='22'/>
    <vcpupin vcpu='6' cpuset='3'/>
    <vcpupin vcpu='7' cpuset='23'/>
    <vcpupin vcpu='8' cpuset='4'/>
    <vcpupin vcpu='9' cpuset='24'/>
    <vcpupin vcpu='10' cpuset='5'/>
    <vcpupin vcpu='11' cpuset='25'/>
    <vcpupin vcpu='12' cpuset='6'/>
    <vcpupin vcpu='13' cpuset='26'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/64d6cd06-33c6-5fc4-f4b8-eb65091b17e7_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='7' threads='2'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/mnt/user/domains/Windows Server 2019 q35/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/17763.737.190906-2324.rs5_release_svc_refresh_SERVER_EVAL_x64FRE_en-us_1.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:fd:44:fe'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:c5:86:48'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x83' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>
 
Last edited:

llowrey

Active Member
Feb 26, 2018
170
142
43
My configuration is a bit different.

Code:
<interface type='network'>
      <mac address='52:54:00:76:9a:89'/>
      <source network='Mellanox'/>
      <model type='e1000e'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>

Here's my network definition:

Code:
<network>
  <name>Mellanox</name>
  <uuid>a4bcc942-67c9-4942-8e3a-f21863ba7c41</uuid>
  <forward mode='hostdev' managed='yes'>
    <pf dev='eth0p0'/>
  </forward>
</network>

I am using the network above to treat the VFs as a pool and libvirt will issue the next available VF to the guest. It looks like you are statically assigning VFs. Is that right?

I'm using the inbox drivers provided by MS and am running Windows 10 Professional x86_64. I have not tried the OFED drivers
 

bbqdt

Member
Sep 15, 2019
93
64
18
Yes, directly assigning them. Changing to a pool like you have didn’t work.

Can you post your whole xml file? I have a feeling it’s some random flag or something.
 

llowrey

Active Member
Feb 26, 2018
170
142
43
Host: Epyc 7502 (32-core)
OS: CentOS 8.1.1911
Kernel: 4.18.0-147.8.1.el8_1.x86_64
CX-3: MCX354A-FCB_A2-A5 (MT_1090120019)
FW: 2.42.5000
OFED: v5.0-2.1.8

Code:
Device #1:
----------

Device type:    ConnectX3
Device:         /dev/mst/mt4099_pci_cr0

Configurations:                              Next Boot
         SRIOV_EN                            True(1)
         NUM_OF_VFS                          16
         LINK_TYPE_P1                        ETH(2)
         LINK_TYPE_P2                        ETH(2)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               True(1)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               True(1)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             PXE(1)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)
Code:
<domain type='kvm'>
  <name>windows_10-1</name>
  <uuid>9d724977-62e1-4d16-bd41-6096707bad27</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel7.6.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <topology sockets='1' cores='8' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='native' discard='unmap'/>
      <source file='/var/lib/libvirt/images/flash/windows_10-1.qcow2'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/isos/virtio-win-1.9.9.iso'/>
      <target dev='sdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <driver queues='8'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:76:9a:89'/>
      <source network='Mellanox'/>
      <model type='e1000e'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>
 
  • Like
Reactions: bbqdt

llowrey

Active Member
Feb 26, 2018
170
142
43
Also...

/etc/modprobe.d/mlnx.conf
Code:
blacklist mlx4_core
blacklist mlx4_en
blacklist mlx5_core
blacklist mlx5_ib

options mlx4_core num_vfs=16,0,0 port_type_array=2,2 probe_vf=1,0,0
/etc/modprobe.d/vfio.conf
Code:
options vfio-pci ids=1022:7901,9005:028c,8086:1521,15b3:1004
... and the following kernel args:

Code:
rd.driver.pre=vfio-pci pci=realloc iommu=pti
I had to pass pci=realloc to get past an issue with the CX-3. I can't recall now what it was.
 
  • Like
Reactions: bbqdt

efschu3

Active Member
Mar 11, 2019
184
78
28
I run into the similar problem. JFYI:
The problem is the in-kernel driver version v4.0.0, this driver has a bug, which was resolved with v4.3.x OFED. Latest LTS driver for cx3(pro) will only build for kernel up to 5.4, acutal it should work on 5.8 and 5.10 kernels too, but it does not. For now I go with kernel 5.4 until Mellanox fixes their LTS driver.
 
  • Like
Reactions: klui

okrasit

Member
Jun 28, 2019
40
32
18
Here's a dirty patch for the old (5.5.15) mainstream kernel driver.
It might apply successfully to later kernels as well.

Code:
From d4fbe94b4adb16a5cc094a50215dc39496c9fc66 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 05:35:13 +0000
Subject: [PATCH] MLX4 patch for vf

---
drivers/net/ethernet/mellanox/mlx4/fw.c | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/fw.c b/drivers/net/ethernet/mellanox/mlx4/fw.c
index 6e501af0e..2d52e29fd 100644
--- a/drivers/net/ethernet/mellanox/mlx4/fw.c
+++ b/drivers/net/ethernet/mellanox/mlx4/fw.c
@@ -1408,7 +1408,7 @@ int mlx4_QUERY_PORT_wrapper(struct mlx4_dev *dev, int slave,
{
     struct mlx4_priv *priv = mlx4_priv(dev);
     u64 def_mac;
-    u8 port_type;
+    u8 port_type, vlan, field;
     u16 short_field;
     int err;
     int admin_link_state;
@@ -1476,6 +1476,16 @@ int mlx4_QUERY_PORT_wrapper(struct mlx4_dev *dev, int slave,
         short_field = dev->caps.pkey_table_len[vhcr->in_modifier];
         MLX4_PUT(outbox->buf, short_field,
              QUERY_PORT_CUR_MAX_PKEY_OFFSET);
+
+        /* Change the mac table size for the VF */
+        MLX4_GET(field, outbox, QUERY_PORT_MAX_MACVLAN_OFFSET);
+        /* keep the origin vlan of the VF */
+        vlan = field >> 4;
+        /* set the field with the prev vlan and the mac defined quota */
+        field = vlan << 4;
+        field |= ilog2(mlx4_get_port_free_macs(dev,
+            priv->port->port + 1));
+        MLX4_PUT(outbox->buf, field, QUERY_PORT_MAX_MACVLAN_OFFSET);
     }
out:
     return err;
--
2.27.0

From 98cc63cd92b23d8545ad1832d55bcae935fb2f24 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 06:27:20 +0000
Subject: [PATCH 2/2] MLX4 get macs

---
drivers/net/ethernet/mellanox/mlx4/mlx4.h |  1 +
drivers/net/ethernet/mellanox/mlx4/port.c | 57 +++++++++++++++++++++++
2 files changed, 58 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4.h b/drivers/net/ethernet/mellanox/mlx4/mlx4.h
index 527b52e48..71ab05246 100644
--- a/drivers/net/ethernet/mellanox/mlx4/mlx4.h
+++ b/drivers/net/ethernet/mellanox/mlx4/mlx4.h
@@ -740,6 +740,7 @@ struct mlx4_catas_err {

#define MLX4_MAX_MAC_NUM    128
#define MLX4_MAC_TABLE_SIZE    (MLX4_MAX_MAC_NUM << 3)
+#define MLX4_VF_MAC_QUOTA   2

struct mlx4_mac_table {
     __be64            entries[MLX4_MAX_MAC_NUM];
diff --git a/drivers/net/ethernet/mellanox/mlx4/port.c b/drivers/net/ethernet/mellanox/mlx4/port.c
index ba6ac31a3..7ea7b9db9 100644
--- a/drivers/net/ethernet/mellanox/mlx4/port.c
+++ b/drivers/net/ethernet/mellanox/mlx4/port.c
@@ -2128,3 +2128,60 @@ int mlx4_max_tc(struct mlx4_dev *dev)
     return num_tc;
}
EXPORT_SYMBOL(mlx4_max_tc);
+
+static int mlx4_get_port_reserved_mac_num(struct mlx4_dev *mdev, int port)
+{
+    struct mlx4_priv *priv = mlx4_priv(mdev);
+    struct resource_allocator *res_alloc;
+    int reserved;
+
+    if (mlx4_is_slave(mdev))
+        return 0;
+
+    res_alloc = &priv->mfunc.master.res_tracker.res_alloc[RES_MAC];
+
+    reserved = (port > 0) ? res_alloc->res_port_rsvd[port - 1] :
+        res_alloc->res_reserved;
+
+    return reserved;
+}
+
+int mlx4_get_port_max_macs(struct mlx4_dev *mdev, int port)
+{
+    struct mlx4_port_info *info = &mlx4_priv(mdev)->port[port];
+
+    /* The maximum value should considers the reserved macs for the vfs */
+    return info->mac_table.max - mlx4_get_port_reserved_mac_num(mdev, port);
+}
+EXPORT_SYMBOL(mlx4_get_port_max_macs);
+
+int mlx4_get_port_total_macs(struct mlx4_dev *mdev, int port)
+{
+    struct mlx4_port_info *info = &mlx4_priv(mdev)->port[port];
+
+    return info->mac_table.total;
+}
+EXPORT_SYMBOL(mlx4_get_port_total_macs);
+
+int mlx4_get_port_free_macs(struct mlx4_dev *mdev, int port)
+{
+    /* slave will get the free macs (log2) from its master */
+    if (mlx4_is_slave(mdev)) {
+        struct mlx4_port_cap port_cap;
+
+        mlx4_QUERY_PORT(mdev, port, &port_cap);
+        return (1 << port_cap.log_max_macs);
+    }
+
+    return (mlx4_get_port_max_macs(mdev, port) -
+        mlx4_get_port_total_macs(mdev, port));
+}
+EXPORT_SYMBOL(mlx4_get_port_free_macs);
+
+bool mlx4_is_available_mac(struct mlx4_dev *mdev, int port)
+{
+    int free_macs = mlx4_get_port_free_macs(mdev, port);
+
+    return free_macs >= MLX4_VF_MAC_QUOTA;
+}
+EXPORT_SYMBOL(mlx4_is_available_mac);
--
2.27.0

From fd2e45a02d177e71f2b2d6ea509d25bc3757ac1b Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 06:42:41 +0000
Subject: [PATCH 3/3] MLX4 get macs headers

---
include/linux/mlx4/device.h | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h
index 36e412c3d..c5ef47a5f 100644
--- a/include/linux/mlx4/device.h
+++ b/include/linux/mlx4/device.h
@@ -1515,6 +1515,10 @@ int mlx4_phys_to_slave_port(struct mlx4_dev *dev, int slave, int port);
int mlx4_get_base_gid_ix(struct mlx4_dev *dev, int slave, int port);

int mlx4_config_vxlan_port(struct mlx4_dev *dev, __be16 udp_port);
+int mlx4_get_port_max_macs(struct mlx4_dev *mdev, int port);
+int mlx4_get_port_total_macs(struct mlx4_dev *mdev, int port);
+int mlx4_get_port_free_macs(struct mlx4_dev *mdev, int port);
+bool mlx4_is_available_mac(struct mlx4_dev *mdev, int port);
int mlx4_disable_rx_port_check(struct mlx4_dev *dev, bool dis);
int mlx4_config_roce_v2_port(struct mlx4_dev *dev, u16 udp_port);
int mlx4_virt2phy_port_map(struct mlx4_dev *dev, u32 port1, u32 port2);
--
2.27.0

From 77037f4e53732f69001634359086b369d51cf6c0 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 07:16:40 +0000
Subject: [PATCH 4/4] MLX4 fix header

---
drivers/net/ethernet/mellanox/mlx4/port.c | 1 +
1 file changed, 1 insertion(+)

diff --git a/drivers/net/ethernet/mellanox/mlx4/port.c b/drivers/net/ethernet/mellanox/mlx4/port.c
index 7ea7b9db9..616a3b73b 100644
--- a/drivers/net/ethernet/mellanox/mlx4/port.c
+++ b/drivers/net/ethernet/mellanox/mlx4/port.c
@@ -39,6 +39,7 @@

#include "mlx4.h"
#include "mlx4_stats.h"
+#include "fw.h"

#define MLX4_MAC_VALID        (1ull << 63)

--
2.27.0

From 3011ace4fd230fec377e62aa1e9386cf0b1ab9e8 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 12:04:37 +0000
Subject: [PATCH 5/5] MLX4 testing

---
drivers/net/ethernet/mellanox/mlx4/fw.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/fw.c b/drivers/net/ethernet/mellanox/mlx4/fw.c
index 2d52e29fd..90d4bac3c 100644
--- a/drivers/net/ethernet/mellanox/mlx4/fw.c
+++ b/drivers/net/ethernet/mellanox/mlx4/fw.c
@@ -820,6 +820,7 @@ int mlx4_QUERY_DEV_CAP(struct mlx4_dev *dev, struct mlx4_dev_cap *dev_cap)
#define QUERY_DEV_CAP_DIAG_RPRT_PER_PORT    0x9c
#define QUERY_DEV_CAP_FW_REASSIGN_MAC        0x9d
#define QUERY_DEV_CAP_VXLAN            0x9e
+#define QUERY_DEV_CAP_ADD_MAC            0x9f
#define QUERY_DEV_CAP_MAD_DEMUX_OFFSET        0xb0
#define QUERY_DEV_CAP_DMFS_HIGH_RATE_QPN_BASE_OFFSET    0xa8
#define QUERY_DEV_CAP_DMFS_HIGH_RATE_QPN_RANGE_OFFSET    0xac
@@ -1327,6 +1328,11 @@ int mlx4_QUERY_DEV_CAP_wrapper(struct mlx4_dev *dev, int slave,
     field &= 0xd7;
     MLX4_PUT(outbox->buf, field, QUERY_DEV_CAP_VXLAN);

+    /* For guests report additional-mac query not available */
+    MLX4_GET(field, outbox->buf, QUERY_DEV_CAP_ADD_MAC);
+    field &= 0xfb;
+    MLX4_PUT(outbox->buf, field, QUERY_DEV_CAP_ADD_MAC);
+
     /* For guests, disable port BEACON */
     MLX4_GET(field, outbox->buf, QUERY_DEV_CAP_PORT_BEACON_OFFSET);
     field &= 0x7f;
@@ -1427,7 +1433,7 @@ int mlx4_QUERY_PORT_wrapper(struct mlx4_dev *dev, int slave,
      * QUERY_PORT general query.
      */
     if (vhcr->op_modifier || vhcr->in_modifier & ~0xFF)
-        return -EINVAL;
+        return -23;

     vhcr->in_modifier = port;

--
2.27.0

From 5193521e6e26ece2b4a9a69e3031dbf9a46a9a5d Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 13:07:20 +0000
Subject: [PATCH 6/6] MLX4 stats

---
.../net/ethernet/mellanox/mlx4/resource_tracker.c | 15 +++++++++++++++
1 file changed, 15 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
index 1187ef137..9bcaa1517 100644
--- a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
+++ b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
@@ -4560,6 +4560,16 @@ enum {
     BUSY_MAX_RETRIES = 10
};

+static void mlx4_if_stat_zeroes(struct mlx4_counter *if_cnt)
+{
+    u8 temp;
+    temp = if_cnt->counter_mode;
+
+    memset(if_cnt, 0, sizeof(*if_cnt));
+    if_cnt->counter_mode = temp;
+    if_cnt->num_ifc = cpu_to_be32(1);
+}
+
int mlx4_QUERY_IF_STAT_wrapper(struct mlx4_dev *dev, int slave,
                    struct mlx4_vhcr *vhcr,
                    struct mlx4_cmd_mailbox *inbox,
@@ -4569,6 +4579,11 @@ int mlx4_QUERY_IF_STAT_wrapper(struct mlx4_dev *dev, int slave,
     int err;
     int index = vhcr->in_modifier & 0xffff;

+    if (index == MLX4_SINK_COUNTER_INDEX(dev)) {
+        mlx4_if_stat_zeroes(outbox->buf);
+        return 0;
+    }
+
     err = get_res(dev, slave, index, RES_COUNTER, NULL);
     if (err)
         return err;
--
2.27.0

From 14f69590cb72ba3cdadfdad43585e95931db86b7 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Wed, 29 Jul 2020 13:25:12 +0000
Subject: [PATCH 7/7] MLX4 stat wrapper

---
drivers/net/ethernet/mellanox/mlx4/resource_tracker.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
index 9bcaa1517..cc0cb76cd 100644
--- a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
+++ b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
@@ -4576,7 +4576,7 @@ int mlx4_QUERY_IF_STAT_wrapper(struct mlx4_dev *dev, int slave,
                    struct mlx4_cmd_mailbox *outbox,
                    struct mlx4_cmd_info *cmd)
{
-    int err;
+    int err = 0;
     int index = vhcr->in_modifier & 0xffff;

     if (index == MLX4_SINK_COUNTER_INDEX(dev)) {
@@ -4585,8 +4585,10 @@ int mlx4_QUERY_IF_STAT_wrapper(struct mlx4_dev *dev, int slave,
     }

     err = get_res(dev, slave, index, RES_COUNTER, NULL);
-    if (err)
-        return err;
+    if (err) {
+        mlx4_if_stat_zeroes(outbox->buf);
+        err = 0;
+    }

     err = mlx4_DMA_wrapper(dev, slave, vhcr, inbox, outbox, cmd);
     put_res(dev, slave, index, RES_COUNTER);
--
2.27.0
 

mimino

Active Member
Nov 2, 2018
189
70
28
@efschu3 It didn't work for me for some reason. I'm still getting "This device cannot start. (Code 10)" from windoze.
I installed it from mlnx-en-4.9-3.1.5.0-debian10.0-x86_64.iso, would that be the correct (fixed) version?

Code:
# modinfo mlx4_en
filename:       /lib/modules/5.4.106-1-pve/updates/dkms/mlx4_en.ko
version:        4.9-3.1.5
 

efschu3

Active Member
Mar 11, 2019
184
78
28
Code:
 mlx4_en
filename:       /lib/modules/5.4.114-1-pve/updates/dkms/mlx4_en.ko
version:        4.9-3.1.5
for proxmox and latest WinOF (not WinOF2) for winVM.
I did not have a Code10, I had Code43

What does your dmesg say when the driver tries to start?

Btw you need mlx4_core, mlx4_en is the ethernet part for the host (4.3 and later does work)
 
Last edited:

mimino

Active Member
Nov 2, 2018
189
70
28
for proxmox and latest WinOF (not WinOF2) for winVM.
I did not have a Code10, I had Code43
I had Code 43 with kernel driver
What does your dmesg say when the driver tries to start?
What dmesg in windows guest? There is no issues with any of the Linux/BSD guests nor LXC containers, this driver works as well as the kernel one.
Btw you need mlx4_core, mlx4_en is the ethernet part for the host (4.3 and later does work)
Of course.
Code:
# lsmod | grep mlx
mlx4_en               139264  0
mlx4_core             352256  1 mlx4_en
mlx_compat             65536  2 mlx4_core,mlx4_en
 

bbqdt

Member
Sep 15, 2019
93
64
18
Here's a dirty patch for the old (5.5.15) mainstream kernel driver.
It might apply successfully to later kernels as well.

Code:
From d4fbe94b4adb16a5cc094a50215dc39496c9fc66 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 05:35:13 +0000
Subject: [PATCH] MLX4 patch for vf

---
drivers/net/ethernet/mellanox/mlx4/fw.c | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/fw.c b/drivers/net/ethernet/mellanox/mlx4/fw.c
index 6e501af0e..2d52e29fd 100644
--- a/drivers/net/ethernet/mellanox/mlx4/fw.c
+++ b/drivers/net/ethernet/mellanox/mlx4/fw.c
@@ -1408,7 +1408,7 @@ int mlx4_QUERY_PORT_wrapper(struct mlx4_dev *dev, int slave,
{
     struct mlx4_priv *priv = mlx4_priv(dev);
     u64 def_mac;
-    u8 port_type;
+    u8 port_type, vlan, field;
     u16 short_field;
     int err;
     int admin_link_state;
@@ -1476,6 +1476,16 @@ int mlx4_QUERY_PORT_wrapper(struct mlx4_dev *dev, int slave,
         short_field = dev->caps.pkey_table_len[vhcr->in_modifier];
         MLX4_PUT(outbox->buf, short_field,
              QUERY_PORT_CUR_MAX_PKEY_OFFSET);
+
+        /* Change the mac table size for the VF */
+        MLX4_GET(field, outbox, QUERY_PORT_MAX_MACVLAN_OFFSET);
+        /* keep the origin vlan of the VF */
+        vlan = field >> 4;
+        /* set the field with the prev vlan and the mac defined quota */
+        field = vlan << 4;
+        field |= ilog2(mlx4_get_port_free_macs(dev,
+            priv->port->port + 1));
+        MLX4_PUT(outbox->buf, field, QUERY_PORT_MAX_MACVLAN_OFFSET);
     }
out:
     return err;
--
2.27.0

From 98cc63cd92b23d8545ad1832d55bcae935fb2f24 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 06:27:20 +0000
Subject: [PATCH 2/2] MLX4 get macs

---
drivers/net/ethernet/mellanox/mlx4/mlx4.h |  1 +
drivers/net/ethernet/mellanox/mlx4/port.c | 57 +++++++++++++++++++++++
2 files changed, 58 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4.h b/drivers/net/ethernet/mellanox/mlx4/mlx4.h
index 527b52e48..71ab05246 100644
--- a/drivers/net/ethernet/mellanox/mlx4/mlx4.h
+++ b/drivers/net/ethernet/mellanox/mlx4/mlx4.h
@@ -740,6 +740,7 @@ struct mlx4_catas_err {

#define MLX4_MAX_MAC_NUM    128
#define MLX4_MAC_TABLE_SIZE    (MLX4_MAX_MAC_NUM << 3)
+#define MLX4_VF_MAC_QUOTA   2

struct mlx4_mac_table {
     __be64            entries[MLX4_MAX_MAC_NUM];
diff --git a/drivers/net/ethernet/mellanox/mlx4/port.c b/drivers/net/ethernet/mellanox/mlx4/port.c
index ba6ac31a3..7ea7b9db9 100644
--- a/drivers/net/ethernet/mellanox/mlx4/port.c
+++ b/drivers/net/ethernet/mellanox/mlx4/port.c
@@ -2128,3 +2128,60 @@ int mlx4_max_tc(struct mlx4_dev *dev)
     return num_tc;
}
EXPORT_SYMBOL(mlx4_max_tc);
+
+static int mlx4_get_port_reserved_mac_num(struct mlx4_dev *mdev, int port)
+{
+    struct mlx4_priv *priv = mlx4_priv(mdev);
+    struct resource_allocator *res_alloc;
+    int reserved;
+
+    if (mlx4_is_slave(mdev))
+        return 0;
+
+    res_alloc = &priv->mfunc.master.res_tracker.res_alloc[RES_MAC];
+
+    reserved = (port > 0) ? res_alloc->res_port_rsvd[port - 1] :
+        res_alloc->res_reserved;
+
+    return reserved;
+}
+
+int mlx4_get_port_max_macs(struct mlx4_dev *mdev, int port)
+{
+    struct mlx4_port_info *info = &mlx4_priv(mdev)->port[port];
+
+    /* The maximum value should considers the reserved macs for the vfs */
+    return info->mac_table.max - mlx4_get_port_reserved_mac_num(mdev, port);
+}
+EXPORT_SYMBOL(mlx4_get_port_max_macs);
+
+int mlx4_get_port_total_macs(struct mlx4_dev *mdev, int port)
+{
+    struct mlx4_port_info *info = &mlx4_priv(mdev)->port[port];
+
+    return info->mac_table.total;
+}
+EXPORT_SYMBOL(mlx4_get_port_total_macs);
+
+int mlx4_get_port_free_macs(struct mlx4_dev *mdev, int port)
+{
+    /* slave will get the free macs (log2) from its master */
+    if (mlx4_is_slave(mdev)) {
+        struct mlx4_port_cap port_cap;
+
+        mlx4_QUERY_PORT(mdev, port, &port_cap);
+        return (1 << port_cap.log_max_macs);
+    }
+
+    return (mlx4_get_port_max_macs(mdev, port) -
+        mlx4_get_port_total_macs(mdev, port));
+}
+EXPORT_SYMBOL(mlx4_get_port_free_macs);
+
+bool mlx4_is_available_mac(struct mlx4_dev *mdev, int port)
+{
+    int free_macs = mlx4_get_port_free_macs(mdev, port);
+
+    return free_macs >= MLX4_VF_MAC_QUOTA;
+}
+EXPORT_SYMBOL(mlx4_is_available_mac);
--
2.27.0

From fd2e45a02d177e71f2b2d6ea509d25bc3757ac1b Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 06:42:41 +0000
Subject: [PATCH 3/3] MLX4 get macs headers

---
include/linux/mlx4/device.h | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h
index 36e412c3d..c5ef47a5f 100644
--- a/include/linux/mlx4/device.h
+++ b/include/linux/mlx4/device.h
@@ -1515,6 +1515,10 @@ int mlx4_phys_to_slave_port(struct mlx4_dev *dev, int slave, int port);
int mlx4_get_base_gid_ix(struct mlx4_dev *dev, int slave, int port);

int mlx4_config_vxlan_port(struct mlx4_dev *dev, __be16 udp_port);
+int mlx4_get_port_max_macs(struct mlx4_dev *mdev, int port);
+int mlx4_get_port_total_macs(struct mlx4_dev *mdev, int port);
+int mlx4_get_port_free_macs(struct mlx4_dev *mdev, int port);
+bool mlx4_is_available_mac(struct mlx4_dev *mdev, int port);
int mlx4_disable_rx_port_check(struct mlx4_dev *dev, bool dis);
int mlx4_config_roce_v2_port(struct mlx4_dev *dev, u16 udp_port);
int mlx4_virt2phy_port_map(struct mlx4_dev *dev, u32 port1, u32 port2);
--
2.27.0

From 77037f4e53732f69001634359086b369d51cf6c0 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 07:16:40 +0000
Subject: [PATCH 4/4] MLX4 fix header

---
drivers/net/ethernet/mellanox/mlx4/port.c | 1 +
1 file changed, 1 insertion(+)

diff --git a/drivers/net/ethernet/mellanox/mlx4/port.c b/drivers/net/ethernet/mellanox/mlx4/port.c
index 7ea7b9db9..616a3b73b 100644
--- a/drivers/net/ethernet/mellanox/mlx4/port.c
+++ b/drivers/net/ethernet/mellanox/mlx4/port.c
@@ -39,6 +39,7 @@

#include "mlx4.h"
#include "mlx4_stats.h"
+#include "fw.h"

#define MLX4_MAC_VALID        (1ull << 63)

--
2.27.0

From 3011ace4fd230fec377e62aa1e9386cf0b1ab9e8 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 12:04:37 +0000
Subject: [PATCH 5/5] MLX4 testing

---
drivers/net/ethernet/mellanox/mlx4/fw.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/fw.c b/drivers/net/ethernet/mellanox/mlx4/fw.c
index 2d52e29fd..90d4bac3c 100644
--- a/drivers/net/ethernet/mellanox/mlx4/fw.c
+++ b/drivers/net/ethernet/mellanox/mlx4/fw.c
@@ -820,6 +820,7 @@ int mlx4_QUERY_DEV_CAP(struct mlx4_dev *dev, struct mlx4_dev_cap *dev_cap)
#define QUERY_DEV_CAP_DIAG_RPRT_PER_PORT    0x9c
#define QUERY_DEV_CAP_FW_REASSIGN_MAC        0x9d
#define QUERY_DEV_CAP_VXLAN            0x9e
+#define QUERY_DEV_CAP_ADD_MAC            0x9f
#define QUERY_DEV_CAP_MAD_DEMUX_OFFSET        0xb0
#define QUERY_DEV_CAP_DMFS_HIGH_RATE_QPN_BASE_OFFSET    0xa8
#define QUERY_DEV_CAP_DMFS_HIGH_RATE_QPN_RANGE_OFFSET    0xac
@@ -1327,6 +1328,11 @@ int mlx4_QUERY_DEV_CAP_wrapper(struct mlx4_dev *dev, int slave,
     field &= 0xd7;
     MLX4_PUT(outbox->buf, field, QUERY_DEV_CAP_VXLAN);

+    /* For guests report additional-mac query not available */
+    MLX4_GET(field, outbox->buf, QUERY_DEV_CAP_ADD_MAC);
+    field &= 0xfb;
+    MLX4_PUT(outbox->buf, field, QUERY_DEV_CAP_ADD_MAC);
+
     /* For guests, disable port BEACON */
     MLX4_GET(field, outbox->buf, QUERY_DEV_CAP_PORT_BEACON_OFFSET);
     field &= 0x7f;
@@ -1427,7 +1433,7 @@ int mlx4_QUERY_PORT_wrapper(struct mlx4_dev *dev, int slave,
      * QUERY_PORT general query.
      */
     if (vhcr->op_modifier || vhcr->in_modifier & ~0xFF)
-        return -EINVAL;
+        return -23;

     vhcr->in_modifier = port;

--
2.27.0

From 5193521e6e26ece2b4a9a69e3031dbf9a46a9a5d Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Tue, 28 Jul 2020 13:07:20 +0000
Subject: [PATCH 6/6] MLX4 stats

---
.../net/ethernet/mellanox/mlx4/resource_tracker.c | 15 +++++++++++++++
1 file changed, 15 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
index 1187ef137..9bcaa1517 100644
--- a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
+++ b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
@@ -4560,6 +4560,16 @@ enum {
     BUSY_MAX_RETRIES = 10
};

+static void mlx4_if_stat_zeroes(struct mlx4_counter *if_cnt)
+{
+    u8 temp;
+    temp = if_cnt->counter_mode;
+
+    memset(if_cnt, 0, sizeof(*if_cnt));
+    if_cnt->counter_mode = temp;
+    if_cnt->num_ifc = cpu_to_be32(1);
+}
+
int mlx4_QUERY_IF_STAT_wrapper(struct mlx4_dev *dev, int slave,
                    struct mlx4_vhcr *vhcr,
                    struct mlx4_cmd_mailbox *inbox,
@@ -4569,6 +4579,11 @@ int mlx4_QUERY_IF_STAT_wrapper(struct mlx4_dev *dev, int slave,
     int err;
     int index = vhcr->in_modifier & 0xffff;

+    if (index == MLX4_SINK_COUNTER_INDEX(dev)) {
+        mlx4_if_stat_zeroes(outbox->buf);
+        return 0;
+    }
+
     err = get_res(dev, slave, index, RES_COUNTER, NULL);
     if (err)
         return err;
--
2.27.0

From 14f69590cb72ba3cdadfdad43585e95931db86b7 Mon Sep 17 00:00:00 2001
From: Mikko Hissa <okrasit@gmail.com>
Date: Wed, 29 Jul 2020 13:25:12 +0000
Subject: [PATCH 7/7] MLX4 stat wrapper

---
drivers/net/ethernet/mellanox/mlx4/resource_tracker.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
index 9bcaa1517..cc0cb76cd 100644
--- a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
+++ b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
@@ -4576,7 +4576,7 @@ int mlx4_QUERY_IF_STAT_wrapper(struct mlx4_dev *dev, int slave,
                    struct mlx4_cmd_mailbox *outbox,
                    struct mlx4_cmd_info *cmd)
{
-    int err;
+    int err = 0;
     int index = vhcr->in_modifier & 0xffff;

     if (index == MLX4_SINK_COUNTER_INDEX(dev)) {
@@ -4585,8 +4585,10 @@ int mlx4_QUERY_IF_STAT_wrapper(struct mlx4_dev *dev, int slave,
     }

     err = get_res(dev, slave, index, RES_COUNTER, NULL);
-    if (err)
-        return err;
+    if (err) {
+        mlx4_if_stat_zeroes(outbox->buf);
+        err = 0;
+    }

     err = mlx4_DMA_wrapper(dev, slave, vhcr, inbox, outbox, cmd);
     put_res(dev, slave, index, RES_COUNTER);
--
2.27.0
Where did you get his patch? I'm curious if it will be (or already has been) included in an official kernel release.
 

okrasit

Member
Jun 28, 2019
40
32
18
Where did you get his patch? I'm curious if it will be (or already has been) included in an official kernel release.
I did it last summer so, no. It was just a quick way to get it to work. It'd require extensive testing & cleaning up.
For me, it's been working fine.


Code:
<interface type='hostdev' managed='yes'>
    <source>
        <address type='pci' domain='0x0000' bus='0x83' slot='0x00' function='0x1'/>
    </source>
    <mac address='38:d5:48:00:00:01' />
</interface>
Code:
2: enp131s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether f4:52:14:8e:a5:41 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 38:d5:48:00:00:01 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 4     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 5     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 6     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 7     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
1623075650798.png

The patch seems ok on 5.12.9 too.
Code:
patching file drivers/net/ethernet/mellanox/mlx4/fw.c
patching file drivers/net/ethernet/mellanox/mlx4/mlx4.h
patching file drivers/net/ethernet/mellanox/mlx4/port.c
Hunk #1 succeeded at 2227 (offset 99 lines).
patching file include/linux/mlx4/device.h
Hunk #1 succeeded at 1494 (offset -21 lines).
patching file drivers/net/ethernet/mellanox/mlx4/port.c
patching file drivers/net/ethernet/mellanox/mlx4/fw.c
patching file drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
patching file drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
 
Last edited:

okrasit

Member
Jun 28, 2019
40
32
18
I did not, but setting it with 'ip link set dev enp2s0 vf 11 mac 01:02:03:04:05:06' did not make any difference.
does mlx4_core spit anything out in the kernel output, when the vm tries to bring the device online?
 

mimino

Active Member
Nov 2, 2018
189
70
28
does mlx4_core spit anything out in the kernel output, when the vm tries to bring the device online?
Absolutely nothing. Disabling/enabling device in windows device manager produces these two lines however:
Code:
[172949.657691] mlx4_core 0000:02:00.0: Received reset from slave:12
[172950.304146] mlx4_core 0000:02:00.0: vhcr command 0xf00 slave:12 in_param 0x0 in_mod=0x107 op_mod=0x1 failed with error:0, status -28
Mellanox explains it like the following (not sure how it applies here):
When creating more than 125 VLANs and SR-IOV mode is enabled, a kernel warning message will be printed indicating that the native VLAN is created but will not work with RoCE traffic.
 

okrasit

Member
Jun 28, 2019
40
32
18
Absolutely nothing. Disabling/enabling device in windows device manager produces these two lines however:
Code:
[172949.657691] mlx4_core 0000:02:00.0: Received reset from slave:12
[172950.304146] mlx4_core 0000:02:00.0: vhcr command 0xf00 slave:12 in_param 0x0 in_mod=0x107 op_mod=0x1 failed with error:0, status -28
Mellanox explains it like the following (not sure how it applies here):
Are you using some non-default parameters for the mlx4_core module?

Here's mine:
Code:
msi_x=1 enable_4k_uar=1 enable_qos=1 log_num_mac=7 log_num_mgm_entry_size=-1 log_mtts_per_seg=4
 

mimino

Active Member
Nov 2, 2018
189
70
28
Are you using some non-default parameters for the mlx4_core module?

Here's mine:
Code:
msi_x=1 enable_4k_uar=1 enable_qos=1 log_num_mac=7 log_num_mgm_entry_size=-1 log_mtts_per_seg=4
I have 8 probed vf's and 8 not probed. Probed are passed through to the LXC containers, not probed - to the VM's. Like I said, linux VM's have zero issues with this.
Code:
num_vfs=16,0,0 port_type_array=2,2 probe_vf=8,0,0