Both ports on Mellanox ConnectX-3 FDR/EN have the same PCI function ID | HP 649282-B21

[einstom]

Hi,

I recently started to use the HP Mellanox QSFP+ cards (both 649282-B21 and 764285-B21) but I'm facing the issue that both ports on the NIC have the exact same PCI address (even the same function id). I wanted to use them for a VPP router but this does not work as DPDK can't handle this edge case.

I had a look on the mlxpci tool but there's literally no public documentation available and the exported python pickle is also not really self-explaining.

The issue also appears when I install it in a HP server with FlexLOM slot.

lshw:

[root@chicken-legend ~]# lshw -c network -businfo
Bus info          Device      Class          Description
========================================================
pci@0000:82:00.0  enp130s0    network        MT27500 Family [ConnectX-3]
pci@0000:82:00.0  enp130s0d1  network        Ethernet interface

mlxconfig:

[root@chicken-legend ~]# mlxconfig -d /dev/mst/mt4099_pci_cr0 q

Device #1:
----------

Device type:    ConnectX3
Device:         /dev/mst/mt4099_pci_cr0

Configurations:                                      Next Boot
         SRIOV_EN                                    False(0)
         NUM_OF_VFS                                  16
         WOL_MAGIC_EN_P2                             False(0)
         LINK_TYPE_P1                                ETH(2)
         PHY_TYPE_P1                                 XFI(2)
         XFI_MODE_P1                                 _10G(0)
         FORCE_MODE_P1                               False(0)
         LINK_TYPE_P2                                ETH(2)
         PHY_TYPE_P2                                 XFI(2)
         XFI_MODE_P2                                 _10G(0)
         FORCE_MODE_P2                               False(0)
         LOG_BAR_SIZE                                5
         BOOT_PKEY_P1                                0
         BOOT_PKEY_P2                                0
         BOOT_OPTION_ROM_EN_P1                       False(0)
         BOOT_VLAN_EN_P1                             False(0)
         BOOT_RETRY_CNT_P1                           0
         LEGACY_BOOT_PROTOCOL_P1                     PXE(1)
         BOOT_VLAN_P1                                1
         BOOT_OPTION_ROM_EN_P2                       False(0)
         BOOT_VLAN_EN_P2                             False(0)
         BOOT_RETRY_CNT_P2                           0
         LEGACY_BOOT_PROTOCOL_P2                     PXE(1)
         BOOT_VLAN_P2                                1
         IP_VER_P1                                   IPv4(0)
         IP_VER_P2                                   IPv4(0)
         CQ_TIMESTAMP                                True(1)
         STEER_FORCE_VLAN                            False(0)

Any ideas? Has anyone also faced this issue or solved it?

 

[klui]

You can try one of two things.

1. Use CX4 or newer.
2. Create some VFs, each will have a different id.