Mellanox ConnectX3 card to ethernet mode issues.

pnunn · Jul 7, 2022

Hi guys,

I'm struggling to get a Mellanox card I've just received in a second hand server to switch to ethernet mode.

I've followed the bouncing ball from the blog post and have managed to install mst and get that running, but, every time I try and change the port type I'm hitting a problem with it not being able to find he default configuration.

Bash:

root@kubuntu:/sys/devices# mlxconfig -d /dev/mst/mt4099_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

Device #1:
----------

Device type:    ConnectX3       
Device:         /dev/mst/mt4099_pciconf0

Configurations:                              Next Boot       New
         LINK_TYPE_P1                        N/A             ETH(2)         
         LINK_TYPE_P2                        N/A             ETH(2)         

 Apply new Configuration? (y/n) [n] : y
Applying... Failed!
-E- Failed to set configuration: Failed to get default parameters: Operation not supported please specify all the parameters for VPI settings.

I've scoured the goog but so far haven't come up with a solution.

Added difficulty factor is that this is in a box running xcp-ng and so I'm doing all the changes from a live kubuntu disk so have to download and re-install everything after each reboot

What fun...

Any ideas how to proceed?

prdtabim · Jul 7, 2022

Looks like the effect of a very old firmware.
Run flint to verify the firmware version

Code:

flint -d /dev/mst/mt4099_pciconf0 q                                     
                   
Image type:            FS2
FW Version:            2.42.5000
FW Release Date:       5.9.2017
Product Version:       02.42.50.00
Rom Info:              type=UEFI version=14.11.45 cpu=AMD64
                       type=PXE version=3.4.752
Device ID:             4099
Description:           Node             Port1            Port2            Sys image
GUIDs:                 ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
MACs:                                       e41d2dxxxxxx     e41d2dxxxxxx
VSD:                 
PSID:                  MT_1270114023

2.42.5000 is the last firmware for Connectx-3.

Just to compare ( my card is Connectx-3 EN since it doesn't support infiniband so thats not have PORT_TYPE in settings )

Code:

mlxconfig -d /dev/mst/mt4099_pciconf0 q

Device #1:
----------

Device type:    ConnectX3     
Device:         /dev/mst/mt4099_pciconf0

Configurations:                              Next Boot
         SRIOV_EN                            False(0)      
         NUM_OF_VFS                          8             
         WOL_MAGIC_EN_P1                     True(1)       
         LOG_BAR_SIZE                        3             
         BOOT_OPTION_ROM_EN_P1               True(1)       
         BOOT_VLAN_EN_P1                     False(0)      
         BOOT_RETRY_CNT_P1                   0             
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)        
         BOOT_VLAN_P1                        1             
         BOOT_OPTION_ROM_EN_P2               True(1)       
         BOOT_VLAN_EN_P2                     False(0)      
         BOOT_RETRY_CNT_P2                   0             
         LEGACY_BOOT_PROTOCOL_P2             PXE(1)        
         BOOT_VLAN_P2                        1             
         IP_VER_P1                           IPv4(0)       
         IP_VER_P2                           IPv4(0)       
         CQ_TIMESTAMP                        True(1)

If the config is empty even after firmware upgrade run

Code:

mlxconfig -d /dev/mst/mt4099_pciconf0 r

to initialize the settings with default values. After that change the PORT_TYPE.

pnunn · Jul 7, 2022

Wow, that was quick, thanks @prbtabin.

OK, the firmware certainly is old, but its the latest I could find for the PSID on the card searching nvidia's web site.

Bash:

flint -d /dev/mst/mt4099_pciconf0  q
Image type:            FS2
FW Version:            2.32.5180
FW Release Date:       6.10.2014
Device ID:             4099
Description:           Node             Port1            Port2            Sys image
GUIDs:                 0002c90300fbb1a0 0002c90300fbb1a1 0002c90300fbb1a2 0002c90300fbb1a3
MACs:                                       0002c9fbb1a0     0002c9fbb1a1
VSD:                   
PSID:                  IBM1020110028

Should I grab a later one and try? If so, which one would you suggest?

prdtabim · Jul 7, 2022

pnunn said:
Wow, that was quick, thanks @prbtabin.

OK, the firmware certainly is old, but its the latest I could find for the PSID on the card searching nvidia's web site.

Bash:

flint -d /dev/mst/mt4099_pciconf0 q Image type: FS2 FW Version: 2.32.5180 FW Release Date: 6.10.2014 Device ID: 4099 Description: Node Port1 Port2 Sys image GUIDs: 0002c90300fbb1a0 0002c90300fbb1a1 0002c90300fbb1a2 0002c90300fbb1a3 MACs: 0002c9fbb1a0 0002c9fbb1a1 VSD: PSID: IBM1020110028

Should I grab a later one and try? If so, which one would you suggest?

Looking here Mellanox Firmware Update - Lenovo Support BR
Your card looks like an IBM branded "ConnectX-3 VPI Dual-port QSFP FDR10/10GbE HCA" .
This page have a firmware that isn't the last but is newer than the actual.
Firmware 2.35.5100 : https://download.lenovo.com/servers/mig/CMA/XSA/0631o/0/mlnx-lnvgy_fw_nic_2.35.5100_linux_32-64.bin
Instructions: https://download.lenovo.com/servers/mig/CMA/XSA/0631o/0/mlnx-lnvgy_fw_nic_2.35.5100_linux_32-64.txt

prdtabim · Jul 7, 2022

pnunn said:
Wow, that was quick, thanks @prbtabin.

OK, the firmware certainly is old, but its the latest I could find for the PSID on the card searching nvidia's web site.

Bash:

flint -d /dev/mst/mt4099_pciconf0 q Image type: FS2 FW Version: 2.32.5180 FW Release Date: 6.10.2014 Device ID: 4099 Description: Node Port1 Port2 Sys image GUIDs: 0002c90300fbb1a0 0002c90300fbb1a1 0002c90300fbb1a2 0002c90300fbb1a3 MACs: 0002c9fbb1a0 0002c9fbb1a1 VSD: PSID: IBM1020110028

Should I grab a later one and try? If so, which one would you suggest?

At nvidia website the last is 2.34.5000 .

Archive of Firmware and Software for Lenovo InfiniBand Adapter Cards

The following table provides earlier versions of firmware for IBM InfiniBand adapter cards.

network.nvidia.com

https://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_34_5000-95Y3456_Ax.bin.zip

Sean Ho · Jul 7, 2022

Remember that all CX353/354 are cross-flashable to each other, but with a change in PSID. You'd use the firmware associated with the PSID for an FCBT card.

pnunn · Jul 7, 2022

Wow.. what a marathon

Thank you VERY much prbtabim. I managed to get the latest firmware to load onto both cards and then managed to set them both to ethernet mode.

TrueNAS now has the card in its list, XCP-NG however continues to be grumpy with no sign of it showing up.

The switch is playing up as well, so now back to the supplier over that.

Thanks Sean, so I guess as long as I'm not using the PSID anywhere, it shouldn't matter?

i386 · Jul 8, 2022

prdtabim said:
2.42.5000 is the last firmware for Connectx-3.

There is newer firmware (2.42.5032), but so far only for lenovo/ibm branded skus. I flashed some of my cx-3 with that firmware

Link with links and flashing steps: https://forums.servethehome.com/ind...x-3-custom-firmware-package.18476/post-291259

Sean Ho · Jul 8, 2022

The PSID is just like a more specific model number including whether it's allowed to run at FDR speeds or not. When you crossflash, just include the -allow_psid_change flag to mstflint. It's all in the flashing instructions on the linked STH thread.

Search

Mellanox ConnectX3 card to ethernet mode issues.

pnunn

New Member

prdtabim

Active Member

pnunn

New Member

prdtabim

Active Member

prdtabim

Active Member

Archive of Firmware and Software for Lenovo InfiniBand Adapter Cards

Sean Ho

seanho.com

pnunn

New Member

i386

Well-Known Member

Sean Ho

seanho.com