Borked Flash of a Mellanox CX354A ConnectX-3 - Missing Second Port

Luzer · Oct 22, 2022

prdtabim said:
Did you tried a linux live distribution to see if the ports work ? just to see if the OS isn't the culprit.

Would it work in a live boot or Ubuntu or so I need to install?

Luzer · Oct 22, 2022

mach3.2 said:

Here's a config from my CX534A flashed with Lenovo's FCBT firmware. Hope it's helpful for you.

Firmware link: Mellanox ConnectX-3 Firmware (For Windows) - Lenovo Support US

Bash:

[root@ESXi2:~] /opt/mellanox/bin/mlxconfig q

Device #1:
----------

Device type:    ConnectX3
Device:         mt4099_pciconf0

Configurations:                              Next Boot
         SRIOV_EN                            True(1)
         NUM_OF_VFS                          16
         LINK_TYPE_P1                        ETH(2)
         PHY_TYPE_P1                         XFI(2)
         XFI_MODE_P1                         _10G(0)
         FORCE_MODE_P1                       False(0)
         LINK_TYPE_P2                        ETH(2)
         PHY_TYPE_P2                         XFI(2)
         XFI_MODE_P2                         _10G(0)
         FORCE_MODE_P2                       False(0)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               False(0)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             None(0)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               False(0)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             None(0)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)
         STEER_FORCE_VLAN                    False(0)

Just spitballing, but there's a recovery header on the card that you can short which will put the card into recovery mode for flashing.

Details on how to do it can be found in the thread linked in post #13.

Thanks for the reminder. I'll give it a try in my next visit to the Colo in a few weeks.

fohdeesha · Oct 22, 2022

Luzer said:
Would it work in a live boot or Ubuntu or so I need to install?

just live boot a debian iso no need to install, you should see both ports

prdtabim · Oct 22, 2022

Luzer said:
Would it work in a live boot or Ubuntu or so I need to install?

No need to install.
The ports show as enpxxs0 ( port1 ) and enpxxs0d1 (port2 ). xx varies depending of the pcie enumeration of your mainboard. You can see the detection using dmesg to show de main log of the system and after that use ethtool to see details.

Luzer · Nov 18, 2022

mach3.2 said:

Here's a config from my CX534A flashed with Lenovo's FCBT firmware. Hope it's helpful for you.

Firmware link: Mellanox ConnectX-3 Firmware (For Windows) - Lenovo Support US

Bash:

[root@ESXi2:~] /opt/mellanox/bin/mlxconfig q

Device #1:
----------

Device type:    ConnectX3
Device:         mt4099_pciconf0

Configurations:                              Next Boot
         SRIOV_EN                            True(1)
         NUM_OF_VFS                          16
         LINK_TYPE_P1                        ETH(2)
         PHY_TYPE_P1                         XFI(2)
         XFI_MODE_P1                         _10G(0)
         FORCE_MODE_P1                       False(0)
         LINK_TYPE_P2                        ETH(2)
         PHY_TYPE_P2                         XFI(2)
         XFI_MODE_P2                         _10G(0)
         FORCE_MODE_P2                       False(0)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               False(0)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             None(0)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               False(0)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             None(0)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)
         STEER_FORCE_VLAN                    False(0)

Just spitballing, but there's a recovery header on the card that you can short which will put the card into recovery mode for flashing.

Details on how to do it can be found in the thread linked in post #13.

I was able to recover it using the linked thread but the 1'st port doesn't want to come up. Windows did see it as a interface and I tried two known working cables and ports. I wonder if it's just a bad connector inside the port.

Recovery reflash:

Code:

PS C:\Users\Administrator> flint -d  mt502_pciconf0 -i C:\image\CX3-2_42_5032.bin burn

    Current FW version on flash:  2.42.5000
   New FW version:               2.42.5032

Burning FS2 FW image without signatures - OK
Restoring signature                     - OK

Check after boot:

Code:

~] /opt/mellanox/bin/mlxfwmanager
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  mt4099_pci_cr0
  Port1 MAC:        24be058b97a1
  Port2 MAC:        24be058b97a2
  Versions:         Current        Available
     FW             2.42.5032      N/A
     PXE            3.4.0754       N/A
     UEFI           14.11.0047     N/A

  Status:           No matching image found



~] /opt/mellanox/bin/mlxconfig -d mt4099_pci_cr0 query

Device #1:
----------

Device type:    ConnectX3
Device:         mt4099_pci_cr0

Configurations:                              Next Boot
         SRIOV_EN                            False(0)
         NUM_OF_VFS                          8
         LINK_TYPE_P1                        ETH(2)
         PHY_TYPE_P1                         XFI(2)
         XFI_MODE_P1                         _10G(0)
         FORCE_MODE_P1                       False(0)
         LINK_TYPE_P2                        ETH(2)
         PHY_TYPE_P2                         0
         XFI_MODE_P2                         _10G(0)
         FORCE_MODE_P2                       False(0)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               False(0)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             None(0)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               False(0)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             None(0)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)
         STEER_FORCE_VLAN                    False(0)

card:

prdtabim · Nov 18, 2022

Luzer said:

I was able to recover it using the linked thread but the 1'st port doesn't want to come up. Windows did see it as a interface and I tried two known working cables and ports. I wonder if it's just a bad connector inside the port.

Recovery reflash:

Code:

PS C:\Users\Administrator> flint -d  mt502_pciconf0 -i C:\image\CX3-2_42_5032.bin burn

    Current FW version on flash:  2.42.5000
   New FW version:               2.42.5032

Burning FS2 FW image without signatures - OK
Restoring signature                     - OK

Check after boot:

Code:

~] /opt/mellanox/bin/mlxfwmanager
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  mt4099_pci_cr0
  Port1 MAC:        24be058b97a1
  Port2 MAC:        24be058b97a2
  Versions:         Current        Available
     FW             2.42.5032      N/A
     PXE            3.4.0754       N/A
     UEFI           14.11.0047     N/A

  Status:           No matching image found



~] /opt/mellanox/bin/mlxconfig -d mt4099_pci_cr0 query

Device #1:
----------

Device type:    ConnectX3
Device:         mt4099_pci_cr0

Configurations:                              Next Boot
         SRIOV_EN                            False(0)
         NUM_OF_VFS                          8
         LINK_TYPE_P1                        ETH(2)
         PHY_TYPE_P1                         XFI(2)
         XFI_MODE_P1                         _10G(0)
         FORCE_MODE_P1                       False(0)
         LINK_TYPE_P2                        ETH(2)
         PHY_TYPE_P2                         0
         XFI_MODE_P2                         _10G(0)
         FORCE_MODE_P2                       False(0)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               False(0)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             None(0)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               False(0)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             None(0)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)
         STEER_FORCE_VLAN                    False(0)

card:
View attachment 25512

This value is incorrect.

Code:

PHY_TYPE_P2                         0

Edit. correting this. The result is XFI under normal use.

Code:

1: XAUI
2: XFI
3: SGMII

NVIDIA Networking Docs

docs.nvidia.com

Luzer · Mar 9, 2023

I had some more time to look at this and it seems the second port on the NIC doesnt work when plugged in to the bottom PCIe port, no. 3, on the Dell R730. I tested two Connectx-3 cards in port 3 on the Dell and only a single port worked and plugged them in to other servers and both ports work.

Is there a spec that I'm missing that would tell me about this or maybe the slot isn't fast enough? Defiantly weird.

Search

Borked Flash of a Mellanox CX354A ConnectX-3 - Missing Second Port

Luzer

Member

Luzer

Member

fohdeesha

Kaini Industries

prdtabim

Active Member

Luzer

Member

prdtabim

Active Member

NVIDIA Networking Docs

Luzer

Member

Attachments