Borked Flash of a Mellanox CX354A ConnectX-3 - Missing Second Port

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Luzer

Luzer

Member
Mar 1, 2015
40
3
8
Midwest
While trying to cross flash my CX354A I think I disabled the second port on the card by flashing the single port firmware. When I run '/opt/mellanox/bin/mlxfwmanager' on my esxi server I only get 'Device #1' and before the flash, on the original FW, I got Device #1 and #2'.

Any suggestions on how to flash the dual port version of the generic FW? I cant make out what FW I should use as I tried fw-ConnectX3-rel-2_42_5000-MCX354A-QCB_Ax-FlexBoot-3.4.752.bin which I thought was the two port FW but esx only has one device listed after I flashed it. Is it the FCB or the QCB version of the FW?

The card is a 'Mellanox CX354A ConnectX-3 QDR QSFP+ 40+10 Dual Port Server HBA 050-0050-02' from ebay.

Thanks.

Running mlxfwmanger
Code: 
[root@esx:~] /opt/mellanox/bin/mlxfwmanager
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  mt4099_pci_cr0
  Port1 MAC:        e41d2d4d5d61
  Port2 MAC:        e41d2d4d5d62
  Versions:         Current        Available
     FW             2.42.5000      N/A
     PXE            3.4.0752       N/A

  Status:           No matching image found

esx mellanox nic8 down but 40gbe vmnic1000802 up.

Code: 
[root@dell-r730:~]  esxcli network nic list
Name          PCI Device    Driver    Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description
------------  ------------  --------  ------------  -----------  -----  ------  -----------------  ----  -------------------------------------------------------
vmnic0        0000:01:00.0  ntg3      Up            Up            1000  Full    00.00.00.00.00.00  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic1        0000:01:00.1  ntg3      Up            Up            1000  Full    00.00.00.00.00.00  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic1000802  0000:86:00.0  nmlx4_en  Up            Up           40000  Full    00.00.00.00.00.00  1500  Mellanox Technologies MT27500 Family [ConnectX-3]
vmnic2        0000:02:00.0  ntg3      Up            Up            1000  Full    00.00.00.00.00.00  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic3        0000:02:00.1  ntg3      Up            Up            1000  Full    00.00.00.00.00.00  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic4        0000:83:00.0  igb       Up            Up            1000  Full    00.00.00.00.00.00  1500  Intel Corporation 82576 Gigabit Network Connection
vmnic5        0000:83:00.1  igb       Up            Up            1000  Full    00.00.00.00.00.00  1500  Intel Corporation 82576 Gigabit Network Connection
vmnic6        0000:84:00.0  igb       Up            Up            1000  Full    00.00.00.00.00.00  1500  Intel Corporation 82576 Gigabit Network Connection
vmnic7        0000:84:00.1  igb       Up            Up            1000  Full    00.00.00.00.00.00  1500  Intel Corporation 82576 Gigabit Network Connection
vmnic8        0000:86:00.0  nmlx4_en  Up            Down             0  Half    00.00.00.00.00.00  1500  Mellanox Technologies MT27500 Family [ConnectX-3]
flint query before flash:
Code: 
Image type:            FS2
FW Version:            2.40.5030
FW Release Date:       4.1.2017
MIC Version:           2.0.0
Config Sectors:        1
PRS Name:              cx3-1_MCX354A_qdr.prs
Product Version:       02.40.50.30
Rom Info:              type=PXE version=3.4.746
Device ID:             4099
Description:           Node             Port1            Port2            Sys image
GUIDs:                 e41xxxxxx60 e41xxxxx61 e41xxxxxxxxx62 e41xxxxxxxxx63
MACs:                                       e41d2d4d5d61     e41d2d4d5d62
VSD:                  
PSID:                  ISL1090110018
 
Last edited:
Luzer

Luzer

Member
Mar 1, 2015
40
3
8
Midwest
prdtabim said:
Hi
Try "mlxconfig q" to see the internal configs.
Most likely the card have the port 2 set as IB.
You may need to set the ports as ETH. see here Port Type Management - SLES15 SP4 - NVIDIA Networking Docs
Click to expand...
Good suggestion.

Looks like it set to ethernet:
Code: 
[root@dell-r730:~] /opt/mellanox/bin/mlxconfig q

Device #1:
----------

Device type:    ConnectX3
Device:         mt4099_pciconf0

Configurations:                              Next Boot
         SRIOV_EN                            False(0)
         NUM_OF_VFS                          8
         LINK_TYPE_P1                        ETH(2)
         LINK_TYPE_P2                        ETH(2)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               False(0)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               False(0)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             None(0)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)
 
Luzer

Luzer

Member
Mar 1, 2015
40
3
8
Midwest
prdtabim said:
If I remember correctly there are parameters missing in the config.
See Examples of mlxconfig Usage - MFT v4.13.0 - NVIDIA Networking Docs
Try to reset the config to defaults.
Code: 
mlxconfig -d <device> reset

Power cycle after this.
Edit. see this too: Supported Configurations and Parameters - MFT v4.11.0 - NVIDIA Networking Docs
Click to expand...
Didn't change. Thanks though
Code: 
[root@esx:~] esxcli network nic list | grep -i mel
vmnic1000802  0000:86:00.0  nmlx4_en  Up            Up           40000  Full    xxx.xxx.xx.xx.xx  1500  Mellanox Technologies MT27500 Family [ConnectX-3]
vmnic8        0000:86:00.0  nmlx4_en  Up            Down             0  Half    xxx.xx.xx.xx.xx  1500  Mellanox Technologies MT27500 Family [ConnectX-3]
 
Luzer

Luzer

Member
Mar 1, 2015
40
3
8
Midwest
Stephan said:
If all else fails, try attached firmware.

FW Version: 2.42.5032
UEFI version=14.11.47 cpu=AMD64
PXE version=3.4.754
Click to expand...
Trying now.

Just rebooted and only one device shows. Thanks for the help.
Code: 
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  mt4099_pci_cr0
  Port1 MAC:        e4xxxxxxx1
  Port2 MAC:        e4xxxxxxxx2
  Versions:         Current        Available
     FW             2.42.5032      N/A
     PXE            3.4.0754       N/A
     UEFI           14.11.0047     N/A

  Status:           No matching image found
Flahed ok
Code: 
[root@esx:~] /opt/mellanox/bin/flint -d mt4099_pci_cr0 -i /tmp/CX3-2_42_5032.bin -allow_psid_change burn

    Current FW version on flash:  2.42.5000
    New FW version:               2.42.5032

Burning FS2 FW image without signatures - OK
Restoring signature                     - OK
 
Last edited:
I

i386

Well-Known Member
Mar 18, 2016
4,245
1,546
113
34
Germany
Did you backed up the original firmware before flashing a new firmware?
There are multiple skus that start with "CX354A", one of them is the cx-3 pro:
cx-33.jpg
Luzer said:
Is it the FCB or the QCB version of the FW?
Click to expand...
FCB is the fdr version (56GBit/s infiniband)
QCB is the qdr version (40GBit/s infiniband)
 
Luzer

Luzer

Member
Mar 1, 2015
40
3
8
Midwest
i386 said:
Did you backed up the original firmware before flashing a new firmware?
There are multiple skus that start with "CX354A", one of them is the cx-3 pro:
View attachment 24966

FCB is the fdr version (56GBit/s infiniband)
QCB is the qdr version (40GBit/s infiniband)
Click to expand...
I did back it up. I'll re-flash with it to see if the other device some back up. Not sure why I didnt think of that.

I believe that this is the 40GB QCB card. Looking for a photo of the card as the server is at at colo.
 
Luzer

Luzer

Member
Mar 1, 2015
40
3
8
Midwest
i386 said:
Did you backed up the original firmware before flashing a new firmware?
There are multiple skus that start with "CX354A", one of them is the cx-3 pro:
View attachment 24966

FCB is the fdr version (56GBit/s infiniband)
QCB is the qdr version (40GBit/s infiniband)
Click to expand...

Orignal Firmware only has one device. Did I brick the other port?
Code: 
[root@esx:~]  /opt/mellanox/bin/mlxfwmanager
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      050-0050-02_01
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; QDR IB (40Gb/s) and 10GigE; PCIe3.0x8 8GT/s; RoHS R6
  PSID:             ISL1090110018
  PCI Device Name:  mt4099_pci_cr0
  Port1 MAC:        xxxxxxxxxxx
  Port2 MAC:        xxxxxxxxxx
  Versions:         Current        Available
     FW             2.40.5030      N/A
     PXE            3.4.0746       N/A

  Status:           No matching image found
Downgrade worked
Code: 
[root@esx:~] /opt/mellanox/bin/flint -d mt4099_pci_cr0 -i /tmp/orig_firmware.mlx -allow_psid_change burn

    Current FW version on flash:  2.42.5032
    New FW version:               2.40.5030

    Note: The new FW version is older than the current FW version on flash.

Do you want to continue ? (y/n) [n] : y


    You are about to replace current PSID on flash - "MT_1090120019" with a different PSID - "ISL1090110018".
    Note: It is highly recommended not to change the PSID.

Do you want to continue ? (y/n) [n] : y
Burning FS2 FW image without signatures - OK
Restoring signature                     - OK
 
P

prdtabim

Active Member
Jan 29, 2022
173
67
28
Luzer said:
Orignal Firmware only has one device. Did I brick the other port?
Code: 
[root@esx:~]  /opt/mellanox/bin/mlxfwmanager
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      050-0050-02_01
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; QDR IB (40Gb/s) and 10GigE; PCIe3.0x8 8GT/s; RoHS R6
  PSID:             ISL1090110018
  PCI Device Name:  mt4099_pci_cr0
  Port1 MAC:        xxxxxxxxxxx
  Port2 MAC:        xxxxxxxxxx
  Versions:         Current        Available
     FW             2.40.5030      N/A
     PXE            3.4.0746       N/A

  Status:           No matching image found
Downgrade worked
Code: 
[root@esx:~] /opt/mellanox/bin/flint -d mt4099_pci_cr0 -i /tmp/orig_firmware.mlx -allow_psid_change burn

    Current FW version on flash:  2.42.5032
    New FW version:               2.40.5030

    Note: The new FW version is older than the current FW version on flash.

Do you want to continue ? (y/n) [n] : y


    You are about to replace current PSID on flash - "MT_1090120019" with a different PSID - "ISL1090110018".
    Note: It is highly recommended not to change the PSID.

Do you want to continue ? (y/n) [n] : y
Burning FS2 FW image without signatures - OK
Restoring signature                     - OK
Click to expand...
Please reset the configuration of the card and post the "mlxconfig q" again.
I'm looking for
Code: 
PHY_TYPE_P1 
PHY_TYPE_P2 
XFI_MODE_P1 
XFI_MODE_P2
 
Luzer

Luzer

Member
Mar 1, 2015
40
3
8
Midwest
i386 said:
Did you backed up the original firmware before flashing a new firmware?
There are multiple skus that start with "CX354A", one of them is the cx-3 pro:
View attachment 24966

FCB is the fdr version (56GBit/s infiniband)
QCB is the qdr version (40GBit/s infiniband)
Click to expand...
Cant find my photo of the card but searching on ebay I see it just a CX354A. The listing I purchased from isnt listed so I cant the original photo.
 

Attachments

Luzer

Luzer

Member
Mar 1, 2015
40
3
8
Midwest
prdtabim said:
Please reset the configuration of the card and post the "mlxconfig q" again.
I'm looking for
Code: 
PHY_TYPE_P1
PHY_TYPE_P2
XFI_MODE_P1
XFI_MODE_P2
Click to expand...

I don't see the settings
Code: 
[root@esx:~] /opt/mellanox/bin/mlxfwmanager
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      050-0050-02_01
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; QDR IB (40Gb/s) and 10GigE; PCIe3.0x8 8GT/s; RoHS R6
  PSID:             ISL1090110018
  PCI Device Name:  mt4099_pci_cr0
  Port1 MAC:        xxxxx
  Port2 MAC:        xxxxx
  Versions:         Current        Available
     FW             2.40.5030      N/A
     PXE            3.4.0746       N/A

  Status:           No matching image found

[root@esx:~] /opt/mellanox/bin/mlxconfig q

Device #1:
----------

Device type:    ConnectX3
Device:         mt4099_pciconf0

Configurations:                              Next Boot
         SRIOV_EN                            False(0)
         NUM_OF_VFS                          8
         LINK_TYPE_P1                        VPI(3)
         LINK_TYPE_P2                        VPI(3)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               True(1)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               True(1)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             PXE(1)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
 
Luzer

Luzer

Member
Mar 1, 2015
40
3
8
Midwest
i386 said:
That's a EMC branded MCX354A-QCBT. Folks here in the forums succesfully flashed them to vanilla mellanox firmware, see this post and newer: https://forums.servethehome.com/ind...rnet-dual-port-qsfp-adapter.20525/post-198004
Click to expand...
Was able to flash:
Code: 
[root@esx:~] /opt/mellanox/bin/flint -d mt4099_pci_cr0 -i /tmp/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn

    Current FW version on flash:  2.40.5030
    New FW version:               2.42.5000


    You are about to replace current PSID on flash - "ISL1090110018" with a different PSID - "MT_1090120019".
    Note: It is highly recommended not to change the PSID.

 Do you want to continue ? (y/n) [n] : y
Burning FS2 FW image without signatures - OK
Restoring signature                     - OK
What's weird is I can set the interface type
Code: 
[root@esx:~] /opt/mellanox/bin/mlxconfig -d  mt4099_pci_cr0 set LINK_TYPE_P1=2

Device #1:
----------

Device type:    ConnectX3
Device:         mt4099_pci_cr0

Configurations:                       y      Next Boot       New
-E- Aborted by user.1                        VPI(3)          ETH(2)
[root@esx:~] /opt/mellanox/bin/mlxconfig -d  mt4099_pci_cr0 set LINK_TYPE_P2=2

Device #1:
----------

Device type:    ConnectX3
Device:         mt4099_pci_cr0

Configurations:                              Next Boot       New
         LINK_TYPE_P2                        VPI(3)          ETH(2)

 Apply new Configuration? (y/n) [n] : y
Applying... Done!

No dice after reboot.
Code: 
[root@dell-r730:~] esxcli network nic list | grep -i mel
vmnic1000802  0000:86:00.0  nmlx4_en  Up            Up           40000  Full    e4:1d:2d:4d:5d:62  1500  Mellanox Technologies MT27500 Family [ConnectX-3]
vmnic8        0000:86:00.0  nmlx4_en  Up            Down             0  Half    e4:1d:2d:4d:5d:61  1500  Mellanox Technologies MT27500 Family [ConnectX-3]

Is there a way to wipe the flash completely then flash with the rom? I wonder if the old flash 'single nic' is stuck on it'.

Code: 
[root@dell-r730:~] /opt/mellanox/bin/mlxconfig q

Device #1:
----------

Device type:    ConnectX3
Device:         mt4099_pciconf0

Configurations:                              Next Boot
         SRIOV_EN                            False(0)
         NUM_OF_VFS                          8
         LINK_TYPE_P1                        VPI(3)
         LINK_TYPE_P2                        ETH(2)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               False(0)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             None(0)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               False(0)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             None(0)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)
 
P

prdtabim

Active Member
Jan 29, 2022
173
67
28
Just to compare. Mine is Connectx3 pro but I believe that the parameters are the same.
Code: 
mstconfig q

Device #1:
----------

Device type:    ConnectX3Pro    
Device:         /sys/bus/pci/devices/0000:33:00.0/config

Configurations:                              Next Boot
         SRIOV_EN                            True(1)         
         NUM_OF_VFS                          8               
         WOL_MAGIC_EN_P2                     True(1)         
         LINK_TYPE_P1                        ETH(2)          
         PHY_TYPE_P1                         XFI(2)          
         XFI_MODE_P1                         _40G(2)         
         FORCE_MODE_P1                       False(0)        
         LINK_TYPE_P2                        ETH(2)          
         PHY_TYPE_P2                         XFI(2)          
         XFI_MODE_P2                         _40G(2)         
         FORCE_MODE_P2                       False(0)        
         LOG_BAR_SIZE                        0               
         BOOT_PKEY_P1                        0               
         BOOT_PKEY_P2                        0               
         BOOT_OPTION_ROM_EN_P1               True(1)         
         BOOT_VLAN_EN_P1                     False(0)        
         BOOT_RETRY_CNT_P1                   0               
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)          
         BOOT_VLAN_P1                        1               
         BOOT_OPTION_ROM_EN_P2               True(1)         
         BOOT_VLAN_EN_P2                     False(0)        
         BOOT_RETRY_CNT_P2                   0               
         LEGACY_BOOT_PROTOCOL_P2             PXE(1)          
         BOOT_VLAN_P2                        1               
         IP_VER_P1                           IPv4(0)         
         IP_VER_P2                           IPv4(0)         
         CQ_TIMESTAMP                        False(0)        
         STEER_FORCE_VLAN                    False(0)
 
M

mach3.2

Active Member
Feb 7, 2022
132
87
28
Here's a config from my CX534A flashed with Lenovo's FCBT firmware. Hope it's helpful for you.

Firmware link: Mellanox ConnectX-3 Firmware (For Windows) - Lenovo Support US

Bash: 
[root@ESXi2:~] /opt/mellanox/bin/mlxconfig q

Device #1:
----------

Device type:    ConnectX3
Device:         mt4099_pciconf0

Configurations:                              Next Boot
         SRIOV_EN                            True(1)
         NUM_OF_VFS                          16
         LINK_TYPE_P1                        ETH(2)
         PHY_TYPE_P1                         XFI(2)
         XFI_MODE_P1                         _10G(0)
         FORCE_MODE_P1                       False(0)
         LINK_TYPE_P2                        ETH(2)
         PHY_TYPE_P2                         XFI(2)
         XFI_MODE_P2                         _10G(0)
         FORCE_MODE_P2                       False(0)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               False(0)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             None(0)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               False(0)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             None(0)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)
         STEER_FORCE_VLAN                    False(0)
Luzer said:
Is there a way to wipe the flash completely then flash with the rom? I wonder if the old flash 'single nic' is stuck on it'.
Click to expand...
Just spitballing, but there's a recovery header on the card that you can short which will put the card into recovery mode for flashing.

Details on how to do it can be found in the thread linked in post #13.
 
You must log in or register to reply here.
Top Bottom