Cross-flash HPE ConnectX-6 VPI

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Jul 19, 2020
51
19
8
I recently picked up a pair of HPE branded ConnectX-6 VPI NICs. Unfortunately, the cards are locked to HPE firmware, so I cannot update it with the latest firmware from Mellanox. So, I would like to cross-flash the cards to the standard Mellanox PSID. But it seems that there are some security features that are preventing this - I cannot access the flash with JP2 open, and the tools will not talk to the card with JP2 installed.

JP2 removed (open):

Code:
$ sudo flint -d 01:00.0 query full
Image type:            FS4
FW Version:            20.27.6202
FW Release Date:       23.6.2020
Part Number:           P06250-B21_Ax
Description:           HPE InfiniBand HDR100/Ethernet 100Gb 1-port 940QSFP56 x16 Adapter
Product Version:       20.27.6202
Rom Info:              type=UEFI version=14.20.25 cpu=AMD64
                       type=PXE version=3.5.903 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             98039b0300******        4
Base MAC:              98039b******            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  HPE0000000035
Security Attributes:   secure-fw
Default Update Method: fw_ctrl
Life cycle:            GA NON SECURED
$ sudo flint -d 01:00.0 hw query
-E- Failed to open Device: File exists. MFE_NO_FLASH_DETECTED
$ sudo flint -d 01:00.0 ri orig_fw.bin
-E- Failed to read image. Read image is not supported
$ sudo flint -d 01:00.0 dc orig_fw.ini
-E- Failed dumping Fw Configuration : Unsupported operation under Secure FW
JP2 installed (shorted):

Code:
$ lspci | grep Mell
01:00.0 Memory controller: Mellanox Technologies MT28908A0 Family [ConnectX-6 Flash Recovery]
81:00.0 Memory controller: Mellanox Technologies MT28908A0 Family [ConnectX-6 Flash Recovery]
c1:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
c1:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
$ sudo flint -d 01:00.0 query full
-E- Cannot open Device: 01:00.0. No such file or directory. MFE_ICMD_NOT_SUPPORTED
$ sudo flint -d 81:00.0 query full
-E- Cannot open Device: 81:00.0. No such file or directory. MFE_ICMD_NOT_SUPPORTED
Just yesterday I was able to use the instructions from https://forums.servethehome.com/ind...ectx-3-en-with-stock-firmware.8689/post-80356 to cross-flashed a pair of ConnectX-5 NICs from an Oracle PSID to a Mellanox PSID - I had to short JP2 to place the cards in recovery mode to make this work, but there were no issues, aside from I was unable to find the .mlx firmware on the Mellanox site, but flashing with the .bin version worked just fine. So the OFED install is fine (it's running the latest, 5.4) and the drivers and what not are all installed correctly. Does anyone know what steps need to be taken to "un-secure" the card, or is this impossible on CX-6 (perhaps implemented with fuses or similar)?
 

tinfoil3d

QSFP28
May 11, 2020
873
400
63
Japan
Pardon for intrusion and stupid question, why would you need to update/change fw if it works? I may be looking into buying one of those in future, anything specific you're updating it for?
 

necr

Active Member
Dec 27, 2017
151
47
28
124
QSFP56 @PCI 4.0 x16 is capable of 200G, would be interesting to see if it's binned chips that can't reach 200G, or a continuation of our favorite QCBT-FCBT reflash.
 
  • Like
Reactions: teslaq

necr

Active Member
Dec 27, 2017
151
47
28
124
Cross-flashed from HPE0000000035 to HPE0000000034 and then to MT_0000000223, card is alive @1G link. As expected, the differences between HDR100 and HDR are minimal. Card is really hot, reaching 100C in under 2 min without cooling in a big tower, had to mount 50x50x10mm fan on top. FS4 image is complicated, with a new ITOC section and signatures. It may be possible to mess with HP secure-fw stuff, but I just used I2C 0x48.

As I only have 1 working card, can't really test if can get to 200G link, need another CX6.

Diff between HDR100 and HDR below. Higher power level, as expected, to keep the PAM4 200G link.

Bash:
-;; Generated automatically by iniprep tool on Wed Jun 30 23:25:28 IDT 2021 from ./cx6_MCX654105A_hpe_vpi_200g_1p_hdr.prs;; FW version: 20.31.1014
+;; Generated automatically by iniprep tool on Wed Jun 30 23:25:28 IDT 2021 from ./cx6_MCX653105A_hpe_vpi_100g_1p_hdr.prs;; FW version: 20.31.1014
 
 ;; Mellanox Technologies LTD
 
 
 ;;[PS_INFO]
-;;Name = P06154-B21_Ax
-;;Description = HPE InfiniBand HDR/Ethernet 200Gb 1-port 940QSFP56 x16 Adapter
+;;Name = P06250-B21_Ax
+;;Description = HPE InfiniBand HDR100/Ethernet 100Gb 1-port 940QSFP56 x16 Adapter
 
 [image_info]
 ;;;;; This section contains info which is shared by FW and burning tool
 
-psid = HPE0000000034
-name = P06154-B21_Ax
-description = HPE InfiniBand HDR/Ethernet 200Gb 1-port 940QSFP56 x16 Adapter
-prs_name = cx6_MCX654105A_hpe_vpi_200g_1p_hdr.prs
+psid = HPE0000000035
+name = P06250-B21_Ax
+description = HPE InfiniBand HDR100/Ethernet 100Gb 1-port 940QSFP56 x16 Adapter
+prs_name = cx6_MCX653105A_hpe_vpi_100g_1p_hdr.prs
 
 
 
@@ -113,7 +113,7 @@
 pcie_cfg_data.pci_cfg_space.cfg_hdr.sub_vendor_id = 0x1590
 pcie_cfg_data.pci_cfg_space.cfg_hdr.device_id = 4123
 nv_config.global.pci.settings.total_vfs = 126
-pcie_cfg_data.pci_cfg_space.cfg_hdr.subsystem_id = 0x02AB
+pcie_cfg_data.pci_cfg_space.cfg_hdr.subsystem_id = 0x02AE
 nv_config.global.pci.settings.fpp_en = 1
 nv_config.port[0].vpi.network_link_type_ib = 0x1
 nv_config.port[0].vpi.network_link_type_eth = 0x1
@@ -137,15 +137,14 @@
 phy_fw_main_config.phy_shomron.lcl[0].eth_speeds_supported.Speed25Gb = 0x1
 phy_fw_main_config.phy_shomron.lcl[0].eth_speeds_supported.Speed40Gb = 0x1
 phy_fw_main_config.phy_shomron.lcl[0].eth_speeds_supported.Speed50Gb = 0x1
+phy_fw_main_config.phy_shomron.lcl[0].eth_speeds_supported.Speed50Gb_Pam4 = 0x1
 phy_fw_main_config.phy_shomron.lcl[0].eth_speeds_supported.Speed100Gb = 0x1
-phy_fw_main_config.phy_shomron.lcl[0].eth_speeds_supported.Speed50Gb_Pam4 = 0x1
 phy_fw_main_config.phy_shomron.lcl[0].eth_speeds_supported.Speed100Gb_Pam4 = 0x1
-phy_fw_main_config.phy_shomron.lcl[0].eth_speeds_supported.Speed200Gb = 0x1
-host_management.ncsi_ini.num_of_bmc = 1
 host_management.ncsi_ini.num_channels_per_pkg = 1
 host_management.mctp_ini.smbus_enable = 0x0004
 host_management.mctp_ini.nic_slave_addr = 0x49
 host_management.mctp_ini.asn_ocbb_enable = 0x1
+host_management.ncsi_ini.num_of_bmc = 1
 host_management.mctp_ini.nic_eid = 0x11
 host_management.ahs_support=1
 secure_fw.hpcs=1
@@ -154,7 +153,11 @@
 nv_config.fw_config.port[0].lldp_nb_param.lldp_nb_rx_mode=2
 nv_config.fw_config.port[0].lldp_nb_param.lldp_nb_tx_mode=2
 nv_config.fw_config.port[0].lldp_nb_param.lldp_msg_tx_interval=0x64
+nv_config.fw_config.port[1].lldp_nb_param.lldp_nb_rx_mode=2
+nv_config.fw_config.port[1].lldp_nb_param.lldp_nb_tx_mode=2
+nv_config.fw_config.port[1].lldp_nb_param.lldp_msg_tx_interval=0x64
 nv_config.fw_config.port[0].keep_link_up.keep_link_up_on_boot = 0x1
+hdr100_device = 1
 host_management.pldm_ini.pldm_supported_features.redfish=1
 host_management.pldm_ini.pldm_sensor_rec_handle_en.rh_adapter_rde_pdr=1
 host_management.pldm_ini.pldm_sensor_rec_handle_en.rh_port_collection_rde_pdr=1
@@ -345,7 +348,7 @@
 gpios.functions.gpio_func[51].mapping= 49
 gpios.functions.gpio_func[51].reverse_polarity= 0
 gpios.configurations.gpios[49].gpio_config_en= 1
-gpios.configurations.gpios[49].mode= 1
+gpios.configurations.gpios[49].mode= 2
 gpios.configurations.gpios[49].def_val= 0
 gpios.configurations.gpios[49].pull_up= 1
 gpios.configurations.gpios[49].pull_en= 1
@@ -385,8 +388,9 @@
 ;;;;; plastic hw and board config data
 boards_management.board_power_consumption_1p = 22270
 boards_management.board_power_consumption_2p = 22270
-boards_management.spare_part = P08354-001
+boards_management.spare_part = P08356-001
 boards_management.product_id = 940QSFP56
+
 
 ;;;;; modules
 
@@ -426,7 +430,7 @@
 module.num[0].m2l.rx.lane[3].local_port= 0
 module.num[0].m2l.tx.lane[3].local_port= 0
 module.num[0].smbus_slv_indx= 2
-module.num[0].module_power_level_supported= 7
+module.num[0].module_power_level_supported= 4
 
 ;;;;; I2C Devices
 
  • Like
Reactions: nasbdh9 and taoj17v

jpmomo

Active Member
Aug 12, 2018
531
192
43
Do I need to "open the bus" with something similar to below?:

Switch Reprogramming through I2C Port
In order to reprogram the switch through the I2C adapter, follow the steps below:

For MSX1710/MSX67XX Switch systems:

  1. Open the bus:
    i2c -a 1 -d 1 /dev/mst/mtusb-1 w 0x60 0x20 0x10
    i2c -a 1 -d 1 /dev/mst/mtusb-1 w 0x62 0x00 0x01
  2. Burn the firmware:
    flint -d /dev/mst/mtusb-1 -i ./fw-SX.bin b
Or is there an additional step?


For MSX6025/6036 Switch systems:

  1. Open the bus:
    i2c -d /dev/mst/mtusb-1 w 0x22 0x1a 0xfb
  2. Route the I2C bus to the switch device:
    i2c -d /dev/mst/mtusb-1 w 0x70 0x0 0x1
  3. Burn the firmware:
    flint -d /dev/mst/mtusb-1 -i ./fw-SX.bin b
I haven't found any info on using the mtusb-1 for flashing a nic.
 

necr

Active Member
Dec 27, 2017
151
47
28
124
Nope, opening bus is not necessary for the NICs. It should be somewhat close to
flint -d /dev/mst/mtusb-1 -i ./fw-SX.bin b
(let me see the history, in case I find it I'll update the command).

Upd.
flint -d mtusb-1 -i .\fw-ConnectX6-rel-20_31_1014.bin --allow_psid_change --override_cache_replacement --no_flash_verify b
 
Last edited: