Changing PSID is unsupported under Secure FW.
- Thread starter jpmomo
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
as a follow up to my own question. the hpe x6-dx card that I have was converted to a generic mellanox card. I forget when/how I was able to do that. The main issue still remains, how to get a card with secure fw enabled to allow_psid_change.
I searched google and this forum and have made some progress but still not there.
My first issue was that I was using the mellanox tools with vmware. I switched over to linux and got a bit further.
The first step is to jump the pins on the card to put into flash recovery mode. If the card does not have the jumper, use a paper clip and then boot with the paper clip jumping the two "pins" (actually just holes).
At this stage you are supposed to be able to burn the new image with the allow_psid_change option. In my setup, I was not able to complete this step properly.
I am getting the following error:
Let me know if anyone has any suggestions.
thanks!
I searched google and this forum and have made some progress but still not there.
My first issue was that I was using the mellanox tools with vmware. I switched over to linux and got a bit further.
The first step is to jump the pins on the card to put into flash recovery mode. If the card does not have the jumper, use a paper clip and then boot with the paper clip jumping the two "pins" (actually just holes).
At this stage you are supposed to be able to burn the new image with the allow_psid_change option. In my setup, I was not able to complete this step properly.
I am getting the following error:
Let me know if anyone has any suggestions.
thanks!
Yeah, it’s I2C. I could do it only with MTUSB-1. If you’re a pro in flint code or you have some other I2C adapter, or deep in reverse engineering of the binary image, you could try these alternatives. If you short recovery pins, the resulting recovery device is not recognized by anything, so that is a dead end for now.
When you mention " If you short recovery pins, the resulting recovery device is not recognized by anything, so that is a dead end for now." are you referring to J7 above (the 2 holes for flash recovery)? I ask because that was supposed to be the first step in trying to flash a nic with secure_fw enabled. That is if you wanted to flash with allow_psid_change. I was able to short those 2 holes and boot up with those 2 holes shorted. the nic showed flash recovery but when I tried to use the flint command, it did not work. I was able to then remove the paper clip jumper and the nic booted correctly with its previous fw in tact.
in one of your previous posts, you mentioned that you were able to do somthing like I am trying with a single port 100G converted to a single port 200G. You mentioned that it got hot without any fan (not that surprised as I make sure to mount an active fan directly on the heatsink or ensure high speed fans in a real server.)
Exactly how did you do that with your hpe single port 100G? Does that nic have a connector on its pcb or just 3 holes like my ss? I am assuming from your above comment, that you did not short J7 as even the mtusb-1 with I2C would not recognize the card. If you just booted the card normally, what commands/tools did you use to get it to burn the 200G image?
thanks for any help to get one step closer!
Exactly how did you do that with your hpe single port 100G? Does that nic have a connector on its pcb or just 3 holes like my ss? I am assuming from your above comment, that you did not short J7 as even the mtusb-1 with I2C would not recognize the card. If you just booted the card normally, what commands/tools did you use to get it to burn the 200G image?
thanks for any help to get one step closer!
thanks again for your help. Please let me know if my summary below matches what you did to burn a mellanox nic with secure_fw to a different psid:
1. You needed to use the MTUSB-1 device with a connection directly to the nic to issue commands via the I2C protocol/interface. Can you describe exactly how you made this connection? ex. did your nic have just the 3 holes? if so, how did you physically connect the MTUSB-1 device? Do you have any pics of the MTUSB-1 device with the cable and how it connects to the nic?
2. Once connected, you were able to use normal flint commands to burn the nic with an image with a different psid. Do you recall the parameters? ex. did you pass the allow_psid_change parameter? or was it something else that allowed the new image file to be burned?
3. rebooted and .....magic! a 100G nic is turned into a functioning 200G nic. I have some sw that can test to make sure that it is actually performing at 200Gbps rates and 400Gbps bidirectionally.
I have been able to do the above directly using just flint from either linux or vmware without the need for MTUSB-1 or I2C. The difference is that the nics that I have been able to do this with were the cx-5 without any secure_fw. the cx6 and cx6-dx seem to be enabled with secure_fw.
Thanks for your help! if I am able to get this working, I may be able to return the favor and send a matching 200G nic so that you can have the matching pair!
1. You needed to use the MTUSB-1 device with a connection directly to the nic to issue commands via the I2C protocol/interface. Can you describe exactly how you made this connection? ex. did your nic have just the 3 holes? if so, how did you physically connect the MTUSB-1 device? Do you have any pics of the MTUSB-1 device with the cable and how it connects to the nic?
2. Once connected, you were able to use normal flint commands to burn the nic with an image with a different psid. Do you recall the parameters? ex. did you pass the allow_psid_change parameter? or was it something else that allowed the new image file to be burned?
3. rebooted and .....magic! a 100G nic is turned into a functioning 200G nic. I have some sw that can test to make sure that it is actually performing at 200Gbps rates and 400Gbps bidirectionally.
I have been able to do the above directly using just flint from either linux or vmware without the need for MTUSB-1 or I2C. The difference is that the nics that I have been able to do this with were the cx-5 without any secure_fw. the cx6 and cx6-dx seem to be enabled with secure_fw.
Thanks for your help! if I am able to get this working, I may be able to return the favor and send a matching 200G nic so that you can have the matching pair!
have you tried the "bb" argument? I have notes from a while back when I was crossflashing some older cx3 stuff, "bb" let me burn raw images and bypass checks that were blocking flashes even with allow-psid-change. no clue if this applies to the newer cards or if these args even still exist in flint. from my notes:
also worth looking into:
I would do these in "normal" mode with the card booted fully, not in the jumpered backup bootloader
edit: yeah nevermind there's no way mellanox would allow these under signed firmware, and as you noticed even the recovery bootloader is locked, you'll need to burn it over i2c like @necr - sadly I don't see any support inside flint for any generic i2c interfaces, only the mellanox mtusb device
Code:
bb - Burn Block - Burns the given image as is.No checks are done.
example crossflash cx3 to cx3pro (bricks card):
flint -d /dev/mst/mt4099_pci_cr0 -i 3xpro.bin bb
Code:
-nofs - when specified, the burn process will not be failsafe. A non-failsafe burn is
required (on the rare occasion) when a new firmware version has modifications in
the Invariant Sector
#example burn new image to empty card
#wipes out guid and mac if i remember right
#backup those first with:
flint -d /dev/mst/mt4099_pci_cr0 query full
#then
flint -d /dev/mst/mt4099_pci_cr0 -i fw-4099-2_31_5050-MCX354A-FCB_A2.bin -nofs -guid 12345678 -mac 0002c9002100 burnedit: yeah nevermind there's no way mellanox would allow these under signed firmware, and as you noticed even the recovery bootloader is locked, you'll need to burn it over i2c like @necr - sadly I don't see any support inside flint for any generic i2c interfaces, only the mellanox mtusb device
Last edited:
Finally success! First I want to thank necr for pushing me to keep at it.
The issue appeared to be 2 things:
1. The mapping of the wires to the nic. I made the wrong assumption that green would go to the ground. It should be G=SDA, R=SCL and W=GND
2. The other issue that made troubleshooting more difficult was that this method did not seem to work with the cx6-DX cards. At least not the one that I had on hand. It did work with both of the cx6 cards and most of the cx5 cards (although I don't think you need to go the I2C (mtusb-1) route with the cx5 cards unless they are using secure_fw.
You can pm me for any details.
The issue appeared to be 2 things:
1. The mapping of the wires to the nic. I made the wrong assumption that green would go to the ground. It should be G=SDA, R=SCL and W=GND
2. The other issue that made troubleshooting more difficult was that this method did not seem to work with the cx6-DX cards. At least not the one that I had on hand. It did work with both of the cx6 cards and most of the cx5 cards (although I don't think you need to go the I2C (mtusb-1) route with the cx5 cards unless they are using secure_fw.
You can pm me for any details.
Just a hint from the mstflint source code
github.com
Simply short the jumper puts the device into non-secure flash recovery.
Then the question becomes, how to enter secure flash recovery.
Also, this looks promising, but I am too lazy to deal with burning all the macs and guids:
mstflint/kernel/mst_main.c at 346d949ed26d039030282c3b59951cd231ad0adf · Mellanox/mstflint
Mstflint - an open source version of MFT (Mellanox Firmware Tools) - Mellanox/mstflint
github.com
{ PCI_DEVICE(MST_MELLANOX_PCI_VENDOR, 0x0213) }, /* MT2892 Family [ConnectX-6 Dx Secure Flash Recovery] */Simply short the jumper puts the device into non-secure flash recovery.
Then the question becomes, how to enter secure flash recovery.
Also, this looks promising, but I am too lazy to deal with burning all the macs and guids:
Last edited:
secure flash recovery still requires a signed image matching the existing psidJust a hint from the mstflint source code
mstflint/kernel/mst_main.c at 346d949ed26d039030282c3b59951cd231ad0adf · Mellanox/mstflint
Mstflint - an open source version of MFT (Mellanox Firmware Tools) - Mellanox/mstflint
{ PCI_DEVICE(MST_MELLANOX_PCI_VENDOR, 0x0213) }, /* MT2892 Family [ConnectX-6 Dx Secure Flash Recovery] */
Simply short the jumper puts the device into non-secure flash recovery.
Then the question becomes, how to enter secure flash recovery.
Also, this looks promising, but I am too lazy to deal with burning all the macs and guids:
shorting the 2 pins for flash recovery mode aka live fish mode, does not allow you to then flash with new fw on the -dx nics. You can however just boot up in normal mode (non live fish) and use the mtusb device to override the psid change and secure-fw issue.
Thanks for the hint from @jpmomo
I am able to change the psid of a secure-fw cx6-dx
We just need to boot the card normally and use mtusb-1 and add --override_cache_replacement flag. It did not work when flashing through pcie the same way.
It took more than 1 hour to flash the firmware and my kernel 5.4.0-122-generic #138-Ubuntu always crashed during the flashing process even though I blacklist the mlx5_core.
Luckily, even though the flash process using i2c was not complete, the psid has changed. So I was able to flash through pcie rather quickly after a reboot.
I highly suspect that this is just a software lock in the flint software (and it is open source) but I don't have the time and incentive to figure it out.
In case this is going to be disabled in the future, here is the version that it worked: flint, mft 4.20.0-34, built on Apr 25 2022, 20:56:44. Git SHA Hash: 62bbc33 MLNX_OFED_LINUX-5.6-2.0.9.0-ubuntu20.04-x86_64
I am able to change the psid of a secure-fw cx6-dx
We just need to boot the card normally and use mtusb-1 and add --override_cache_replacement flag. It did not work when flashing through pcie the same way.
It took more than 1 hour to flash the firmware and my kernel 5.4.0-122-generic #138-Ubuntu always crashed during the flashing process even though I blacklist the mlx5_core.
Luckily, even though the flash process using i2c was not complete, the psid has changed. So I was able to flash through pcie rather quickly after a reboot.
I highly suspect that this is just a software lock in the flint software (and it is open source) but I don't have the time and incentive to figure it out.
In case this is going to be disabled in the future, here is the version that it worked: flint, mft 4.20.0-34, built on Apr 25 2022, 20:56:44. Git SHA Hash: 62bbc33 MLNX_OFED_LINUX-5.6-2.0.9.0-ubuntu20.04-x86_64
Last edited:
Hey, can you tell me how it was achieved? I am very interested in this, and I want to use to convert my CX6 NIC. What commands are needed to use MTUSB-1 to refresh the firmware and how to back up the original firmware?