Flashing the Cisco CX623106A NIC to OEM firmware

[bitbckt]

I'm breaking this conversation out of the Great Deals thread here.

As I posted on that thread, I've received my order from the linked eBay seller, and confirmed that what I got is what they sold. The boards seem to work fine as they arrive, but we'd all generally prefer to be on OEM firmware to continue getting updates without involving Cisco.

Problem the first: these NICs come with secure_fw which prevents PSID changes over PCI-e.

I've broken out the I2C pins and hooked them up to my trusty MTUSB, however the NIC doesn't seem to want to talk. I've confirmed pinout, and tried several variations of MLNX_OFED versions (LTS and master), in combination with Debian 12 "bookworm" and Debian 11 "bullseye". No dice. I hooked up my Saleae and it just seems... silent?

Thinking I may have some issue with my MTUSB, I broke out the soldering iron once more and pulled a ConnectX-5 Ex out of a server and broke out its I2C pins. That worked swimmingly.

Problem the second: how do I get these things to answer the phone on I2C? Maybe livefish is required?

 

[NablaSquaredG]

Which command have you tried for flashing?

Have you verified the pinout? Maybe ConnectX-6 uses a different I²C pinout.


I'm awaiting my MTUSB...

 

[bitbckt]

Which command have you tried for flashing?

That's the thing: it won't even respond to a query over I2C. I've tried all sorts of variations of burn, but that's pointless if the MTUSB can't open the device at all.

Have you verified the pinout? Maybe ConnectX-6 uses a different I²C pinout.

It's the same pinout as the ConnectX-5. White=GND on the DB9/Molex 2510 lead in the adapter kit. The silkscreen is correct according to my oscilloscope.

 

[bitbckt]

I'll also add that I've done an strace on flint -d /dev/mst/mtusb-1 q and it seems to be retrying the connection correctly 3 times. On the ConnectX-5, the first attempt succeeds every time.

 

[NablaSquaredG]

Have you tried in livefish mode, i.e. with the FNP Jumper set?

It's probably not possible to dump the firmware from the card due to secure-fw?

 

[bitbckt]

Have you tried in livefish mode, i.e. with the FNP Jumper set?

It's probably not possible to dump the firmware from the card due to secure-fw?

I haven’t tried livefish yet - I had to stop fiddling with this for the day. Life stuff.

 

[NablaSquaredG]

Cisco FW ini doesn't contain anything obvious that would disable I²C access...

 

[oneplane]

Does it show up using i2cdetect (if raw I2C is available)

 

[bitbckt]

Well, this turned out to be easier than I anticipated.

tl;dr livefish mode works fine. No MTUSB is required for this card.

Throughout my testing, I've had mlx5_core and i2c-diolan-u2c blacklisted in modprobe, and installed as /bin/false to prevent autoloading. How that is accomplished will vary by distro.

I'm using mft 4.24.0 on Debian 12 "bookworm".

I soldered a two-pin header to J7 on the board, bridged it with a jumper, and booted. The link lights on the back of the card will remain solid orange. This is normal.

I issued the following:

# mst start
# mst status // /dev/mst/mt4125_pciconf0 is the CX-6.
# flint -d /dev/mst/mt530_pciconf0 --psid MT_0000000437 -i fw-ConnectX6Dx-rel-22_35_2302-MCX623106AS-CDA_Ax-UEFI-14.28.16-FlexBoot-3.6.805.signed.bin -ocr --nofs --allow_psid_change --yes burn
# shutdown -r now

Remove the jumper and start the machine. When the machine came back up, I got:

# mst start
# flint -d /dev/mst/mt4125_pciconf0 q

Image type:            FS4
FW Version:            22.35.2302
FW Release Date:       27.12.2022
Product Version:       22.35.2302
Rom Info:              type=UEFI version=14.28.16 cpu=AMD64,AARCH64
                               type=PXE version=3.6.805 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             e8ebd30300d09292        4
Base MAC:              e8ebd3d09292            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  MT_0000000437
Security Attributes:   secure-fw

Note that the PSID has changed, but secure-fw is still set.

However, I can still flash using PSID-matching firmware now that it's been set to the NVIDIA part number.

# flint -d /dev/mst/mt4125_pciconf0 -i fw-ConnectX6Dx-rel-22_37_1014-MCX623106AS-CDA_Ax-UEFI-14.30.13-FlexBoot-3.7.102.signed.bin burn

    Current FW version on flash:  22.35.2302
    New FW version:               22.37.1014

FSMST_INITIALIZE -   OK
Writing Boot image component -   OK
Restoring signature                     - OK
-I- To load new FW run mlxfwreset or reboot machine.

 

[bitbckt]

Following on with that last firmware update:

# mlxfwreset -d /dev/mst/mt4125_pciconf0 reset

Minimal reset level for device, /dev/mst/mt4125_pciconf0:

3: Driver restart and PCI reset
Continue with reset?[y/N] y
-I- Sending Reset Command To Fw             -Done
-I- Resetting PCI                           -Done
-I- Restarting MST                          -Done
# flint -d /dev/mst/mt4125_pciconf0 q
Image type:            FS4
FW Version:            22.37.1014
FW Release Date:       4.5.2023
Product Version:       22.37.1014
Rom Info:              type=UEFI version=14.30.13 cpu=AMD64,AARCH64
                               type=PXE version=3.7.102 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             e8ebd30300d09292        4
Base MAC:              e8ebd3d09292            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  MT_0000000437
Security Attributes:   secure-fw

 

[NablaSquaredG]

Note that the PSID has changed, but secure-fw is still set.

Probably because you have flashed with a signed fw.

Can you try flashing with MCX623106AN-CDAT? This is apparently unsigned fw.

 

[bitbckt]

Probably because you have flashed with a signed fw.

Can you try flashing with MCX623106AN-CDAT? This is apparently unsigned fw.

I'll try that now.

 

[bitbckt]

Probably because you have flashed with a signed fw.

Can you try flashing with MCX623106AN-CDAT? This is apparently unsigned fw.

I'm able to flash that firmware, and change to that PSID, but the card doesn't seem to want to come out of livefish mode after that. It remains on /dev/mst/mt530_pciconf0 and doesn't respond to query.

 

[NablaSquaredG]

I'm able to flash that firmware, and change to that PSID, but the card doesn't seem to want to come out of livefish mode after that. It remains on /dev/mst/mt530_pciconf0 and doesn't respond to query.

Interesting. What happens if you flash without --nofs?

 

[bitbckt]

Interesting. What happens if you flash without --nofs?

No change in behavior. If I flash back to the AS firmware, it recovers just fine.

 

[NablaSquaredG]

Interesting... Well, I guess I don't need my MTUSB-1 then. Better to have it and not need it than to need it and not have it.

 

[bitbckt]

Probably the best route forward for most people will be to leave it on secure_fw, and get the PSID changed with livefish mode. Future updates from NVIDIA should apply without any problems.

That probably doesn't necessarily involve soldering, but it was very convenient to just install/remove a jumper during my testing.

I'll also note that after each FW update I've tried, I've also tried to get it talking over I2C. It remains completely silent, as though it's expecting a handshake that the MTUSB isn't providing. That's been a bit frustrating - I don't think I've encountered a MLNX device that I did not work over I2C before.

 

[Thomas42]

Is it possible to enable crypto support with flashing firmware?

 

[bitbckt]

That would be the -AC firmware. I'm not sure off-hand. Cisco has that part number, so it seems possible on the face of it.

 

[ano]

now Im sad I only bought 6.. the good thing is my lab.. has 6!