Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

Hello!

I have the opportunity to purchase some used dual-port Mellanox MCX354A-QCBT ConnectX-3 cards with VPI network adapter. (10 GbE Ethernet and 40 Gbps InfiniBand.) These would be perfect for my cluster after correct configuration..

They were originally pulled from EMC systems and currently have EMC firmware installed. I want to flash the original stock (Mellanox) firmware and return them to functionally equivalant stock Mellanox ConnectX-3 cards.

I found a few threads about this ( https://forums.servethehome.com/index.php?threads/anyone-flashing-mellanox-connectx-3-in-qty.16298/ and https://forums.servethehome.com/ind...lanox-connectx-3-en-with-stock-firmware.8689/ ). These discussions indicate that it's a risky process and that the original firmware files are not easy to find. If it goes horribly wrong for a particular card is it difficult to restore using a hardware flasher?

Has anybody tried this recently, and what have your experiences been? Are there any particular hints or tips that I should follow? I intend to connect them to a Voltaire 4036 or 4036E switch, depending on what's available and how cost-effective the options are. Thanks!!

Kind regards,

ullbeking

 

i crossflash qcbt to fcbt two month ago ,so far so good

 

Thank you for the positive report, @Freebsd1976!

Please could you provide a brief outline of what you did, or what instructions you followed, to do this? What host machine and OS did you use? Where did you get the firmware from, and did you do it by software flashing or hardware flashing?

 

basically i follow the first link you post
and i use windows flash and software flash
only problem is you need find the firmware link , i found in the this forum but now i cannot remember which post it is

 

found the post , in the post there are two links about firmware, both 2.40.5000.tgz and 2.40.5030.tgz,and i use the first one.

https://forums.servethehome.com/index.php?threads/connectx-3-hp-to-mellanox-firmware-help.12452/page-2

 

@Freebsd1976, you rock my world. You really do. Thank you.

 

@Freebsd1976, what firmware was on your MCX354A-QCBT before you flashed it to -FCBT? My point is my -QCBT card has EMC firmware and I want to replace it by Mellanox firmware. I think this issue is different from changing -QCBT to -FCBT. Please could you let me know what the firmware was before and after? (OEM or Mellanox.) Thanks!!

 

Mellanox firmware, I think Flash Oem to mellanox should be same

 

I just flashed a card last week as well. If you dig on their website they have all the versions of firmware available for download.

Get their firmware tools zip first. Then interrogate your card and see what you need. Go from there.

The issue I'm facing is the card won't recognize an Arista or Cisco DAC. Might need to break down and buy a Mellanox AOC. I had a similar issue with Cisco cards, they only recognize Cisco cables. My Mellanox branded cards take any DAC I can find.

 

Oh, you can save off your original firmware and flash back if you make a mistake.

 

Yes, I tried digging around but I wasn't sure of the correct one to use. Obviously I want to flash to the latest "good" firmware that makes my card into a standard Mellanox card rather than an EMC OEM card. One potentially major issue on my mind is this: Might there be anything about the EMC OEM hardware or firmware that prevents cross-flashing?

The Mellanox firmware tools? Will this work with the EMC firmware? (I haven't actually purchased the card yet but I'm currently submitting an order for express delivery because this is kind of important to me.)

Of course. Are there standard flashing tools that I can use to do this? For example, would `flashrom` work?

Please could you give me some examples of useful tools for reading original firmware and flashing new firmware? My understanding from my research so far is that the EMC (or Mellanox?) tools do not allow cross-flashing and that using more flexible or generic tools will be necessary.

Do you know if it's possible to flash the firmware using a hardware flasher? This would obviously require extracting the actual firmware image from the .exe file on the Mellanox website.

 

You're overthinking this. Mellanox makes it REALLY easy.

They have firmware tools like flint. Look at the first link. Those are all commands from Mellanox's tools. The firmware tools are a download on their site.

You run one of their tools to find the current firmware and PSID. From that you then use the chooser on their website to find the latest that's compatible with your card. You run flint with the new bin file and about 30s later you have new firmware. You can run all of this if the card is physically present but the OS doesn't detect it. They're built to handle undetected cards.

You don't need hardware flashing tools or anything. Their tools allow you to pull off all of the current firmware and save it. Their own tools have everything you'd need. They have really nice documentation that walks you through it as well.

They even have a section of their website with vendor specific details. I re-flashed a Fujitsu card to be a standard Mellanox one.

From what I understand the cards are 100% generic with vendor add-ons in flash only. If this is an EMC version of a Connectx-3 then you're fine.

And just to clarify, you have an EMC version of a Mellanox Connectx-3 that you want to flash to a Mellanox generic Connectx-3 right? Your last post made me think you're trying to toss custom firmware on there. Flint will allow that as well, but that's way out of my league.

 

On the second link (https://forums.servethehome.com/ind...lanox-connectx-3-en-with-stock-firmware.8689/)

Post #11 are the exact steps to follow.

 

Thank you @oddball !! This has been a huge help. I spent the evening experimenting with some of the flashing tools, compiling them, etc, and am getting a better feel now for how this is supposed to work. I’ll let you know how I get on once I receive my experimental HCAs.