Flash OEM (IBM) Mellanox ConnectX-3 EN with stock firmware?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

josiah

New Member
Feb 4, 2019
7
0
1
I wish it were the binary firmware file, but according to the IBM instructions where I downloaded it, it is to be executed in some proprietary software to perform the update. mstflint does not recognize it. Peaking a little at the .bin file, it is an executable, not a firmware image.
I'm not sure if you saw this or if it appliesto you but the same firmware (v 2.40.5048) appears to be available through some sort of windows installer here: Mellanox ConnectX-3 Firmware Update v2.40.5048 for Windows - US


Also, since I'm only being initiated into this I was wondering how much card revisions matter? I believe Stereodude had a 00D9691 part that was Rev A5. I also have some IBM cards but they are 00W0054 parts with Rev A1.

I can crossflash them using the generic firmware available for the CX312A-XCBT but it specifically says A2-A6 so I wasn't sure if this means I'll run into problems later?

Cheers
 
Last edited:

747builder

Active Member
Dec 17, 2017
112
58
28
I'm not sure if you saw this or if it appliesto you but the same firmware (v 2.40.5048) appears to be available through some sort of windows installer here: Mellanox ConnectX-3 Firmware Update v2.40.5048 for Windows - US


Also, since I'm only being initiated into this I was wondering how much card revisions matter? I believe Stereodude had a 00D9691 part that was Rev A5. I also have some IBM cards but they are 00W0054 parts with Rev A1.

I can crossflash them using the generic firmware available for the CX312A-XCBT but it specifically says A2-A6 so I wasn't sure if this means I'll run into problems later?

Cheers
Do both ports work after the flash to the generic firmware on the IBM 00W0055(aka 00W0054)?? mine are also rev A1 and wanting to cross flash mine to the official Mellanox firmware.
 

josiah

New Member
Feb 4, 2019
7
0
1
Do both ports work after the flash to the generic firmware on the IBM 00W0055(aka 00W0054)?? mine are also rev A1 and wanting to cross flash mine to the official Mellanox firmware.
I'm happy to report that yes, after cross-flashing, both ports are working. You should have no problems but you still probably want to follow the guide on page 1 of this thread to back up the card info.

Cheers
 

747builder

Active Member
Dec 17, 2017
112
58
28
I'm happy to report that yes, after cross-flashing, both ports are working. You should have no problems but you still probably want to follow the guide on page 1 of this thread to back up the card info.

Cheers
Been poking around looking to see if there were FreeBSD tools to cross-flash and I'm not seeing anything.

Did anybody try to do this from a Live Linux USB stick since I don't have any Linux servers around and which distro would be best?
 

mgiammarco

New Member
Jan 6, 2015
29
0
1
52
Hello,
I write here because it seems the most appropriate thread.
I have a ConnectX EN IBM version.
With firmware 2.7 under linux it had a problem (command 0x54 failed: fw status = 0x2) so I flashed with IBM firmware 2.9.1000.
At reboot I was able to reenable ports and use it.
After a reboot and a kernel update now the ports are not working anymore. I have gone back to previous linux kernel, to previous card firmware, without luck.
Have you some idea how I can reenable ports? Can I flash some default configuration?
If I flash Mellanox firmware?
Thanks in advance for any help,
Mario
 

mgiammarco

New Member
Jan 6, 2015
29
0
1
52
Hello,
I write here because it seems the most appropriate thread.
I have a ConnectX EN IBM version.
With firmware 2.7 under linux it had a problem (command 0x54 failed: fw status = 0x2) so I flashed with IBM firmware 2.9.1000.
At reboot I was able to reenable ports and use it.
After a reboot and a kernel update now the ports are not working anymore. I have gone back to previous linux kernel, to previous card firmware, without luck.
Have you some idea how I can reenable ports? Can I flash some default configuration?
If I flash Mellanox firmware?
Thanks in advance for any help,
Mario
I have discovered that if, after reboot, I detach and reattach cables, it starts working. How is it possible?
 

i386

Well-Known Member
Mar 18, 2016
4,403
1,635
113
35
Germany
My guess:
The ports are in "auto config" mode and try to detect if they should use infiniband or ethernet. When the drivers are not laoded the card probably defaults to infiniband. Removing the cable/transceiver and then reinserting the cable causes the card to detect the correct protocol.
 

mgiammarco

New Member
Jan 6, 2015
29
0
1
52
Thanks for reply, how can I force ports to use only ethernet protocol? I suppose I need to use some custom command.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
246
43
Thanks for reply, how can I force ports to use only ethernet protocol? I suppose I need to use some custom command.
Are you using Windows or Linux?

On Windows you'll want to open the properties page for the mellanox board. The device that controls port type is absolutely buried and you'd never find it unless you either read the manual in its entirety or someone tells you. Device Manager > System Devices (near the bottom) > Mellanox {Your NIC Here} > Properties >Port Protocol tab. Set your ports to Eth.

On Linux see below:
Mellanox cards very rarely have issues with transceivers or DACs, as long as what you've got will work with the Cisco switch it should work with the CX3 without issue. You can either put the NIC in Ethernet mode with the firmware utils or add the override parameter to a modprobe definition ala:

echo "options mlx4_core port_type_array=2,2 num_vfs=4,1 probe_vf=4,1" > /etc/modprobe.d/mlx4_core.conf

You only care about "port_type_array=2,2" here.
 

MSX-fan

New Member
May 2, 2019
7
0
1
After double reading this whole chain, it seems that we do not have any luck for cross flash from non pro to CX3 pro. Anyone with success on trying that?
 

TeleFragger

Active Member
Oct 26, 2016
264
55
28
51
is there a way to pull firmware off of a card?
I have old ConnectX-1 cards and I somehow got 2.10.720 on one card a long time ago and I cant find me fw file.
so I'm curious how to read this into a bin so I can flash some others I have...

any help appreciated...
 

DistantStar

New Member
Dec 21, 2019
20
5
3
Got one of the cheapie X-3's off ebay. Tried to update to to firmware 2.42.5000, which is the recommend version for running with the Vmware EXSi 3.17.70.1 driver.
Getting MFE_VERIFY_ERROR, would this indicate a bad flash chip?

2020-05-01.png
 

DistantStar

New Member
Dec 21, 2019
20
5
3
The NIC appears to be completely dead after rebooting ESXI. Ebay seller will be shipping a replacement, not sure if I'll try the firmware upgrade a second time...
 

zmarty

New Member
Aug 21, 2020
1
4
3
Bought cheap MCX353A-FCBT ConnectX-3 Primergy Fujitsu branded card from eBay: MCX353A-FCBT FUJITSU CONNECTX3 VPI 1 PORT QSFP 40GBE ADAPTER CARD W/BOTH BRACKET | eBay

Reported PSID: FJT1060100019

This Fujitsu PSID prevented me from upgrading to the latest firmware version

Read forums here, figured out how to reflash to latest firmware and stock PSID.

When you reflash you should see: You are about to replace current PSID on flash - "FJT1060100019" with a different PSID - "MT_1100120019"

Only issue I see so far is connecting a Linux to Windows machine only gets about 13 Gbps with multiple threads iperf3 (see iperf3 at 40Gbps and above), will need to investigate/tune further. I can get 40 Gbps reliably between these cards, either directly or through a 40 Gbps switch, using multiple iperf processes in parallel. I was not getting the correct speed in the beginning because I was using one of the cards in a PCIe 2 slot instead of 3.

References:
- 家庭向け(仮) 40GbE NIC 導入方法 | 自宅ラックエンジニアの活動ログ
- https://forums.servethehome.com/ind...ectx-3-en-with-stock-firmware.8689/post-80356

===== LINUX REFLASH =====

Ubuntu install example: apt install mstflint

lspci | grep Mellanox (to get the device ID 06:00.0 below - replace with your own device ID!)

mstflint -d 06:00.0 query full > flint_query.txt

mstflint -d 06:00.0 hw query > flint_hwinfo.txt

mstflint -d 06:00.0 ri orig_firmware.bin

mstflint -d 06:00.0 dc orig_firmware.ini

mstflint -d 06:00.0 rrom orig_rom.bin

mstvpd -m 06:00.0 > orig_vpd.txt

# Download firmware for MCX353A-FCBT from Firmware for ConnectX®-3 IB/VPI

wget http://www.mellanox.com/downloads/f...00-MCX353A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip

unzip fw-ConnectX3-rel-2_42_5000-MCX353A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip

mstflint -d 06:00.0 -i fw-ConnectX3-rel-2_42_5000-MCX353A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn


===== WINDOWS REFLASH =====

Run cmd windows as Administrator!

Install MFT: Mellanox Firmware Tools (MFT)

mst start

mst status
(gets list of devices change mt4099_pci_cr0 below with your device id)

flint -d mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX353A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn
 
Last edited:

bookemdano

New Member
Jun 29, 2011
15
0
1
Thanks for the info all. I'm just dipping my toes in the water here, have a lot to learn. I was just given an older IBM ConnectX2 card and would like to flash it to the vanilla Mellanox fw. I tried to use ZMarty's post above as a guide just to get the original fw dumped, but ran into some snags and I was hoping someone more knowledgeable about this process can weigh in on what I should do.

Card Info from stickers:

FRU PN: 00E1493
REV: A2

Mellanox Technologies PN: MNPH29D-XSR

Fired up an ubuntu live CD to run these commands:

Code:
lspci | grep Mellanox
01:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
Code:
sudo mstflint -d 01:00.0 hw query > flint_hwinfo.txt
fint_hwinfo.txt
Code:
HW Info:
  HwDevId               400
  HwRevId               0xb0
Flash Info:
  Type                  M25Pxx
  TotalSize             0x200000
  Banks                 0x1
  SectorSize            0x10000
  WriteBlockSize        0x10
  CmdSet                0x80
Code:
sudo mstflint -d 01:00.0 query full > flint_query.txt
flint_query.txt
Code:
Failed to identify the device - Can not create SignatureManager!
Image type:            FS2
FW Version:            2.9.1000
MIC Version:           1.2.0
Device ID:             26448
Description:      Port1            Port2
MACs:                      xxxxxxxxxxxx     xxxxxxxxxxxx
VSD:                  
PSID:                  IBM0F30000010
Code:
sudo mstflint -d 01:00.0 ri orig_firmware.bin
Failed to identify the device - Can not create SignatureManager!
Code:
sudo mstflint -d 01:00.0 dc orig_firmware.ini
Failed to identify the device - Can not create SignatureManager!
Did produce a long file with configuration parameters. Not sure if it's worth posting the whole thing here.

Code:
sudo mstflint -d 01:00.0 rrom orig_rom.bin
Failed to identify the device - Can not create SignatureManager!
-E- Read ROM failed. The FW does not contain a ROM section
Code:
sudo mstvpd -m 01:00.0 > origvpd.txt
Produces file with a lot of invalid characters but also some intelligible strings.

I stopped there. Would appreciate it greatly if anyone can help me figure out what Mellanox firmware to flash on this card (if that is even possible) and what to do about the "Can not create SignatureManager!" and "The FW does not contain a ROM section" errors when trying to dump the current fw.

Many thanks.

Edit: Just saw the original post on page 1 saying that not all cards have a ROM. So I guess no problem there.
 
Last edited:

Falloutboy

Active Member
Oct 23, 2011
306
32
28
For reference to : sudo -s then enter your password that way you only have to enter sudo one time and not on every line when you are finished type exit otherwise the shell will still be elevated.

Also and I might be doing it the wrong way but I have always used the long form of the device name unless it is nessesary to specifiy a card in a particular PCIe slot I.E /dev/mst/mt26428_pci_cr0 etc I don't know if this will make any difference with commands returning garbage or not.

to get the .vpd data I use this.
mlxburn -d /dev/mst/mt26428_pci_confx -vpd > orig_vpd.txt
 
Last edited:

ieronymous

Member
Oct 26, 2021
44
5
8
2) flint -d mt26448_pci_cr0 -i fw.bin burn -allow_psid_change

where "mt26448_pci_cr0" is the name output from commands 0 or 1
As @Stereodude mentioned, wouldn't it be preferable to use the pciconfX device and not pci_crX ?

I am still to flash mine yet, since i haven t found yet a safely used by many way yet.
Mine are HP branded with firmware-version as seen below.
/# ethtool -i enp59s0
driver: mlx4_en
version: 4.0-0
firmware-version: 2.42.5044
expansion-rom-version:
bus-info: 0000:3b:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

I coudlnt even find if HP had published a new firmware on their site. The results are not very clear to which card they are referring to.
Also you get different results depend to what you re typing in the search bar MT27520 (no results) or ConnectX-3 Pro (enough results)
and this link https://support.hpe.com/connect/s/product?language=en_US&kmpmoid=7832027&tab=driversAndSoftware that dictates
the word ethernet only. Mine is connectx-3 pro converged maybe (it can act as InfiniBand and ethernet depend on the configuration)?