Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

P0rt4lN3T

Member
Nov 10, 2023
46
0
6
You are calculating based on GIGABYTES on the pcie lanws, pcie 3.0 is 8GIGABITS per lane
All networking equipment is based on GIGABITS
View attachment 32682
x16 is 16GB/s, a 40GbE two port needs at most a x8 slot. There is no limitation to the PCIe slots, its something else
hope that actually clears it up for you

2697 V2 has 40x pcie lanes, how its split up depends on the motherboard. 2470v2 on the other hand is only 24 lanes.but both systems have plenty wide pcie lanes with dual cpus (2x individual cpus)
Ok so i was running only 16GB memory ram module on each machine.. made an upgrade to 96GB ram on both machines.. and somehow magically it started working 40gbps single mode transmission datarate and 64gbps full aggregate troughput which matches the limitation on the PCIe 3.0 x8 now..

So now i can confirm that the MCX354A card is working as expected in Ethernet mode.
 

Attachments

NablaSquaredG

Layer 1 Magician
Aug 17, 2020
1,320
800
113
Ok so i was running only 16GB memory ram module on each machine.. made an upgrade to 96GB ram on both machines.. and somehow magically it started working 40gbps single mode transmission datarate and 64gbps full aggregate troughput which matches the limitation on the PCIe 3.0 x8 now..
on linux, default socket / rx buffer size is chosen depending on the system memory.. I don't know whether that applies to windows too.
 

P0rt4lN3T

Member
Nov 10, 2023
46
0
6
You are calculating based on GIGABYTES on the pcie lanws, pcie 3.0 is 8GIGABITS per lane
All networking equipment is based on GIGABITS
View attachment 32682
x16 is 16GB/s, a 40GbE two port needs at most a x8 slot. There is no limitation to the PCIe slots, its something else
hope that actually clears it up for you

2697 V2 has 40x pcie lanes, how its split up depends on the motherboard. 2470v2 on the other hand is only 24 lanes.but both systems have plenty wide pcie lanes with dual cpus (2x individual cpus)
Hi i have one last query...if get a NIC Card 100G same as my MCX455A-ECAT which is full lenght pcie X16 does it means i have 64gbps TX and 64gbps RX total aggregate troughput 128Gbps ? and on PCIe MCX354-FCBT which has smaller pci-e x8 riser.. it can only get 64gbps full troughput aggregate right?
 

DaveLTX

Active Member
Dec 5, 2021
150
34
28
Hi i have one last query...if get a NIC Card 100G same as my MCX455A-ECAT which is full lenght pcie X16 does it means i have 64gbps TX and 64gbps RX total aggregate troughput 128Gbps ? and on PCIe MCX354-FCBT which has smaller pci-e x8 riser.. it can only get 64gbps full troughput aggregate right?
pcie is bidirectional. RX and TX is done independently.
Yes, on a x8 connection and pcie 3.0 (the last part is important!)
You only get 64gigabits per direction.
You will need full x16 for 100gbps.
 

Spelon1

New Member
Feb 5, 2021
7
1
3
Trying to flash some cards I bought on ebay, sadly some of them had hardware access disabled with key. Anyone know if there is a way to reset the key/card with pins or command?
 

kevindd992002

Member
Oct 4, 2021
110
4
18
When flashing OEM CX3 cards with the fw from Nvidia, do the GUID's and MAC's get rewritten to a generic one which makes us want to flash with the switch that writes a MAC address to the card or something?
 

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
When flashing OEM CX3 cards with the fw from Nvidia, do the GUID's and MAC's get rewritten to a generic one which makes us want to flash with the switch that writes a MAC address to the card or something?
No, at least not with the "normal" flashing process that is described in multiple threads.
If you use the recovery mode the guids and mac will be blank.
 

kevindd992002

Member
Oct 4, 2021
110
4
18
No, at least not with the "normal" flashing process that is described in multiple threads.
If you use the recovery mode the guids and mac will be blank.
If you enable SRIOV though, do you need to assign the VLAN and the unique MAC per VF? If so, what mft command does this?
 

inf3rno

Member
Mar 24, 2021
36
1
8
If you're just flashing a MCX354A-QCBT (649281-B21 on ebay) to a MCX354A-FCBT (full 40gbE enabled), you don't need to compile a firmware image - you can just flash the premade latest MCX354A-FCBT image from their site - this is nice because the latest firmware sources available for compiling are actually quite old.

When you use the firmware sources with the stock unedited FCBT ini file to compile a new image, this is all you're generating anyway. Except the pre-generated images on mellanox's site have been compiled with much newer firmware sources

from my notes (I usually just do this with a debian or ubuntu live ISO):

Code:
##Download latest Mellanox Firmware tools and install them  + dependencies:
apt update && apt install gcc make dkms unzip linux-headers-$(uname -r)
wget https://www.mellanox.com/downloads/MFT/mft-4.18.0-106-x86_64-deb.tgz
tar -xvf mft-4.18.0-106-x86_64-deb.tgz && cd mft-4.18.0-106-x86_64-deb
./install.sh

#Start MST service:
mst start
mst status
#copy the dev address with cr0 in it, like:
/dev/mst/mt4099_pci_cr0
#Use that address in the following commands

#Download latest FCBT firmware and unzip:
wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip
unzip fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip

#Backup the cards current board definition file just in case
flint -d /dev/mst/mt4099_pci_cr0 dc orig_firmware.ini

#Flash the FCBT image to the card
flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn

Cold boot the server when done
Some useful config commands to config the card when it comes back up:

Code:
#start mst service again
mst start

#get detailed firmware info
mlxfwmanager --query

#get the device ID again:
mst status

#use it to see the cards current configuration:
mlxconfig -d /dev/mst/mt4099_pci_cr0 query

use it in config commands to configure the card
#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2


#turn off bootrom crap
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P1=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P2=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P1=0
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P2=0


##optional: delete bootrom off the card, so it doesn't slow down boot by popping up crap
##this is safe to do and supported by mellanox
flint -d /dev/mst/mt4099_pci_cr0 --allow_rom_change drom
I recently tried this and it did not work. It wrote that the firmware is not made for the HP card. So I guess Nvidia updated the firmware and added some protection against flashing cheaper cards or idk.
 

DaveLTX

Active Member
Dec 5, 2021
150
34
28
I recently tried this and it did not work. It wrote that the firmware is not made for the HP card. So I guess Nvidia updated the firmware and added some protection against flashing cheaper cards or idk.
did you follow exactly? and precisely what card do you have?
 

Jamy

Member
Mar 29, 2017
44
6
8
48
I'm relatively new to the 10g plus networking world. Are these Mellanox MCX354A-FCBT cards able to do 25g? I just picked up a Unifi switch that has 4 25g ports.
 

DaveLTX

Active Member
Dec 5, 2021
150
34
28
I'm relatively new to the 10g plus networking world. Are these Mellanox MCX354A-FCBT cards able to do 25g? I just picked up a Unifi switch that has 4 25g ports.
No absolutely not. 25G is SFP28. These are QSFP+.