Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

[P0rt4lN3T]

You are calculating based on GIGABYTES on the pcie lanws, pcie 3.0 is 8GIGABITS per lane
All networking equipment is based on GIGABITS
View attachment 32682
x16 is 16GB/s, a 40GbE two port needs at most a x8 slot. There is no limitation to the PCIe slots, its something else
hope that actually clears it up for you

2697 V2 has 40x pcie lanes, how its split up depends on the motherboard. 2470v2 on the other hand is only 24 lanes.but both systems have plenty wide pcie lanes with dual cpus (2x individual cpus)

Ok so i was running only 16GB memory ram module on each machine.. made an upgrade to 96GB ram on both machines.. and somehow magically it started working 40gbps single mode transmission datarate and 64gbps full aggregate troughput which matches the limitation on the PCIe 3.0 x8 now..

So now i can confirm that the MCX354A card is working as expected in Ethernet mode.

 

[NablaSquaredG]

Ok so i was running only 16GB memory ram module on each machine.. made an upgrade to 96GB ram on both machines.. and somehow magically it started working 40gbps single mode transmission datarate and 64gbps full aggregate troughput which matches the limitation on the PCIe 3.0 x8 now..

on linux, default socket / rx buffer size is chosen depending on the system memory.. I don't know whether that applies to windows too.

 

[P0rt4lN3T]

on linux, default socket / rx buffer size is chosen depending on the system memory.. I don't know whether that applies to windows too.

Hi I was testing on a Linux OS system called RouterOS x86_64 v7.12

 

[P0rt4lN3T]

You are calculating based on GIGABYTES on the pcie lanws, pcie 3.0 is 8GIGABITS per lane
All networking equipment is based on GIGABITS
View attachment 32682
x16 is 16GB/s, a 40GbE two port needs at most a x8 slot. There is no limitation to the PCIe slots, its something else
hope that actually clears it up for you

2697 V2 has 40x pcie lanes, how its split up depends on the motherboard. 2470v2 on the other hand is only 24 lanes.but both systems have plenty wide pcie lanes with dual cpus (2x individual cpus)

Hi i have one last query...if get a NIC Card 100G same as my MCX455A-ECAT which is full lenght pcie X16 does it means i have 64gbps TX and 64gbps RX total aggregate troughput 128Gbps ? and on PCIe MCX354-FCBT which has smaller pci-e x8 riser.. it can only get 64gbps full troughput aggregate right?

 

[DaveLTX]

Hi i have one last query...if get a NIC Card 100G same as my MCX455A-ECAT which is full lenght pcie X16 does it means i have 64gbps TX and 64gbps RX total aggregate troughput 128Gbps ? and on PCIe MCX354-FCBT which has smaller pci-e x8 riser.. it can only get 64gbps full troughput aggregate right?

pcie is bidirectional. RX and TX is done independently.
Yes, on a x8 connection and pcie 3.0 (the last part is important!)
You only get 64gigabits per direction.
You will need full x16 for 100gbps.

 

[Spelon1]

Trying to flash some cards I bought on ebay, sadly some of them had hardware access disabled with key. Anyone know if there is a way to reset the key/card with pins or command?

 

[kevindd992002]

When flashing OEM CX3 cards with the fw from Nvidia, do the GUID's and MAC's get rewritten to a generic one which makes us want to flash with the switch that writes a MAC address to the card or something?

 

[i386]

When flashing OEM CX3 cards with the fw from Nvidia, do the GUID's and MAC's get rewritten to a generic one which makes us want to flash with the switch that writes a MAC address to the card or something?

No, at least not with the "normal" flashing process that is described in multiple threads.
If you use the recovery mode the guids and mac will be blank.

 

[kevindd992002]

No, at least not with the "normal" flashing process that is described in multiple threads.
If you use the recovery mode the guids and mac will be blank.

If you enable SRIOV though, do you need to assign the VLAN and the unique MAC per VF? If so, what mft command does this?

 

[inf3rno]

If you're just flashing a MCX354A-QCBT (649281-B21 on ebay) to a MCX354A-FCBT (full 40gbE enabled), you don't need to compile a firmware image - you can just flash the premade latest MCX354A-FCBT image from their site - this is nice because the latest firmware sources available for compiling are actually quite old.

When you use the firmware sources with the stock unedited FCBT ini file to compile a new image, this is all you're generating anyway. Except the pre-generated images on mellanox's site have been compiled with much newer firmware sources

from my notes (I usually just do this with a debian or ubuntu live ISO):

##Download latest Mellanox Firmware tools and install them  + dependencies:
apt update && apt install gcc make dkms unzip linux-headers-$(uname -r)
wget https://www.mellanox.com/downloads/MFT/mft-4.18.0-106-x86_64-deb.tgz
tar -xvf mft-4.18.0-106-x86_64-deb.tgz && cd mft-4.18.0-106-x86_64-deb
./install.sh

#Start MST service:
mst start
mst status
#copy the dev address with cr0 in it, like:
/dev/mst/mt4099_pci_cr0
#Use that address in the following commands

#Download latest FCBT firmware and unzip:
wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip
unzip fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip

#Backup the cards current board definition file just in case
flint -d /dev/mst/mt4099_pci_cr0 dc orig_firmware.ini

#Flash the FCBT image to the card
flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn

Cold boot the server when done

Some useful config commands to config the card when it comes back up:

#start mst service again
mst start

#get detailed firmware info
mlxfwmanager --query

#get the device ID again:
mst status

#use it to see the cards current configuration:
mlxconfig -d /dev/mst/mt4099_pci_cr0 query

use it in config commands to configure the card
#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2


#turn off bootrom crap
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P1=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P2=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P1=0
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P2=0


##optional: delete bootrom off the card, so it doesn't slow down boot by popping up crap
##this is safe to do and supported by mellanox
flint -d /dev/mst/mt4099_pci_cr0 --allow_rom_change drom

I recently tried this and it did not work. It wrote that the firmware is not made for the HP card. So I guess Nvidia updated the firmware and added some protection against flashing cheaper cards or idk.

 

[DaveLTX]

I recently tried this and it did not work. It wrote that the firmware is not made for the HP card. So I guess Nvidia updated the firmware and added some protection against flashing cheaper cards or idk.

did you follow exactly? and precisely what card do you have?

 

[Jamy]

I'm relatively new to the 10g plus networking world. Are these Mellanox MCX354A-FCBT cards able to do 25g? I just picked up a Unifi switch that has 4 25g ports.

 

[DaveLTX]

I'm relatively new to the 10g plus networking world. Are these Mellanox MCX354A-FCBT cards able to do 25g? I just picked up a Unifi switch that has 4 25g ports.

No absolutely not. 25G is SFP28. These are QSFP+.

 

[dmerillat]

I've got a weird one. MCX354A-FCB_A2-A5. I think, it was originally HP 689283-b21 flexlom before I crossflashed it.

PSID was HP_0230240019 and HP has a slightly newer version, 2.42.5030

The HP firmware was locked to HP modules and didn't play nice with the cheap xqx2502 40GB-LR4 modules on ebay.

Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  /dev/mst/mt4099_pci_cr0
  Port1 MAC:        24be05978a41
  Port2 MAC:        24be05978a42
  Versions:         Current        Available     
     FW             2.42.5000      N/A           
     PXE            3.4.0752       N/A           

  Status:           No matching image found

Crossflash worked, and the DAC comes up, but I can't get any optical info out of it either port.

# ethtool -m enp65s0d1
netlink error: Input/output error
mlx4_core 0000:41:00.0: MLX4_CMD_MAD_IFC Get Module ID attr(ff60) port(2) i2c_addr(50) offset(0) size(1): Response Mad Status(91c) - I2C bus is constantly busy

With HP firmware it was giving an io error on the kaiam but working with the DAC.

It's unfortunate because my ICX6610 won't give me optical info for its end either.

I've searched but I'm not seeing that error anywhere.

 

[DaveLTX]

I've got a weird one. MCX354A-FCB_A2-A5. I think, it was originally HP 689283-b21 flexlom before I crossflashed it.

PSID was HP_0230240019 and HP has a slightly newer version, 2.42.5030

The HP firmware was locked to HP modules and didn't play nice with the cheap xqx2502 40GB-LR4 modules on ebay.

Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  /dev/mst/mt4099_pci_cr0
  Port1 MAC:        24be05978a41
  Port2 MAC:        24be05978a42
  Versions:         Current        Available    
     FW             2.42.5000      N/A          
     PXE            3.4.0752       N/A          

  Status:           No matching image found

Crossflash worked, and the DAC comes up, but I can't get any optical info out of it either port.

# ethtool -m enp65s0d1
netlink error: Input/output error
mlx4_core 0000:41:00.0: MLX4_CMD_MAD_IFC Get Module ID attr(ff60) port(2) i2c_addr(50) offset(0) size(1): Response Mad Status(91c) - I2C bus is constantly busy

With HP firmware it was giving an io error on the kaiam but working with the DAC.

It's unfortunate because my ICX6610 won't give me optical info for its end either.

I've searched but I'm not seeing that error anywhere.

Only 281 can be cross flashed and still work. There are no mellanox equivalents afaik for 283. Same for the 700 series cards.

 

[dmerillat]

Only 281 can be cross flashed and still work. There are no mellanox equivalents afaik for 283. Same for the 700 series cards.

Well damn. It's really close to working, and in fact it did work after flashing stock firmware and doing a warm reset. I was getting traffic on both the DAC and the 40g-lr4. After a cold reboot it couldn't detect the modules and had driver resets, so something isn't initializing properly.

Ideas on convincing the HP firmware to accept third-party modules, or should I just grab a different OEM that can be cross-flashed to stock.

 

[DaveLTX]

Well damn. It's really close to working, and in fact it did work after flashing stock firmware and doing a warm reset. I was getting traffic on both the DAC and the 40g-lr4. After a cold reboot it couldn't detect the modules and had driver resets, so something isn't initializing properly.

Ideas on convincing the HP firmware to accept third-party modules, or should I just grab a different OEM that can be cross-flashed to stock.

Probably best to pick up a different card. I distinctly might remember that these even without vendor lock (I'm not sure it even has it but i don't have a QSFP AOC) would not work with any module that is over 1.5W

I might be wrong but that's way lower than most QSFP+ modules

 

[dmerillat]

It worked with the 2.5w LR4 I tested with, but that may have been the problem.

E: wrong button.

I've got a pair of them so the one with HP firmware got swapped in and is running things, and the cross-flashed one is on my lab bench where I can screw around with it without knocking my network down. There's a few things to check, to see if the .ini or config has any changes, play with various settings.

What's a cheap ebay QSFP+ card that behaves well with 40G-LR4/LR4-lite? I'm not married to these HP flexloms.

 

[dmerillat]

After far too much time spent, it's semi-working.

I had to go back to stock 2.40.5000 with the firmware kit and build a MCX354A-FCB_A2-A5 image myself with a modded .ini where I copied various GPIO settings and power levels over.

I've got the link up and data flowing. I wouldn't recommend it, it'd be a lot easier just to buy the right card.

 

[DaveLTX]

After far too much time spent, it's semi-working.

I had to go back to stock 2.40.5000 with the firmware kit and build a MCX354A-FCB_A2-A5 image myself with a modded .ini where I copied various GPIO settings and power levels over.

I've got the link up and data flowing. I wouldn't recommend it, it'd be a lot easier just to buy the right card.

Like I said, its the wrong board. 283-B21 and onwards are Rev C (meaning C1-C5 etc) and there exists no mellanox firmware for that.
Any Rev A firmware does not work. I tried myself and the only way is to dump the firmware on them and extract them then modify the ini but I haven't been able to do that myself. Maybe someone can chime in

On the flipside, they dont need flashing like the 281 cards to enable 40G, they already have it enabled