Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

Churchill

Admiral
Jan 6, 2016
808
196
43
I did what @kiteboarder did on 3 cards and worked like a champ through the process.

My only question is how can I truly test the cards to see if they will appear as full Ethernet devices?
 
  • Like
Reactions: kiteboarder

Milos

New Member
Feb 22, 2018
25
2
3
Black hole with internet
These are the notes I took down for myself when I did this a year or so back. YMMV. Please be careful using -allowpsidchange option.

Flashing firmware on HP 649281-B21 to stock Mellanox MCX354A-FCBT (ConnectX-3) guide:

For Windows Server 2016:

Step 1 - download and install:
Mellanox WinMFT_x64.exe (latest version from Mellanox Website under firmware tools)

Step 2 - download firmware for your card from Mellanox Website. Mine was an HP 649281-B21 which means i needed the MCX354A-FCBT firmware. Be sure which card is yours.
Rename downloaded firmware file to fw.bin
copy it to:
C:\Program Files\Mellanox\WinMFT>

Step 3 - open command prompt as administrator
cd to directory:
C:\Program Files\Mellanox\WinMFT>

Run:
0) mst status
1) mlxfwmanager
2) flint -d mt4099_pci_cr0 -i fw.bin -allow_psid_change burn

where "mt4099_pci_cr0" is the name output from commands 0 or 1

It should say "You are about to replace current PSID on flash - "HP_02802xxxxx" with a different PSID... etc.
Do you want to continue, type Y.


Flashing is done , 3 cards all is ok thanks a lot.
 
  • Like
Reactions: kiteboarder

cuco

Member
Feb 13, 2018
57
4
8
36
Hello,
i bought two Fujitsu single port QSFP cards. MT4099 so ConnectX-3 but i cant exactly figure out which hardware version it is, to choose the right firmware.
This is the Link i purchased them from:
Fujitsu ConnectX-3 VPI 1-Port QSFP FDR 56GbE LP - S26361-F4533-L102 | eBay

Another question. I will use the cards in a server 2019 envirenment just to connect my alsways on server with my optional running server via a Brocase 6610. No special configs with VLans or so. Should or do i have to flash them to ethernet mode?
Perhaps anyone can help. Would be very kind.

Thanks a lot!
 

mb300sd

Active Member
Aug 1, 2016
204
80
28
31
Hello,
i bought two Fujitsu single port QSFP cards. MT4099 so ConnectX-3 but i cant exactly figure out which hardware version it is, to choose the right firmware.
This is the Link i purchased them from:
Fujitsu ConnectX-3 VPI 1-Port QSFP FDR 56GbE LP - S26361-F4533-L102 | eBay

Another question. I will use the cards in a server 2019 envirenment just to connect my alsways on server with my optional running server via a Brocase 6610. No special configs with VLans or so. Should or do i have to flash them to ethernet mode?
Perhaps anyone can help. Would be very kind.

Thanks a lot!
Looks like MCX353A-FCBT to me. Change to ethernet mode in device manager (click the network adapter, View->Devices by connection, open the properties for the device 1 up from it)
 

cuco

Member
Feb 13, 2018
57
4
8
36
Ok this seems to have worked. I uses mighty fohdeesha's example.
Card is set to ethernet mode as well. I use the WinOF 5.50.51 drivers. But there is now WakeOnLan option.
When i read right 5.50.51 does not support WOL.
I tried to use the 5.35 with the latest firmware. but this does not seem to work. the system doesnt use the driver.
I am running on Server2019 and their only is a server 2016 5.35 driver. Any idea how to get WOL to work? Do i have to downgrade the firmware?
 

KC8FLB

Member
Aug 12, 2018
71
54
18
These are the notes I took down for myself when I did this a year or so back. YMMV. Please be careful using -allowpsidchange option.

Flashing firmware on HP 649281-B21 to stock Mellanox MCX354A-FCBT (ConnectX-3) guide:

For Windows Server 2016:

Step 1 - download and install:
Mellanox WinMFT_x64.exe (latest version from Mellanox Website under firmware tools)

Step 2 - download firmware for your card from Mellanox Website. Mine was an HP 649281-B21 which means i needed the MCX354A-FCBT firmware. Be sure which card is yours.
Rename downloaded firmware file to fw.bin
copy it to:
C:\Program Files\Mellanox\WinMFT>

Step 3 - open command prompt as administrator
cd to directory:
C:\Program Files\Mellanox\WinMFT>

Run:
0) mst status
1) mlxfwmanager
2) flint -d mt4099_pci_cr0 -i fw.bin -allow_psid_change burn

where "mt4099_pci_cr0" is the name output from commands 0 or 1

It should say "You are about to replace current PSID on flash - "HP_02802xxxxx" with a different PSID... etc.
Do you want to continue, type Y.
Thank you, worked wonderfully on a Win 10 box.
 
  • Like
Reactions: kiteboarder

saverio

New Member
Jun 24, 2020
5
0
1
Hello,
I have flashed a CX354A-QCBT to -FCBT using the guide in this thread. I tried to cross connect the two ports using an FS.com QSFP DAC, but the link is 10Gb, not 40. If I try to set it manually to 40000 with ethtool, it doesn't go up.


Could it be a cable issue?

Code:
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  0000:02:00.0
  Port1 MAC:        506b4b7cf4b1
  Port2 MAC:        506b4b7cf4b2
  Versions:         Current        Available
     FW             2.42.5000      2.42.5000
     PXE            3.4.0752       3.4.0752

  Status:           Up to date
Code:
ethtool enp2s0
Settings for enp2s0:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                56000baseCR4/Full
                                56000baseSR4/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: yes
 

Freebsd1976

Active Member
Feb 23, 2018
295
44
28
Hello,
I have flashed a CX354A-QCBT to -FCBT using the guide in this thread. I tried to cross connect the two ports using an FS.com QSFP DAC, but the link is 10Gb, not 40. If I try to set it manually to 40000 with ethtool, it doesn't go up.


Could it be a cable issue?

Code:
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  0000:02:00.0
  Port1 MAC:        506b4b7cf4b1
  Port2 MAC:        506b4b7cf4b2
  Versions:         Current        Available
     FW             2.42.5000      2.42.5000
     PXE            3.4.0752       3.4.0752

  Status:           Up to date
Code:
ethtool enp2s0
Settings for enp2s0:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                56000baseCR4/Full
                                56000baseSR4/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: yes
Maybe or may not be , I have flash 5 qcbt to fcbt .
1 one of them only one port can work on 40/56 ,the second port work on 10g.
2. the other four card work as normal fcbt .
 
Jan 10, 2019
129
25
18
blog.azureinfra.com
and for the Windows Users: (very similar to the opening post:)

#Download Tools: https://www.mellanox.com/downloads/MFT/WinMFT_x64_4_15_0_104.exe
#Download Firmware: http://www.mellanox.com/downloads/f...00-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip

#Install the tools, then go to
CD .\Program Files\Mellanox\WinMFT

#Query the device name (use the 2nd name returned looking like: mt4099_pciconf0)
mdevices status

#Backup old firmware
flint -d mt4099_pciconf0 dc orig_fw.ini

#Burn new firmware
flint -d mt4099_pciconf0 -i c:\<folder>\fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn

#disable boot stuff
mlxconfig -d mt4099_pciconf0 set BOOT_OPTION_ROM_EN_P1=false
mlxconfig -d mt4099_pciconf0 set BOOT_OPTION_ROM_EN_P2=false
mlxconfig -d mt4099_pciconf0 set LEGACY_BOOT_PROTOCOL_P1=0
mlxconfig -d mt4099_pciconf0 set LEGACY_BOOT_PROTOCOL_P2=0


#remove the boot ROM
flint -d mt4099_pciconf0 --allow_rom_change drom

#check current settings
mlxconfig -d mt4099_pciconf0 query


#Reboot Machine
 

dpsi

New Member
Dec 14, 2015
3
0
1
Just wanted to mention that I was able to do this succesfully with a Revision A6 649281-B21 using the A2-A5 firmware.
 

tryingtorunservers

New Member
Nov 28, 2020
8
0
1
Hi, I was able to flash two MCX354A-QCBT to MCX354A-FCBT firmware, but I seem to be getting just ~12Gbps speed when running ipref tests between a Linux NAS and Windows 10 client even though I confirmed that the firmware were flashed correctly, and the OSs reports 40Gbps link speed on both hosts. I'm using Mellanox MC2207130-001 cable that supports 40GbE. Does the cross-flashing actually not unlock 40GbE? Am I CPU-bottlenecked (the NAS is running an embedded CPU)? What could I be doing wrong?

Thanks a lot for your help,

ipref on the Linux NAS:

Code:
-----------------------------------------------------------
Server listening on 1234
-----------------------------------------------------------
Accepted connection from 192.168.101.100, port 60195
[  5] local 192.168.101.1 port 1234 connected to 192.168.101.100 port 60197
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  1.27 GBytes  10.9 Gbits/sec
[  5]   1.00-2.00   sec  1.44 GBytes  12.3 Gbits/sec
[  5]   2.00-3.00   sec  1.45 GBytes  12.5 Gbits/sec
[  5]   3.00-4.00   sec  1.49 GBytes  12.8 Gbits/sec
[  5]   4.00-5.00   sec  1.43 GBytes  12.3 Gbits/sec
[  5]   5.00-6.00   sec  1.48 GBytes  12.7 Gbits/sec
[  5]   6.00-7.00   sec  1.42 GBytes  12.2 Gbits/sec
[  5]   7.00-8.00   sec  1.39 GBytes  11.9 Gbits/sec
[  5]   8.00-9.00   sec  1.33 GBytes  11.4 Gbits/sec
[  5]   9.00-10.00  sec  1.30 GBytes  11.2 Gbits/sec
[  5]  10.00-10.03  sec  39.9 MBytes  10.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.03  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.03  sec  14.0 GBytes  12.0 Gbits/sec                  receiver
Code:
> .\iperf3.exe -c 192.168.101.1 -p 1234 -d
send_parameters:
{
        "tcp":  true,
        "omit": 0,
        "time": 10,
        "parallel":     1,
        "len":  131072,
        "client_version":       "3.1.3"
}
Connecting to host 192.168.101.1, port 1234
SO_SNDBUF is 212992
[  4] local 192.168.101.100 port 60197 connected to 192.168.101.1 port 1234
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.30 GBytes  11.1 Gbits/sec
[  4]   1.00-2.00   sec  1.45 GBytes  12.5 Gbits/sec
[  4]   2.00-3.00   sec  1.46 GBytes  12.5 Gbits/sec
[  4]   3.00-4.00   sec  1.48 GBytes  12.7 Gbits/sec
[  4]   4.00-5.00   sec  1.44 GBytes  12.3 Gbits/sec
[  4]   5.00-6.00   sec  1.48 GBytes  12.7 Gbits/sec
[  4]   6.00-7.00   sec  1.42 GBytes  12.2 Gbits/sec
[  4]   7.00-8.00   sec  1.37 GBytes  11.8 Gbits/sec
[  4]   8.00-9.00   sec  1.34 GBytes  11.5 Gbits/sec
send_results
{
        "cpu_util_total":       83.620775,
        "cpu_util_user":        34.689192,
        "cpu_util_system":      48.931583,
        "sender_has_retransmits":       0,
        "streams":      [{
                        "id":   1,
                        "bytes":        15074459648,
                        "retransmits":  -1,
                        "jitter":       0,
                        "errors":       0,
                        "packets":      0
                }]
}
get_results
{
        "cpu_util_total":       2.208995,
        "cpu_util_user":        0.031556,
        "cpu_util_system":      2.177375,
        "sender_has_retransmits":       -1,
        "streams":      [{
                        "id":   1,
                        "bytes":        15074459648,
                        "retransmits":  -1,
                        "jitter":       0,
                        "errors":       0,
                        "packets":      0
                }]
}
[  4]   9.00-10.00  sec  1.30 GBytes  11.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  14.0 GBytes  12.1 Gbits/sec                  sender
[  4]   0.00-10.00  sec  14.0 GBytes  12.1 Gbits/sec                  receiver

iperf Done.

Linux NAS:
Code:
# mlxfwmanager --query
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  0000:07:00.0
  Port1 MAC:        0002c9a595a1
  Port2 MAC:        0002c9a595a2
  Versions:         Current        Available
     FW             2.42.5000      N/A
     PXE            3.4.0752       N/A

  Status:           No matching image found
Code:
# ethtool eth5
Settings for eth5:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                56000baseCR4/Full
                                56000baseSR4/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Advertised link modes:  1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Speed: Unknown!
        Duplex: Unknown! (255)
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: no
Windows 10 client:
Code:
> mlxfwmanager --query
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019
  PCI Device Name:  mt4099_pci_cr0
  Port1 MAC:        0002c9e78d51
  Port2 MAC:        0002c9e78d52
  Versions:         Current        Available
     FW             2.42.5000      N/A
     PXE            3.4.0752       N/A

  Status:           No matching image found
Code:
> Get-NetAdapter | select interfaceDescription, name, status, linkSpeed

interfaceDescription                     name            Status      LinkSpeed
--------------------                     ----            ------      ---------
Mellanox ConnectX-3 Ethernet Adapter #4  Ethernet 11     Up          40 Gbps
 

NablaSquaredG

Active Member
Aug 17, 2020
386
146
43
I can confirm that the speed of crossflashed (Oracle branded QCBT -> FCBT) CX3's is not capped, so likely another issue with your system.

What's the product number of your CPU?

What are your other network settings (e.g. MTU)?
 

kiteboarder

Active Member
May 10, 2016
108
53
28
42
Just a guess, you may have one card in a system's pci-e slot that's only gen 2 x 4 lanes which would net you a max of 16 Gb/s even though the 40Gb link is active.
 

tryingtorunservers

New Member
Nov 28, 2020
8
0
1
Thanks a lot for the note.

Re: network settings:
I started looking into MTU settings, they're all default at 1500. Learning how to change them and will report back how things are. I assume I want them to be like 9000? I also changed windows driver to tune for single port traffic and that seems to have added 1-2Gbps

Re: CPU
Windows client: Intel Core i7-6700K
Linux NAS (Synology DS1621+): AMD Ryzen V1500B

Re: PCI-E
Windows client: PCI-E 3.0 x8
Linux NAS (Synology DS1621+): PCI-E 3.0 x4

Doing some napkin math, PCI-E 3.0 x4 should at least support one port at 40Gbps?
 

kiteboarder

Active Member
May 10, 2016
108
53
28
42
PCI-E gen 3 is 8 Gb/s per lane, so right there you're down to 32 Gb/s max theoretical. And even then it won't be that high due to a little bit of coding overhead.

I don't recommend changing the MTU unless it's a direct computer to computer link.

I also recommend being honest with what your NAS can do read/write wise. Unless it can really do more than 16 Gb/s (2 GB/s) then don't worry about your link performance - it's good enough. ;-)
 

tryingtorunservers

New Member
Nov 28, 2020
8
0
1
Thank you for the note.

I don't recommend changing the MTU unless it's a direct computer to computer link.
I'm doing a direct link right now but plan to move to a switch at some point. I'm curious how MTU matters for computer/computer vs. through a switch?

I also recommend being honest with what your NAS can do read/write wise. Unless it can really do more than 16 Gb/s (2 GB/s) then don't worry about your link performance - it's good enough. ;-)
My plan was to add NVMe SSDs as caches in the NAS and hoping the caches will make it comparable to at least a SATA SSD. This way, I can just use the NAS as if it's an internal drive with drive redundancy built-in. I'm running ipref to ensure the network won't be a bottleneck since the NVMe SSDs have 2.0GB/s+ sequential performance.
 

i386

Well-Known Member
Mar 18, 2016
2,677
792
113
32
Germany
IPerf on windows is not the same as iperf
but I seem to be getting just ~12Gbps speed when running ipref tests between a Linux NAS and Windows 10 client
That's because the windows implementation of iperf is not optimized.
Boot linux from a live cd/usb on the windows host and try iperf or boot windows on both machines and use ntttcp for network testing.
 

inf3rno

New Member
Mar 24, 2021
9
1
3
Hello,
i bought two Fujitsu single port QSFP cards. MT4099 so ConnectX-3 but i cant exactly figure out which hardware version it is, to choose the right firmware.
This is the Link i purchased them from:
Fujitsu ConnectX-3 VPI 1-Port QSFP FDR 56GbE LP - S26361-F4533-L102 | eBay

Another question. I will use the cards in a server 2019 envirenment just to connect my alsways on server with my optional running server via a Brocase 6610. No special configs with VLans or so. Should or do i have to flash them to ethernet mode?
Perhaps anyone can help. Would be very kind.

Thanks a lot!
Did you manage to convert the Fujitsu?